KMImpact Challenge 2011
The Power of a New, Web-Based Collaborative Tool to Deliver Essential Information
to Health Professionals in Developing Countries
Johns Hopkins Bloomberg School of Public Health
Center for Communication Programs
Knowledge for Health (K4Health) Project
111 Market Place, Suite 310
Baltimore, MD 21202
toolkits@k4health.org
Describe the KM initiative
Limited access to information remains a barrier to evidence-based health care in developing
countries. The INFO Project at Johns Hopkins and partner organizations developed online
collections of essential information vetted by public health experts for use by health
professionals in developing countries. These toolkits required considerable resources due to the
collaborative work flow and programming for the interface. The follow-on project, Knowledge
for Health (K4Health), endeavored to use information technology to improve efficiency in and
simplify the work flow. An inter-disciplinary team at K4Health worked together to develop an
easy-to-use, web-based, collaborative tool, called the K4Health eToolkit Application, that public
health practitioners without specialized IT skills could use to create these electronic libraries or
websites. Public health practitioners and information scientists identified goals, tasks, and
work-flow patterns that drove the interface design. Software developers identified open-source
software called Drupal that could automate and support the work flow and defined the scope
of possibilities for the interface design. Through an iterative process, software developers
adapted the open-source software to meet the defined goals while the public health
practitioners and information scientists evaluated web-based prototypes.
Describe the approaches utilized to measure / assess this KM initiative
Ongoing monitoring of user feedback and technical support questions have helped inform
improvements to the eToolkit Application. For example, several users indicated that they need
the ability to manually decide the ordering of resources on a page, in lieu of the automated
design currently in place. K4Health implemented an interim solution and is currently updating
the software to address this issue fully in the next release.
During 2011, K4Health will also conduct formal usability testing of the eToolkit Application with
new and existing users in both developed and developing countries. Usability testing is
concerned with how well users can use the functionality in a given interactive interface. It
involves asking users to perform one or more tasks and then recording their performance and
answers to questions. A well-established evaluation method in the development lifecycle of
computer systems and software, usability testing can be a simple and inexpensive method that
yields high returns in improving the interface of a product. Employing the “think-aloud”
1
KMImpact Challenge 2011
technique during usability testing further enhances the evaluation methodology. While test
participants are using the system, the evaluators ask them to “think out loud” to gain insight
into the users’ thought processes. This combined technique allows evaluators to not only
observe what the participants are doing with the interface, but also why they are doing it.
K4Health has already conducted pilot tests with one user in South Africa and two users in the
U.S. Evaluators recorded audio from the test sessions and the participants’ mouse movements
using screen video capturing software (CamStudio and Jing). The pilot tests identified useful
and actionable findings about usability of the eToolkit Application. In addition, the evaluators
revised some of the test tasks that were confusing to the test participants and scaled down the
number of tasks to ensure they could be accomplished in the time allotted for each session.
K4Health plans to conduct usability testing on an annual basis. In addition, K4Health will
conduct interviews or focus-group discussions with existing users to explore their perceptions
of the software and understand better how they integrate it into their workflow.
What was the purpose or motivation for assessing this KM initiative?
Usability testing will ensure that the eToolkit Application meets the needs of the intended
audience and it will inform the iterative design of the software, helping K4Health to decide the
future direction of the software.
What were the most important lessons learned about the assessment process? -250
Pilot usability testing has already identified useful information about usability issues with the
software that cannot be gained from indirect monitoring methods, such as analysis of technical
support questions. For example, the evaluators discovered several usability problems with the
software’s resource record forms including difficulty understanding the meaning of certain
fields and not being clear on which fields were required.
Pilot testing was an important part of the evaluation process. The evaluators discovered that
they had to revise certain test tasks to avoid confusing the test participants. Also, the
evaluators allotted 10 to 20 minutes for the test, but it actually took about one hour to
complete each test session. Many of the test tasks were actually more difficult for the test
participants than the evaluators had expected, which therefore required more time to
complete.
Using the “think-aloud” technique takes some practice to implement. While it is not intended
to be used as a method to guide the user, test participants who are unfamiliar with the
technique tend to use it as a way to gain reassurance about their performance and guidance
about “what to do next.” Evaluators need practice to explain clearly the intent of the usability
test and think-aloud technique, and most of all, to build rapport and put the test participant at
ease before starting the test.
2
KMImpact Challenge 2011
What would you do differently next time?
K4Health had worked on toolkits with intended users of the eToolkit Application before
developing the software, which informed the software design. However, more formal formative
research to understand specifically what users want and how they think and behave would
have better informed the software design, saving time and resources. In addition, conducting
usability testing with simple paper prototypes in the early phases of the development process
would have helped avoid unnecessary software coding and recoding.
What advice would you give to others based on your experience?
Monitoring and evaluation specialists recommend conducting formative research about the
intended audience before designing or implementing a new program. User experience
professionals also recommend conducting user research before embarking on new software
development. This phase tends to be neglected due to limited time or resources. However,
conducting this critical phase at the beginning will only save time and resources later in the
process. In addition, usability testing is such a valuable evaluation method that it should take
place throughout the lifecycle of a product, particularly in the early stages before the interface
is even designed.
What do you think are the main unanswered questions or challenges related to this
field of work?
One of the main challenges of usability testing is promoting a culture in your organization
where management and staff recognize the need for it and understand the value of it. Another
challenge is getting usability testing incorporated into the entire product development lifecycle,
especially in the early phases. Managers and software developers are anxious to produce
deliverables as quickly as possible and may overlook the importance of the principle to test
early and to test often in order to save valuable time and resources later on.
3