BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Long-term Usability Testing for
Public Health Information Technology:
BioSense 2.0
Amanda Recker
Jamie Pina, MSPH, PhD
Barbara L. Massoudi, MPH, PhD
RTI International
2013 International Symposium on Human Factors and
Ergonomics in Health Care: Advancing the Cause
March 11, 2013
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
BioSense 1.0: Web-based syndromic surveillance
• Mandated in the Public Health Security and Bioterrorism (BT)
Preparedness and Response Act of 2002
• Nationwide integrated system for early detection and
assessment of potential BT-related illness
• 2003 Funding provided by Congress to CDC
• Development of BioSense infrastructure started, initial focus on:
– VA and DoD
– Direct reporting to CDC of detailed clinical data by civilian hospitals
• Began soliciting more limited data from health departments
(HDs) that had already established automated systems for EDbased syndromic surveillance
– By 2007, 8 state/local HDs connected
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Recommendations from Prior Evaluations
• Strengthen state and local public health engagement
– Enhance state/local HD syndromic surveillance capacity
– Increase participation of state/local HD syndromic surveillance
systems (improve coverage)
– Share data with HDs from hospitals reporting directly to CDC
– Share governance with public health community
•
•
•
•
•
Leverage investments in electronic health records (EHRs)
GAO, 2008: Adopt an “open, distributed computing model”
Improve utility of the data and data sources
Preparedness role: Greater “all hazards” emphasis
Expand uses for broader spectrum of public health concerns
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
BioSense 2.0: Timeline
• June 2010: Redesign begins
• November 2011: Opened for business
• November 2011 – June 2013: Onboarding new jurisdictions
– 35 jurisdictions signed the Data Use Agreement (DUA)
– 17 fully onboarded
• April 2012: Retired BioSense 1.0
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
BioSense 2.0: Approach
• Shift from a need-to-know to a need-to-share and co-create
approach
• User-centered design
– Stakeholders engaged in every step of the redesign
– HDs fully control “their data” at the level of granularity they choose
– More options for data sharing with other jurisdictions and CDC
• Alignment with ONC and Meaningful Use
– Agreed-upon core syndromic surveillance data elements
– Collaborations with public health professional associations
– Funding to states: Meaningful Use syndromic surveillance adoption,
build capacity, join BioSense 2.0
• Cloud technology: distributed, easy to adopt, cost effective,
secure
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Application Home Page
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Encephalitis, Meningitis, WNV = CNS Inflammatory Disease
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Why Long-term Usability Testing?
•
•
•
•
•
Long user-centered design lifecycle
Expectation management
Stakeholder ownership
BioSense 2.0 continuously changing and growing
Longitudinal usability testing
– User satisfaction
– Efficiency
• Functions not changing: building a query, viewing results,
analyzing the data, sending and saving information
• Two approaches to testing
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
1. User-Centered Design
•
•
•
•
Qualitative data collection methods
The user knows best
Test the right participants
Broad range of public health professionals
– Public health generalist – less sophisticated users
– Syndromic surveillance epidemiologist – more sophisticated users
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
2. Activity-Centered Design
• Quantitative data collection methods
• Behavior vs. opinions
• Based on empirical data
– Time-on-task analysis
– Mouse-click analysis
– Pathway analysis
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Basic Activities in Interaction Design
1.
2.
3.
4.
(Rogers, Sharp, & Preece, 2011)
(Rogers, Sharp, & Preece, 2011)
Establish requirements
Design alternatives
Develop prototype
Conduct evaluation
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
How to Choose Users
• Interact directly with the system
– Epidemiologists, state and local public health professionals
• Manage direct users – decision-makers
– Public health directors
• Use similar syndromic applications
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Generating Design Alternatives
• User-centered design
–
–
–
–
Open question response
Focus groups
Expectation testing
SUS
• Expert evaluation
– Usability heuristics (Nielsen)
• Activity-centered design
– Morae software
– Scenarios and tasks
– Closed/open question
response
– Time on tasks
– Mouse-click analysis
– Pathway analysis
– Critical incidents
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Scenarios and Tasks
Scenario: “Over dinner at [a public health conference] an argument
has erupted, but luckily as a BioSense 2.0 user you can settle this
dispute. Health authorities in Virginia suspect that the flu season
was more severe than it was in Michigan.”
Task: “Please determine which state, Virginia or Michigan, had more
cases of influenza-like illness (ILI) starting in October 1, 2010 through
March 1, 2011.”
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Pathway Efficiency Analysis
• Time on task
• Mouse-click analysis
• Pathway analysis
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Design Alternatives
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
How to Choose Among These Alternatives?
• If one person says something is a problem, do you change the
design?
• Resolved conflicting alternatives
• Conducted feasibility analysis
• In the end…relied on face validity
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
What Happens After Design Changes?
•
•
•
•
User training through webinars and videos
Expert user testing
Focus group sessions
Continually comparing SUS scores
BioSense
Public Health Surveillance Through Collaboration
https://biosen.se
Future Activities
•
•
•
•
Incorporate eye tracking into testing protocols
Conduct on-site testing and evaluation
Task analyses of routine versus event surveillance
Information models for routine and event surveillance