Error
?
Evaluation of
electronic medical records
- a clinical task perspective
Presentation of thesis
Hallvard Lærum
Norwegian University of
Science and Technology,
Trondheim
The Kvalis Project
• Quality assurance of electronic
medical records in hospitals
• Funded by The Research Council of
Norway
• Interdisciplinary, three PhD fellows
– Sociology: Gro Underland
– Informatics: Gunnar Ellingsen
– Medicine: Hallvard Lærum
Evaluation of electronic medical records
Descriptive aspects
What is the level of functionality of current EMR
systems?
Where are they found?
Explorative aspects
How may the effects of various EMR systems on
physicians’ clinical practice in various hospitals be
evaluated?
Investigations
How does the method perform in various contexts?
Reliability and validity studies
Establishing contact with
research centers
internationally
Keeping up a website
Department of Medical Informatics &
Clinical Epidemiology,
Oregon Health and Science University
Building a
research
network
Establishing contact
with hospitals
Establishing contact with
EMR vendors
Arranging monthly seminars
The 3rd Scandinavian University Course
in medical informatics
Arranging meetings
Descriptive aspects of the thesis
EMR systems are very common in Norwegian
hospitals (80% of hospital beds as of 2002)
No systematic
evaluations of
Norwegian EMRs
published by 1999
The systems are essentially document-based,
covering the narrative parts of the medical
records, lab data and radiology reports.
Explorative aspects of the thesis 1
Considerations for development of an
evaluation method
• Applicable to various EMR systems and hospitals
– Not tied to a single system or hospital
• Carried out using limited resources
– Fast results
• Used in operational reviews, incorporating
organizational issues
– Laboratory simulations less feasible
• Involve measurement close to the site of impact
– Involving how clinical work is done
Explorative aspects 2
The first task inventory
Diagnostics and Treatment
Planning
Other
Define
problem
Get existing
plan
General
logistics
Get medical
knowledge
Get info
about resources
Communication
Get patient
data
Negotiate
with patient
Administrative
Create new
information
Inform
Patient
Research
Inform
Patient
Document
plan
Quality
Control
Negotiate
with patient
Teaching
Execute
intervention
Patient's relation
to society
Document
activity
Other
• Direct observations, 40 h
–
–
–
–
20 h St.Olavs Hospital, 20h RiTø (Gunnar E.)
7h transcribed verbatim
Basis for hierarchical task analysis
Noticed that the PCs were not much in use
• The first task inventory: 104 tasks
– Too large for practical use
– Task wording rather technical
– Covert tasks not included (e.g. cognitive tasks)
Video observations – Flow of information
• Information seeking behaviour
– Looking for incidents where needed
information was not found
• Tagging of video recordings
• Stimulated recall* not sucessful
– Too time-consuming (and boring!) for
an average clinician
– Recruitment problems
• Prohibitively resource intensive
– 9 patients, 4.5 h video recordings, two
physicians
• Noticed that PCs were not
much in use
* Kushniruk & Patel 1995
OntoLog Courtesy of Jon Heggland, IDI NTNU
!
Simplification of the task list
• Task list to be used in questionnaires and
interview guides
• Considerations for the process
–
–
–
–
–
–
–
Emphasis on information-related tasks essential for patient care
Wording adapted to clinicians’ way of speaking
Covert tasks were included
Tasks suitable for most specialties were retained
Tasks representing negligible work were deleted
Gorman’s five information needs were taken into account
Tasks should be supported by functionality found in current
EMR systems or in those expected in the near future
• Multiple iterations by work group (2 physicians, 2
informaticians)
A list of relevant tasks for the clinician
Norwegian
English translation
1. Få oversikt over pasientens problemstilling
Review the patient's problems
2. Lete frem enkeltopplysninger fra pasientjournalen
Seek out specific information from patient records
3. Følge resultatene av en bestemt prøve eller us. over tid
Follow the results of a test or investigation over time
4. Slå opp svar på nye prøver eller undersøkelser
Obtain the results from new tests or investigations
5. Føre daglige og/eller forefallende journalnotater
Enter daily notes
6. Få tak i oppl. om prosedyre for utredning eller behandling
Obtain information on investigation or treatment procedures
7. Få svar på spørsmål om generell med.-faglig kunnskap
Answer questions concerning general medical knowledge
8. Få ut samledata for en gruppe pasienter
Produce data reviews for specific patient groups
9. Rekvirere klin-kjem. lab.analyser
Order clinical biochemical laboratory analyses
10. Slå opp svar på klin.-kjem. lab.analyser
Obtain the results from clinical biochemical laboratory analyses
11. Rekvirere rtg.us., UL eller CT
Order X-ray, ultrasound or CT investigations
12. Slå opp svar på rtg., UL eller CT
Obtain the results from X-ray, ultrasound, or CT investigations
13. Rekvirere. andre supplerende undersøkelser
Order other supplementary investigations
14. Slå opp svar på andre supplerende undersøkelser
Obtain the results from other supplemental investigations
15. Henvise pas. til annen avdeling eller spesialist
Refer the patient to other departments or specialists
16. Ordinere behandling direkte (med., op. eller annen)
Order treatment directly (e.g. medicines, operations etc.)
17. Skrive resept
Write prescriptions
18. Skrive sykmelding
Complete sick-leave forms
19. Samle inn pasientopplysninger til ulike legeerklæringer
Collect patient data for various medical declarations
20. Gi skriftlig individuell info til pasienten
Give written individual information to patients
21. Gi skriftlig generell medisinsk-faglig info til pasienten
Give written general information to patients about the illness
22. Samle inn opplysninger til epikrise
Collect patient information for discharge reports
23. Kontrollere og signere ferdig skrevne diktater
Check and sign typed dictations
The first Questionnaire was compiled
• Translated sections (two-way translation validation involving native
english speaking translator)
– End User Computing Satisfaction scale (Doll & Torkzadeh)
– Global user satisfaction questions (SGUS, Anderson & Aydin)
– Computer literacy (Brown & Coney)
• New sections
– Frequency of PC /EMR use for the 23 clinical tasks
– Basic availability of computers
Pilot study (n=22)
Investigations in this thesis
• National survey, 2001
• Local survey in Arendal, 2002
• Validation and reliability studies,
2003
The national survey Feb 2001
• Survey
– 314 physicians in 32 hospital units in 19
hospitals
– 227 (72.3%) responded after one reminder
(135 respondents were reminded)
• Telephone interviews with key IT
personnel in each hospital
– What clinical tasks could be performed using
the EMR system?
– A set of minimal functionality requirements,
incl. requirements for integration of external
systems
The section covering PC/EMR use
Missing
responses!
Implemented functions
DIPS
General
Medical knowledge
Aggregated data
Supplementary
investigations
Actions
Information to patient
Communication and
verification
DocuLive
Infomedix
1. Review the patient's problems
2. Seek out specific information from patient records
3. Follow the results of a test or investigation over time
4. Obtain the results from new tests or investigations
5. Enter daily notes
6. Obtain info on investigation or treatment procedures
7. Answer questions concerning general medical knowledge
8. Produce data reviews for specific patient groups
9. Order clinical biochemical laboratory analyses
10. Obtain the results from clin. biochemical lab. analyses
11. Order X-ray, ultrasound or CT investigations
12. Obtain the results from X-ray, ultrasound or CT inv.
13. Order other supplementary investigations
14. Obtain the results from other supplementary inv.
15. Refer the patient to other departments or specialists
16. Order treatment directly (medical, surgery. or other)
17. Write prescriptions
18. Complete sick-leave forms
19. Collect patient info for various medical declarations
20. Give written specific information to patients
21. Give written general information to patients
22. Collect patient information for discharge reports
23. Check and sign typed dictations
0%
50 %
100 %
50 %
100 %
50 %
Percent of respondents offered minimal
functionality for the given task
• There is a considerable difference in implemented functions
between systems
100 %
Use and implementation of EMR systems
A lot of implemented functionality appears not to be used
DIPS
General
Medical knowledge
Aggregated data
Supplementary investigations
Actions
Patient info
Communication and flow of
information
DocuLive
Infomedix
1. Review the patient's problems
2. Seek out specific information from patient records
3. Follow the results of a test or investigation over time
4. Obtain the results from new tests or investigations
5. Enter daily notes
6. Obtain info on investigation or treatment procedures
7. Answer questions concerning general medical knowledge
8. Produce data reviews for specific patient groups
9. Order clinical biochemical laboratory analyses
10. Obtain the results from clin. biochemical lab. analyses
11. Order X-ray, ultrasound or CT investigations
12. Obtain the results from X-ray, ultrasound or CT inv.
13. Order other supplementary investigations
14. Obtain the results from other supplementary inv.
15. Refer the patient to other departments or specialists
16. Order treatment directly (medical, surgery. or other)
17. Write prescriptions
18. Complete sick-leave forms
19. Collect patient info for various medical declarations
20. Give written specific information to patients
21. Give written general information to patients
22. Collect patient information for discharge reports
23. Check and sign typed dictations
0%
Cut-off: Respondents
answering ”half of the
time” or better are users
50 %
100 %
50 %
100 %
50 %
100 %
Percent of respondents offered functionality/ using it
Use EMR Use both EMR
and other
software
Type of
program not
stated
Implementation
Use and implementation of EMR systems
Some physicians enter daily notes themselves
DIPS
General
Medical knowledge
Aggregated data
Supplementary investigations
Actions
Patient info
Communication and flow of
information
DocuLive
Infomedix
1. Review the patient's problems
2. Seek out specific information from patient records
3. Follow the results of a test or investigation over time
4. Obtain the results from new tests or investigations
5. Enter daily notes (i.e. progress notes)
6. Obtain info on investigation or treatment procedures
7. Answer questions concerning general medical knowledge
8. Produce data reviews for specific patient groups
9. Order clinical biochemical laboratory analyses
10. Obtain the results from clin. biochemical lab. analyses
11. Order X-ray, ultrasound or CT investigations
12. Obtain the results from X-ray, ultrasound or CT inv.
13. Order other supplementary investigations
14. Obtain the results from other supplementary inv.
15. Refer the patient to other departments or specialists
16. Order treatment directly (medical, surgery. or other)
17. Write prescriptions
18. Complete sick-leave forms
19. Collect patient info for various medical declarations
20. Give written specific information to patients
21. Give written general information to patients
22. Collect patient information for discharge reports
23. Check and sign typed dictations
0%
50 %
100 %
50 %
100 %
50 %
100 %
Percent of respondents offered functionality/ using it
Use EMR Use both EMR
and other
software
Type of
program not
stated
Implementation
Use and implementation of EMR systems
Obtaining lab data (and other results) is popular, order entry is not
DIPS
General
Medical knowledge
Aggregated data
Supplementary investigations
Actions
Patient info
Communication and flow of
information
DocuLive
Infomedix
1. Review the patient's problems
2. Seek out specific information from patient records
3. Follow the results of a test or investigation over time
4. Obtain the results from new tests or investigations
5. Enter daily notes
6. Obtain info on investigation or treatment procedures
7. Answer questions concerning general medical knowledge
8. Produce data reviews for specific patient groups
9. Order clinical biochemical laboratory analyses
10. Obtain the results from clin. biochemical lab. analyses
11. Order X-ray, ultrasound or CT investigations
12. Obtain the results from X-ray, ultrasound or CT inv.
13. Order other supplementary investigations
14. Obtain the results from other supplementary inv.
15. Refer the patient to other departments or specialists
16. Order treatment directly (medical, surgery. or other)
17. Write prescriptions
18. Complete sick-leave forms
19. Collect patient info for various medical declarations
20. Give written specific information to patients
21. Give written general information to patients
22. Collect patient information for discharge reports
23. Check and sign typed dictations
0%
50 %
100 %
50 %
100 %
50 %
100 %
Percent of respondents offered functionality/ using it
Use EMR Use both EMR
and other
software
Type of
program not
stated
Implementation
Use and implementation of EMR systems
Big surprise: Nobody is using the EMR system to write prescriptions or
complete sick-leave forms
DIPS
General
Medical knowledge
Aggregated data
Supplementary investigations
Actions
Patient info
Communication and flow of
information
DocuLive
Infomedix
1. Review the patient's problems
2. Seek out specific information from patient records
3. Follow the results of a test or investigation over time
4. Obtain the results from new tests or investigations
5. Enter daily notes
6. Obtain info on investigation or treatment procedures
7. Answer questions concerning general medical knowledge
8. Produce data reviews for specific patient groups
9. Order clinical biochemical laboratory analyses
10. Obtain the results from clin. biochemical lab. analyses
11. Order X-ray, ultrasound or CT investigations
12. Obtain the results from X-ray, ultrasound or CT inv.
13. Order other supplementary investigations
14. Obtain the results from other supplementary inv.
15. Refer the patient to other departments or specialists
16. Order treatment directly (medical, surgery. or other)
17. Write prescriptions
18. Complete sick-leave forms
19. Collect patient info for various medical declarations
20. Give written specific information to patients
21. Give written general information to patients
22. Collect patient information for discharge reports
23. Check and sign typed dictations
0%
50 %
100 %
50 %
100 %
50 %
100 %
Percent of respondents offered functionality/ using it
Use EMR Use both EMR
and other
software
Type of
program not
stated
Implementation
Why this lack of EMR use? – 1
Answers provided by the questionnaire is limited
Low computer literacy?
Con: The physicians had at least a basic knowledge of
computers (average score 72.2 ±1.6, 92% owned a computer)
Pro: Specific EMR system training may be needed
Lack of available computers?
Con: Some PCs were available: 93% a computer in their office,
97% had a computer available in other rooms used for clinical
work.
Pro: Clinicians’ work is not stationary. No hospitals have yet
implemented mobile computing, and no hospitals can afford a
PC in every room (personal communications)
I wish the EMR could
be more like WordStar
Where’s the PC
when I need it?
Why the lack of EMR use? - 2
Low usability of the EMR system as a whole?
Prescriptions: Navigating to the prescription module,
finding the correct medication, typing dosage,
administration route and package size and printing on the
correct type of paper may represent more work than doing
it by hand.
Sick-leave forms: Slow system response times (12-15 sec)
The EMR is not complete
Pro: Until all relevant clinical information is found in the
EMR, it will serve a secondary role.
Typing is a
Pro: The functionality of current EMR systems is limited
secretary’s job!
Resistance to new work roles?
Pro: The usage patterns found in the national study are
conform to traditional work roles.
Pro: Work role issues were the most prevalent theme in
the answers to the open-ended questions in the validity
study (Paper 4).
Follow-up studies in each hospital are necessary
I know my prescriptions by heart!
I’d like to find all
the information
Order entry is a
in one place
nurse’s job!
Local study of EMR system in Aust-Agder
Hospital, Arendal
(Paper 2 and 3)
• The paper-based medical record
is scanned and obliterated!
• Questionnaire rev.2
(versions for medical secretaries and nurses
developed separately)
– 70 of 80 physicians (88%)
– 79 of 85 medical secretaries (93%)
– 172 of 235 nurses (73%)
• Interviews
– 8-12 representatives of each profession,
0.5-2 h interviews
The EMR in Aust-Agder Hospital
2nd revision of the questionnaire
Physician’s use of the regular EMR at Aust-Agder Hospital
Much higher frequency of EMR use than in other hospitals having the same system
The physicians reported a less frequent use
of the scanned document images
Physicians: Change in ease of performing the clinical tasks
Most tasks related to information retrieval are reportedly
easier using the system, but 33% of internists found task 1 and 2 more difficult
User satisfaction of physicians
The physicians are relatively satisfied with the regular EMR, but not with the use of the scanned document images
Validation and reliability studies
3rd revision of questionnaire
• Test-retest reliability study
– 37 of 96 physicians (39%) from three hospitals having different EMR
systems completed the questionnaire two times. ”Test” questionnaire:
55.2% (52 of 96), ”Retest” questionnaire. 71% (37/52).
• Content validity study – interviews
– 10 physicians, 1 h interviews
– Relevance of tasks, estimation of accuracy of answers to task-oriented
questions, themes in open-ended questions.
• Criterion validity using data from local and national studies
– Criteria
•
•
•
General information retrieval vs. tasks involving information retrieval
Overall work performance vs.task performance
Task performance vs. user satisfaction
– Majority of tasks covered, median correlation coefficient 0.445-0.513
Test-retest reliability: Weighted kappa was generally high
Problematic tasks were related to functionality not available
locally, and problems in discering the EMR from other software
Themes appearing in the interviews
•
Work role issues (8/10 physicians) – resistance to doing ”clerikal” tasks
•
Various wording problems (7/10 physicians)
•
Questions regarding use of non-existent functionality (7/10 physicians)
•
Distinguishing EMR from other software or media (6/10 physicians)
– Tasks 10 and 19
– The third method would be the "ask-the-nurse" method. This is convenient, though,
then I may do other things. [In the future] It could be that it will be so easy to do it,
that I could do it myself…if it's really easy, a completely negligible task. But if it
takes some time..if I have to wait or something..then I feel that it should be a
medical secretary's task, at least in a hospital. (respiratory diseases)
– Tasks 16, 4 and 21
– I don't understand what you mean with "directly"…write orders on the [order entry
form], request or order an operation…one other [example] is requesting treatment
by physiotherapist (orthopedy)
– Task 3
– Some questions are difficult to answer, as we can't log on [to the EMR system] and
find results from X-ray investigations (plastic surgery)
– Task 4
– Is [the separate lab system] regarded as a part of [the EMR system]? (neurology)
Considerations of self-reported use
• Sources of error in self-reported use
–
–
–
–
–
Telescoping and other memory-related effects
Implicit expectations of the questionnaire
Willingness to respond
Strategic responses
Other
– There seem to be more reasons for the user to report too
high frequency of use than too low.
• Use of the EMR system may not necessarily
benefit the patient
– EMR system used the wrong way
– Unintended effects of using the EMR system
– Logical errors in the EMR system
Application of the task list and the questionnaire
• The questionnaire may be used for screening, focusing the
evaluation effort and providing a basis for further exploration
– Separating reported use from non-use
• The combination of EMR use and task performance may
identifiy problematic aspects of the EMR system.
• The questionnaire should always be combined with a qualitative
study to investigate the ”why” (and validate the findings)
Thank you!
Extra information
National study: Inclusion of hospitals
• Inclusion of hospitals
– The EMR should at least contain medical narratives (admission reports, progress
notes, discharge reports), directly or indirectly updated by and electronically
available to the physicians. (minimal functional requirement for task 1)
National study: Lessons learned in
this survey:
• Missing responses
– ”Use EMR/Other program” questions frequently not
answered (median 10%, IQR 5-15%)
• Two hospital units (8 physicians) had to be
excluded post hoc
– 1: Minimal requirements for EMR implementation not met
– 2: Two EMR systems in use simultaneously
• Error discovered one year after publication
– Task not supported after all in one hospital unit (11
physicians)
– Lab results not integrated in the EMR in this unit, available
in stand-alone system only.
– Correction sent to BMJ (paper 1) Nov 2002.
Local study: User satisfaction of physicians, nurses and medical secretaries
The medical secretaries are most satisfied, the physicians least satisfied with the system
Local study: Use of hospital information system for individual tasks
Local study: Results of interviews
• All professions found that the patient data were more
accessible when stored electronically.
• Physicians (internists) found searching in scanned
multiple documents time-consuming and difficult
• The medical secretaries found that generation,
handling, fetching and delivery of paper documents
and logistics of paper-based patient records had
diminished dramatically.
• Nurses were still using pen and paper documenting
their activities.
Changes in rev.3 of the questionnaire
• Added ”escape” choices in task-related
questions
– Task not relevant for me
– Task not possible using the local EMR system
Validation: Interviews 1: Accuracy
Discordant
interpretation
for 1/10 in task 6,
”Obtain
information on
investigation or
treatment
procedures”
validation: Interviews 2: Relevance of tasks
Task 8, ”Produce data
reviews for specific patient
groups” was not found
relevant by a majority of
the physicians.
(not part of one’s job/
infrequently performed)