Workshop supported by EFMI WG Eval working group
Towards an International Minimum Dataset
for Monitoring National Health
Information System Implementations
Hannele HYPPÖNENa, Elske AMMENWERTHb, Christian NØHRc,
Arild FAXVAAGd , Åke WALLDIUSe
a
Information Department, National Institute for Health and Welfare, Helsinki, Finland
b
University for Health Sciences, Medical Informatics and Technology, Hall in Tyrol,
Austria
c
Aalborg University, Department of Development and Planning, Aalborg, Denmark
d
Norwegian Research Centre for Electronic Patient Records, Trondheim, Norway
e
Centre for User Oriented IT Design, Royal Institute of Technology, Stockholm, Sweden
Abstract. Differing measures used to monitor eHealth implementations makes it
difficult to compare and cumulate results. This EFMI WG Eval -supported workshop draws from lessons learnt to work towards a minimum international dataset
for monitoring eHealth implementations. The outcome is a first draft set of joint
indicators and a proposal for measures needed in order for future collaboration.
The results will benefit policy makers, developers and researchers in pursuit of evidence-based management of national eHealth systems.
Keywords. eHealth policy, Implementation, Evaluation
1. Background
A workshop was arranged in MIE 2009 on core issues and challenges in large-scale
evaluation for supporting system development, implementation and positive impacts
[1]. The work was based on UK, Canada and Australia national evaluation frameworks.
Most EU Member States are now becoming aware of an urgent need for (continuous)
evaluation activities, to better control policy progress and learn from challenges and
experiences [9]. Documented frameworks defining the needed evidence for management of NHIS in different stages of implementation are still few.
The discussions in Sarajevo helped in refining the Finnish evaluation methodology and piloting it in practice [2]. In Medinfo 2010 a panel was arranged on monitoring
the effects of health information systems [3], where an idea was presented to generate
some level of international agreement on the 'minimum data set' for monitoring Information systems implementation in order to facilitate international comparability. The
suggestion was supported, and in the conference some attendants agreed to form an
informal network to start working on the idea. This workshop is based on this idea,
drawing on the work already done in different countries as a basis for collaboration.
2. Goal and outcomes of the workshop
The goal of the workshop is to work on differences, similarities and needs for a minimum international dataset for monitoring progress and effects of eHealth System Implementation. The anticipated outcome is a first draft set of formative and summative
indicators and a plan for further measures needed.
3. Outline of the workshop and speakers
Dimensions and categories of evidence to support decisions in the national eHealth
system implementation and follow-up are presented as reference points for discussion
of the basis of future work. The outline is:
o National examples (H. Hyppönen, C. Nohr, Å.Walldius, A. Faxvaag)
o Group work + Discussion (Moderator E. Ammenwerth)
o Comparison and feasibility of the indicators (Moderator E. Ammenwerth)
o Future work and conclusions
Addressed audience: The workshop is intended for those interested and involved
in large scale eHealth research, development and use: policy makers and eHealth authorities, developers and researchers of national eHealth systems in pursuit of national
evaluation activities in different phases of the system life cycle.
Scientific challenges discussed: 1) What are the prerequisites for a minimum international dataset for providing information for development and implementation
decisions, 2) What forums of collaboration are needed to create an internationally and
nationally useful dataset.
Educational goals of the workshop: Participants will learn about planning and
carrying out large scale national monitoring and evaluation activities. By reflecting the
indicators against the GEP-HI guidelines the workshop participants will also deepen
their understanding about the guidelines.
4. Examples of national data collection, results and their use
The Finnish dataset development and information categories (Hannele Hyppönen)
The Finnish NHIS evaluation model is based on national-level evaluation programs in
UK, Canada and Australia. DeLone & McLean model on IS success was a basis was
extended with contextual factors using the model of network of activity systems with
formative elements using the human centred IS development model [4]1. This conceptual model was used to extract dimensions for monitoring NHIS progress and effects.
Outcomes of national results using the model [e.g. 2, 6] will be presented.
1
The contextual framework is based on activity theory and the cultural-historical approach, drawing also from sociology of technology
Danish experiences on monitoring HIS implementation (Christian Nohr)
The Danish EHR-observatory has monitored the HIS implementation from 1999 to
2006. Basic measurements of EHR coverage have been obtained in six consecutive
years [4]. Since 2006 regular monitoring has been absent, but public institutions and
private consultants have issued a number of status reports. There is however no consistent measures or datasets to enable comparison or progress evaluation. The status
reports shows importance of independence of monitoring activities from stakeholder
interest. The university based e-health observatory has launched several new activities
to monitor progress focusing on clinicians and citizens specific use of e-health services.
Swedish HIS monitoring experiences (Åke Walldius)
In Sweden ten years of experiences from the UsersAward project in which questionnaires and have been combined in domain specific longitudinal user satisfaction surveys. User experience, satisfaction, usability and design approach – are reflected upon
in the surveys. The experience shows that for surveys to attain impact through actionable directives for design, quantitative data from all relevant stakeholders' have to be
combined with qualitative data about user experience".
Norwegian HIS monitoring experiences (Arild Faxvaag)
Norway
began
monitoring
HIS
implementations
in
2008
(ref
http://www.ncbi.nlm.nih.gov/pubmed/20841713). Many questions were adapted from
the Danish EHR observatory study. A new survey was conducted in 2010. In this second study we collected more data on the experiences of clinician users, made a stakeholder analysis and presented results with the perspectives of the various stakeholders.
Some of the results are to be presented during the workshop. [7]
References
[1]
[2]
[3]
[4]
[5]
[6]
[7]
Hyppönen H, Doupi P, Hämäläinen P, Komulainen J, Nykänen P, Suomi R.Planning for national health
information system evaluation. Stud Health Technol Inform. 2009;150:972-6.
Hyppönen H, Viitanen J, Reponen J, Doupi P, Jormanainen V, Lääveri T, Vänskä J, Winblad I, Hämäläinen P. Large-scale eHealth Systems: Providing Information to Support Evidence-based Management.
eTELEMED 2011: The Third International Conference on eHealth, Telemedicine, and Social Medicine.
February 23-28, 2011 - Gosier, Guadeloupe, France
Nohr C, Faxvaag A, Lau F. Monitoring the effects of health information systems. Panel arranged at
Medinfo 2010 (Conference Programme, p. 44).
Hyppönen, H."eHealth Services and Technology: Challenges for Co-Development". Human Technology, Vol. 3, Nr 2, May 2007, pp. 188-213.
Hämäläinen, P., Reponen, J., and Winblad. I. "eHealth of Finland: Check point 2008”. Reports 37/2008,
National Institute for Health and Welfare, Helsinki, Finland.
Nøhr, C, Andersen, SK, Bernstein, K, Bruun-Rasmussen, M & Vingtoft, S 2007, 'Diffusion of Electronic Health Records: six Years of Empirical Data', in KA Kuhn, JR Warren & T Leong (eds), P. Hämäläinen, J. Reponen and I.Winblad. "eHealth of Finland: Check point 20082. Reports 37/2008, National
Institute for Health and Welfare, Helsinki, Finland.
Heimly V, Grimsmo A, Henningsen TP, Faxvaag A. Diffusion and use of Electronic Health Record
systems in Norway. Stud Health Technol Inform. 2010;160(Pt 1):381-5.