University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Evidence Based EHealth
Frances S Mair
President Telemedicine & E-Health
Section, RSM;
Professor of Primary Care Research
University of Liverpool
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Enthusiasm for New
Technologies in Health Care
Delivery


The possibilities that new technologies
present to health care systems; practitioners,
and carers and patients are considerable.
Policymakers have been quick to recognise
the possibilities of new technologies in the
health service.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
The Era of Evidence Based
Medicine
NICE - promotes clinical and cost
effective practice through its
technology appraisals, clinical
guidelines and audit tools.
 Cochrane and Systematic Reviews

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Is there Evidence to Support the
Widespread Implementation of
E-Health ?
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
“A descriptive feast
but an evaluative
famine”
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Systematic Reviews
Telehealthcare : satisfaction,
efficacy, and cost effectiveness
 Online peer to peer interaction
 Effects of computers on primary
care consultations

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Telehealthcare and
Patient Satisfaction
Systematic review of studies conducted
world-wide and published between 1966
and 1998.
Mair FS, Whitten P. BMJ 2000;320: 1517-1520.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Patient Satisfaction
Studies - Numbers
32 studies identified.
Sample sizes:
 20 or less (10);
 100 or less (14);
 greater than 100 (7);
 not specified (1).
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Patient Satisfaction
Studies - Trial Designs
RCT (1)
 random patient selection (2)
 case control design (1)
 selection criteria not specified or
participants represented
consecutive referrals, convenience
samples, or volunteers (28).

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Patient Satisfaction
Studies - Study Methods
Simple survey instruments (26)
 Exact methods not specified (5)
 Qualitative methods (1)

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Conclusions of Systematic
Review
Methodological deficiencies (low
sample sizes, context and study
designs) of the published research
limits the generalisability of the
findings.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Is Telehealthcare Cost
Effective?
Systematic review of studies conducted
world-wide and published between
1990 and 1999.
Mair et al. J Telemed and Telecare 2000;6 (Suppl
1):38-40.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Inclusion Criteria
Papers describing original research
concerning the use of telemedicine
in the provision of direct patient
care
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Results




83 papers identified initially
39 met eligibility criteria (4 excluded
due to multiple publication)
46% (16/35) of studies profoundly
flawed
54% (19/35) studies subjected to full
checklist review.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Study Design
RCT - 1/19 (5%)
 Random patient selection - 1/19
(5%)
 In general, methods very poorly
described.

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Appropriate Costs And
Benefits Considered?
None of the economic analyses
were “future proof.”
 The perspective and measurement
of both costs and benefits was, for
the most part, neither transparent
nor comprehensive.

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Analysis Appropriate to the
Local Environment?
4/19 based in US prisons
 11/19 were of virtually no
relevance to the NHS
 The remainder were of only limited
value to the NHS context

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
CONCLUSIONS

Existing literature is limited

Serious methodological flaws identified

Cost effectiveness of telemedicine
remains unanswered
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Is Telehealthcare Effective as An
Alternative to Face to Face
Care?
Currell et al. Telemedicine Vs. face
to face patient care: effects on
professional practice and health
care outcomes (Cochrane Review).
In: The Cochrane Library, Issue 2, 2000. Oxford.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Criteria
Studies which compare the
provision of patient care face to
face with care given using
telecommunications technologies,
in which at least two
communications media are used
interactively. (1966-1999)
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Results
7 trials included:
 telemedicine in an A+E dept (1)
 video-consultations between
primary care and hospital
outpatients (1)
 home telecare or patient self
monitoring of chronic disease (5)
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
What did these studies show?
No detrimental effects.
 Did not show unequivocal benefits
and the findings did not constitute
evidence of safety.
 No studies included any formal
economic analysis.

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Conclusions
Telemedicine is feasible
 Little evidence of clinical benefits
 No analysable data about cost
effectiveness of telemedicine
 Demonstrates the need for
FURTHER RESEARCH

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Systematic Review of Online
Peer to Peer Interactions

To compile and evaluate the evidence
on the effects on health and social
outcomes of computer based peer to
peer communities and electronic self
support groups, used by people to
discuss health related issues remotely.
Eysenbach et al. BMJ 2004;328:1166 (15 May),
doi:10.1136/bmj.328.7449.1166
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Results





45 papers (38 studies)
Only 6 evaluated "pure" peer to peer
communities.
Outcomes measured most often were
depression and social support
measures.
Most studies did not show an effect.
No evidence of virtual communities
harming people.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Conclusions


No robust evidence exists of benefits of
consumer led peer to peer communities.
Research required to evaluate under
which conditions and for whom
electronic support groups are effective
and how effectiveness in delivering
social support electronically can be
maximised.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Systematic Review of Effects of
Computers on Primary Care
Consultations
Systematic review of world literature from
1980 to 1997.



61 studies of effects of computers on
practitioners' performance
17 evaluated impact on patient
outcomes
20 studied practitioners' or patients'
attitudes.
University of Liverpool
Department of Primary Care



Findings
Mersey Primary Care R&D Consortium
Can improve practitioner performance,
particularly for health promotion
interventions.
Possibly at expense of patient initiated
activities, making many practitioners
suspicious of the negative impact on
relationships with patients.
Dearth of evidence evaluating effects on
patient outcomes.
Mitchell and Sullivan. BMJ 2001;322:279-282
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Prescribing Safety Features of
GP Computer Systems
Study looking at the potential safety
features of GP Computer Systems for
prescribing, such as contraindication
alerts
Fernando et al. BMJ, May 2004; 328: 1171 - 1172.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Results


The safety features of computing
systems currently in use in about 75%
of UK practices have clinically important
deficiencies.
All may fail to warn in a situation when
a warning is expected, thus potentially
creating a health hazard to patients.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Evidence Base is Poor


Systematic reviews suggest a degree of
scepticism warranted.
A poor evidence base is not an
insurmountable obstacle to normalisation
into service, as the introduction and
routinisation of many other ways of
delivering care has shown.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Many Claims Less Real
Evidence
“Many ehealth sites report huge numbers
of users and testimonials of their
benefits. Unfortunately, similar
testimonials can be found for snake oil,
and pornography sites boast large
numbers of hits.”
(Gustafson & Wyatt. BMJ 2004;328:1150 (15 May),
doi:10.1136/bmj.328.7449.1150)
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Should the randomised controlled
trial (RCT) be seen as the "gold
standard" of medical research and
dominate e-health research?
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
RCTs
Joint teleconsultations (virtual
outreach) vs. standard outpatient
appointments for patients referred by
their GP for a specialist opinion: a
randomised trial.
Lancet. 2002 Jun 8;359(9322):1961-8.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Results
FU appointments
  in pt satisfaction
  in tests and investigations
 More expensive

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Conclusions

Efficient operation of such services
will require appropriate selection of
patients, significant service
reorganisation, and provision of
logistical support
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Integrating Evaluation &
Development of
E-health Services


Studies of 7 telehealthcare evaluation
projects (4 RCTs; 3 pragmatic service
evaluations).
Projects spanned a range of specialties—
dermatology, psychiatry, respiratory
medicine, cardiology, and oncology.
Finch, May, Mair et al. BMJ 2003;327:12051209 (22 November), doi:10.1136/bmj.327.7425.1205
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Integrating Evaluation and
Service Development
Key problems :
(1) operationalising clinical practice,
(2) managing conflict between
evaluation and service provision,
(3) making sense of the study
findings.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Resistance to Change


A major part of the conflict around
service provision and evaluation
concerned the management of risk.
For clinicians the introduction of ehealth systems and its evaluation
highlighted the possibility of increased
risk from what they perceived as new
forms of practice.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Evaluation is Complex



Often underestimated.
Distinction between quantitative
outcomes and the workability of the
system.
Need pragmatic approaches to
evaluation, that permit both quantitative
and qualitative methods to improve the
quality and relevance of research.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
“If you're doing HTA in the rather broader sense
[than clinical trials] [the studies are] additionally
complicated. You cannot usually do a randomised
trial, you have to do some other sort of design, you
have to create your own measures. You can't just be
satisfied with, `Oh it's all right, mortality is the
measure for a cancer trial,' it's a key measure, …….
you've got to create measures. And I think if you add
to that the problems of doing informatics research in
general,……..I think you've got a whole series of
other issues to do with attempts to apply clinical
paradigms to informatics research."—Clinical leader,
site 4
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Workability
Workability, must be given greater
attention in the design of evaluation
studies in order to improve both the
quality of such research and its
relevance for clinical practice.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Flexibility Essential

Participants in studies applying RCT
study designs found that trying to
impose sufficient constraint on the
system for the purpose of
measurement conflicted with the
dynamic nature of the health
service environment, where some
flexibility is necessary.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Benefits of non RCT designs

Evaluators who had adopted nonrandomised designs felt they were
producing results they could use,
even though they too experienced
problems with integrating
telehealthcare systems into existing
practice.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Different Types of
Knowledge?
Experimental quantitative knowledge
about outcomes
 Experiential qualitative knowledge
about workability

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Quantitative knowledge
• Regarded as having higher status
• Easier for researchers to publish?
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Qualitative Knowledge
• Seen as more useful for judging the utility
of the system in practice
• Inform service development.
• Important implications for commissioning
processes, which must permit greater
acknowledgement of the practical value of
research methods that produce knowledge
about processes rather than healthcare
outcomes.
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
“Clinicians placed in an environment where they
are busy and constantly interrupted by colleagues
or synchronous technologies such as the telephone,
pager, and email, are "designed" to produce error
and inefficiency. Designers of busy clinical
services thus need to factor in human cognitive
limits and the work loads generated by other
services over which they have no control”.
Coiera. BMJ 2004;328:1197-1199 (15 May),
doi:10.1136/bmj.328.7449.1197
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
RCTs not always appropriate –
Example -Evaluating ICRS
Identify, describe, understand and
disseminate knowledge about the
contexts of ICRS application and
normalization, where new services
and systems are implemented and
practical problems identified and
solved in the specific operational
contexts of a range of care settings
University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Conclusions
E-Health – great potential…little
good evidence.
 Evaluation of such services are
complex and require flexibility.
 RCTs can be used but are not
always appropriate and will not
themselves provide all the answers.

University of Liverpool
Department of Primary Care
Mersey Primary Care R&D Consortium
Those funding e-health
initiatives should incorporate
evaluation.
Evaluation is expensive but
necessary!