Publication Bias in Health
Informatics:
Results of a survey
Elske Ammenwerth
Innsbruck, Austria
Nicolette de Keizer
Amsterdam, The Netherlands
UMIT
Aim


What percentage of IT evaluation
studies does not result in
internationally available publications?
What are the reasons for not
publishing the results of IT
evaluation studies
Methods - Survey

E-mail based survey:




742 email adresses of:




Which system, which aspect and how did you
evaluate?
Where was it published
What were reasons not to publish (6
predefined categories or free text)
EFMI/IMIA WG on assessment of HIS,
AMIA WG on evaluation,
Pubmed (paper 2001-2006 on evaluation of IT)
136 CIOs
Results



Response rate CIOs= 3.7% (5/136)
Response rate= 18.3% (136/742)
141 respondents:



118 valid information on IT evaluations
studies
19 no evaluation studies conducted
4 unclear/unusable answers
Results
Under-representation some non-Englishspeaking countries and commercial domain
other
25%
4%
6%
5%
USA
39%
7%
AUS
5%
DE
5%
CA
8%
NL
8%
UK
10%
78%
Academia
IT management
Industry
Governmental instit
unknown
Results

217 evaluation studies, 213 publications
120
103
100
2
4
9
80
11
0
82
60
40
77
20
20
12
0
international
publication
local publication
one
two
three
internal
publication
four
five
no publication
Total
Results
Source of publications of 103 evaluation studies
Only other type of
journal
13%
Both medical and
HI journal
3%
Only HI journal
30%
Only
medical,nursing,p
harmaceutical
journal
29%
Both HI journal
and proceedings
6%
Only HI
proceedings
19%
Results
40
37
35
33
30
25
20
“The results and systems are too specific to
interest audience in other countries”
“May
“The
publish
results
following
were not
validation.
all positive.
Probably
not The
[journal
system
name]
administrators
which seems
and
to be
“Overwhelmed with ongoing tasks to
closed
stakeholders
shop for those
of theoutside
systemofwould
the
implement
EMR.
Always
a
lack
of
time
“The
evaluations
were
for a
academic
not like
communities
to accept
the
” done
result.”
when
funding
is
coming
fromthe
outside.”
government
agency.
“The setup
(e.g. Once
amount
ofreports
“The
organization
I
work(ed)
in
wereinterviews)
delivered, no
activities
wasfurther
not robust
enough.”
prohibited
publication
(well,
not
were funded. The majority of authors
expressly,
but it was
made clear that
were
consultants,
not academics.
24
“Not subjected
tobe
our
such a publication
not
Publication
is not of would
interest
for
welcome).” university ethics process,
consultants.”
therefore not published.”
17
15
15
11
10
4
5
0
Not of global
interest
Publication
in
preparation
No time for
publication
Limited
scientific
quality
Political
corectness
Only internal
interest
Other
Discussion of methods




Strong academic bias
English-speaking countries bias
No guarantee that provided
information is complete or correct
Results only illustrative, not
representative for the IT community
in health informatics
Discussion of results



Only half of the studies
internationally published
Limited number of HI journals or
proceedings and large variation of
medical journals
Various reasons for non-publication
but no strong hints for publication
bias due to negative results
Recommendations to increase
published evaluation studies for EBHI

Raise number of conducted studies


Raise number of submitted papers


GEP-HI
Raise acceptance rate of submissions


Rewards for evaluation, budget, outside IT
project team
STARE-HI
Increase internationally available papers


Motivate authors to publish, EVALDB
(http://evaldb.umit.at )