TITLE:
Effectiveness of Teaching Evidence-Based Medicine to Undergraduate
Medical Students: A BEME Systematic Review
RUNNING HEAD:
Teaching EBM to Medical Students
AUTHORS:
Seyed-Foad Ahmadi, MD, MPH 1; Hamid R Baradaran, MD, PhD 2; Emad Ahmadi, MD 3.
1
I. Postdoctoral Fellow (previously) at Center for Educational Research in Medical
Sciences, Iran University of Medical Sciences, Tehran, Iran; II. Postdoctoral Fellow at
Department of Medicine, University of California Irvine, Irvine, CA, USA.
2
Associate Professor of Clinical Epidemiology/ Deputy of Research at Center for
Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran,
Iran.
3
I. Postdoctoral Fellow (previously) at Digestive Diseases Research Institute, Tehran
University of Medical Sciences, Tehran, Iran; II. Postdoctoral Fellow at Department of
Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA,
USA.
Ahmadi et al
2
Teaching EBM to Medical Students
CORRESPONDENCE:
Hamid R Baradaran
Associate Professor of Clinical Epidemiology/ Deputy of Research
Center for Educational Research in Medical Sciences
Iran University of Medical Sciences
Zip Code: 14496-14535
Tehran, Iran.
Tel: +98-21-86702216
Fax: +98-21-86702237
Email: baradaran.hr@iums.ac.ir
DECLARATION OF INTEREST:
The authors have no competing interest to declare.
Ahmadi et al
3
Teaching EBM to Medical Students
ABSTRACT
Background: Despite the widespread teaching of evidence-based medicine (EBM) to medical
students, the relevant literature has not been synthesized appropriately as to its value and
effectiveness.
Aim: To systematically review the literature regarding [l1]the impact of teaching EBM to medical
students on their EBM knowledge, attitudes, skills, and behaviors.
Methods: MEDLINE, SCOPUS, Web of science, ERIC, CINAHL, and Current Controlled
Trials up to May 2011 were searched; backward and forward reference checking of included and
relevant studies was also carried out. Two investigators independently extracted data and
assessed the quality of the studies.
Results: 10,111 potential studies were initially found, of which 27 were included in the review:
Six studies examined the effect of clinically integrated methods, of which five had a low quality
and the other one used no validated assessment tool. Twelve studies evaluated the effects of
seminars, workshops and short courses, of which eleven had a low quality and the other one
lacked a validated assessment tool. Six studies examined e-learning, of which five having a high
or acceptable quality reported e-learning to be as effective as traditional teaching in improving
knowledge, attitudes and skills. One robust study found problem-based learning less effective
compared to usual teaching. Two studies with high or moderate quality linked multicomponent
interventions to improved knowledge and attitudes. No included study assessed the long-term
effects of the teaching of EBM.
Conclusions: Our findings indicated that some EBM teaching strategies have the potential to
improve knowledge, attitudes and skills in undergraduate medical students, but the evidenced
Ahmadi et al
4
Teaching EBM to Medical Students
base does not demonstrate superiority of one method. There is no evidence[l2] demonstrating
transfer to clinical practice.
Keywords: Teaching, Evidence-Based Medicine, Undergraduate Medical Education, Outcome
Measure, Systematic Review.
Ahmadi et al
5
Teaching EBM to Medical Students
INTRODUCTION
Evidence-based medicine (EBM) is “the conscientious, explicit and judicious use of current best
evidence in making decisions about the care of individual patients” (Sackett et al., 1996). The
practice of EBM usually requires the following five steps: 1) Translating the uncertainties into
answerable questions (asking); 2) Searching for and retrieving evidence to answer the questions
(acquiring); 3) Critically appraising the evidence for validity and clinical importance (appraising); 4)
Applying the appraised evidence to inform the clinical decisions (applying); and 5) Evaluating the
performance in the pervious four steps (assessing) (Dawes et al., 2005).
The Teaching of EBM has become increasingly popular in both undergraduate and postgraduate
medical education programs worldwide (Crilly et al., 2009). EBM is now a component of the
foundation years training program in the UK, (Colleges., 2007) the focus of graduate assessment
in the USA (Stewart, 2001) and a requirement of practicing physicians in Canada (Frank et al.,
2005). However, there is limited robust-evidenced research that has examined the teaching
methods of EBM (Hatala and Guyatt, 2002).
EBM experts have systematically reviewed the literature regarding teaching EBM to
postgraduates (Coomarasamy and Khan, 2004, Flores-Mateo and Argimon, 2007) and allied
health professionals (Dizon et al., 2012), teaching critical appraisal (Parkes et al., 2001, Taylor et
al., 2000b, Norman and Shannon, 1998), assessing the effectiveness of journal clubs (Harris et
al., 2011, Ebbert et al., 2001), evaluation methods of EBM education (Shaneyfelt et al., 2006,
Walczak et al., 2010), and barriers to EBM application by residents (van Dijk et al., 2010).
However, the most effective methods for teaching EBM to undergraduate medical students have
remained unclear.
Ahmadi et al
6
Teaching EBM to Medical Students
Hence, the aim of this systematic review was to evaluate the effect of various EBM teaching
strategies on medical students’ knowledge, attitudes, skills, and behaviors. In addition, the
teaching of EBM is reported to be improved by breaking it into the steps of asking, acquiring (or
accessing), appraising, applying, plus an evaluating (or assessing) step (Del Mar et al., 2004).
Therefore, we also examined whether the educational interventions could improve the above
EBM steps.
Ahmadi et al
7
Teaching EBM to Medical Students
METHODS
Criteria for inclusion of studies:
We included the comparative studies i.e. randomized controlled trials, non-randomized
controlled trials, and self-controlled trials that: A) had recruited undergraduate medical students
(defined as medical school students who have not yet enrolled in the residency programs), B) had
carried out at least one educational intervention (defined as coordinated educational activity, of
any medium, duration or format) to teach EBM, and C) had objectively assessed the impact of
the intervention(s) on students’ knowledge, attitudes, skills, or behaviors using tests,
questionnaires, clinical performance, etc. Self-reported perceived knowledge, skills or behaviors
were not eligible since they are loosely connected to their objective measurements (Caspi et al.,
2006, Khan et al., 2001).
Identification and selection of studies:
We searched the following databases up to May 2011: MEDLINE, SCOPUS, ISI Web of
Science, Educational Resource Information Center (ERIC), and Cumulative Index to Nursing
and Allied Health Literature (CINAHL) using the following search strategy:
((evidence-based medic*) OR (evidence based medic*) OR (evidence-based practic*) OR
(evidence based practic*) OR (critic* AND apprais*) OR (pre-filter*) OR (prefilter*) OR (predigest*) OR (predigest*)) AND (educat* OR teach* OR cours* OR workshop* OR learn* OR
instruct* OR curriculum* OR (journal* AND club*) OR (case discuss*)) AND (student* OR
intern OR interns OR internship* OR (clinical clerk*) OR undergraduat*).
We also searched the Current Controlled Trials for relevant unpublished studies. For this
purpose, we tailored the above search strategy accordingly. Furthermore, we performed a
backward and forward reference checking by: A) screening the references of our included studies
Ahmadi et al
8
Teaching EBM to Medical Students
and relevant systematic reviews, and B) screening the studies that have cited any of our included
studies as their references (citation checking). We performed the latter using Science Citation
Index and SCOPUS.
The retrieved studies were imported into EndNote X3 software and the duplicated studies were
removed. The remaining studies were subsequently screened for inclusion based upon their titles
and abstracts initially, and their full-text finally. One of the two investigators (SFA, EA) decided
upon including each study (this step was not performed in duplicate).
Data abstraction and risk of bias assessment:
Two investigators (SFA and EA) independently summarized the study characteristics, key
results, and quality indicators using an electronic data abstraction form in Microsoft Excel
Software. Disagreements between the two investigators were resolved by third reviewer
negotiation. For studies with unclear or inadequate results, we sent an electronic data abstraction
form to the corresponding author and requested further details.
For quality assessment, two sets of criteria were used: A) a set of criteria developed by the
Cochrane Effective Practice and Organisation of Care (EPOC) group (Parkes et al., 2001) and
“using validated assessment tools” criterion (the investigators used this set to code the overall
risk of bias as high, moderate, or low); and B) a modified version of another criteria developed
by Reed et al. to appraise the reports of medical education interventions (Reed et al., 2005).
These criteria are available in Table 2.
Synthesis of results:
We synthesized the results qualitatively by tabulating the characteristics of the included studies
(Table 1) and whether they fulfill the quality criteria (Table 2). We also classified the studies
Ahmadi et al
9
Teaching EBM to Medical Students
based on their interventions, and discussed the effects of the interventions on the knowledge,
attitudes, skills, and behaviors of asking, acquiring, appraising, and applying.
Inter-rater agreement was quantified using Kappa scores. To calculate the Kappa scores for our
data abstraction, we compared the codes that the two investigators assigned to the study designs,
intervention categories, and assessment types of the included studies (Table 1). To calculate the
Kappa scores for our quality assessments, we compared the assigned codes to the quality criteria
(Table 2).
We attempted to meta-analyze the results of the studies with similar outcome assessments and
with minimal diversity in their study designs, participants, and interventions. However, we found
few studies with the above characteristics and therefore felt meta-analysis to be an inappropriate
statistical endeavor in this context.
Ahmadi et al
10
Teaching EBM to Medical Students
RESULTS
Description of included studies:
We retrieved 10,111 records in total, of which 27 were included in this study (Table 1). Included
studies with non-randomized designs and high risks of bias were predominant: The number of
self-controlled trials, parallel non-randomized controlled trials, and randomized controlled trials
was 11, 6, and 10, respectively (Table 1). In addition, the number of studies with a high,
moderate, and low risk of bias was 17, 5, and 5, respectively (Table 2). No included study
evaluated the long-term effects of the intervention(s), and only 16 studies reported their results in
adequate detail (Table 2). Kappa values were 0.87, 0.69,and 0.72 for the inter-rater agreement in
data abstraction and in the two parts of the quality assessment, respectively.
Effects of clinically integrated methods:
The teaching of EBM is believed to be more effective if it is integrated into clinical practice
(Coomarasamy and Khan, 2004). We identified seven studies evaluating such clinically
integrated methods (West et al., 2011, Lai and Nalliah, 2010, Aronoff et al., 2010, Lai and Teng,
2009, Krueger, 2006, Alper and Vinson, 2005, Dorsch et al., 2004), while the remaining twenty
studies evaluated standalone methods in no clinical practice context. Dorsch et al. reported the
earliest clinically integrated teaching of EBM to the students, in which they observed slightly
improved asking skills, acquiring attitudes and skills, and appraising skills (Dorsch et al., 2004).
Another study also observed no effect of a clinically integrated method on acquiring attitudes
(Lai and Nalliah, 2010). However, they reported no a priori sample size calculation; thus, they
might lack sufficient power to detect a possibly existent educational effect.
The other five studies of clinically integrated methods reported improved acquiring skills (Alper
and Vinson, 2005), appraising knowledge and skills (Krueger, 2006), and EBM knowledge
Ahmadi et al
11
Teaching EBM to Medical Students
(West et al., 2011) and skills (Lai and Teng, 2009, Aronoff et al., 2010, West et al., 2011).
However, four of them had a high risk of bias (Table 2), and the only exception – with a
moderate risk of bias – lacked a validated assessment tool (Krueger, 2006). In addition, in one of
these five “positive” studies, participants who were educated earlier received lower post-test
scores, which indicates that the educational effect might be short-term (Lai and Teng, 2009).
Effects of short instructions:
Eleven studies examined the effect of seminars, workshops and short courses (Sastre et al., 2011,
Taheri et al., 2008, Weberschock et al., 2005, Gruppen et al., 2005, Sanchez-Mendiola, 2004,
Fritsche et al., 2002, Rosenberg et al., 1998, Landry et al., 1994, Frasca et al., 1992, Bennett et
al., 1987, Radack and Valanis, 1986), from which two studies reported no effect on acquiring
behavior or appraising skills (Radack and Valanis, 1986, Landry et al., 1994), and another study
reported no effect on EBM knowledge but improved attitudes towards the use of scientific
evidence (Sanchez-Mendiola, 2004). The other eight studies found positive effects. However, ten
of these eleven studies had a high risk of bias, and the only exception with an acceptable quality
(Rosenberg et al., 1998) lacked a validated assessment tool (Table 2).
Notably, in the study by Weberschock et al. medical students successfully delivered the
compulsory EBM course to their peers, which yielded improved knowledge and skills of EBM
(Weberschock et al., 2005).
Effects of e-learning:
Six studies investigated the effects of the online or computer-assisted courses and instructions
(Bradley et al., 2005, Bolboaca and Jantschi, 2006b, Schilling et al., 2006, Davis et al., 2007,
Davis et al., 2008, Hadley et al., 2010), from which three studies with a low risk of bias reported
computer-assisted sessions to be as effective as usual teaching sessions in teaching acquiring
Ahmadi et al
12
Teaching EBM to Medical Students
knowledge, appraising knowledge and skills, and EBM knowledge and attitudes, generally
(Bradley et al., 2005, Davis et al., 2007, Davis et al., 2008). Similarly, another study with a
moderate risk of bias found online modules similarly effective as usual teaching in training
asking, acquiring, and appraising knowledge (Hadley et al., 2010).
The other two studies compared e-learning with no intervention: one study linked an online
module to improved skills of acquiring and calculation of number needed to treat (Schilling et
al., 2006), while the other study correlated a CD-ROM e-course to improved EBM knowledge
(Bolboaca and Jantschi, 2006b). However, the former used no validated assessment tool –
despite its acceptable quality – and the latter had a high risk of bias (Table 2).
Effects of problem-based learning:
Johnston et al. compared a stand-alone problem-based learning intervention with usual teaching
in a high-quality study (Table 2) and found “usual” teaching more favorable in improving EBM
knowledge and attitudes (Johnston et al., 2009).
In another study from McMaster University (the pioneer of problem-based learning), educators
used problem-based material but no real problem-based learning strategy to teach EBM (Bennett
et al., 1987); therefore, their results were not elaborated here.
Effects of other multi-component interventions:
Lee and colleagues compared a multi-component intervention – consisting of short courses plus
self-reading and practice – with no intervention in a randomized controlled trial (RCT) and
observed an improved knowledge of decision analysis but no improved knowledge of costeffectiveness or sensitivity analyses (Lee et al., 2007). The authors presented their intervention as
a clinically integrated teaching method; however, we categorized their teaching as stand-alone
since their teaching was not carried out in a clinical context. This study also used no validated
Ahmadi et al
13
Teaching EBM to Medical Students
assessment tool (Table 2). Furthermore, their intention-to-treat analysis is questionable since
participants consented to enter the study after being randomly allocated, and those who did not
consent were not included in analyses.
In a cross-over RCT, Leung et al. compared the effects of adding the following interventions to a
workshop: A) Providing guides and “InfoRetriever” on personal digital assistants (PDAs); B)
Providing educational pocket cards of the same guides; and C) No intervention (Leung et al.,
2003). They observed improved perceptions regarding the use of EBM and the integration of
EBM in clinical teaching in both active arms although the PDA arm yielded a larger effect.
Notably, despite the researchers’ potential control over participants’ assignment and study
design, the participants were rotated through the three study arms in a disorganized manner, and
neither the total number of experiments per arm nor the crossing-over order was balanced.
However, the study had a low risk of bias (Table 2).
Ahmadi et al
14
Teaching EBM to Medical Students
DISCUSSION
The findings of this systematic review showed that teaching EBM has the potential to improve
knowledge, attitudes and skills in undergraduate medical students. However, there is still
insufficient evidence to support the statement that EBM teaching either improves students’
behaviors or yields a long-term mastery of EBM. In addition, we found no study assessing
patient outcomes or health delivery processes, possibly because undergraduate students would be
rarely the final decision makers regarding patient management. Studies of the clinically
integrated methods and short instructions were weak and inconsistent. In contrast, a number of
robust studies supported the use of e-learning strategies. A single strong study found problembased learning less effective than usual teaching. Finally, few studies linked other
multicomponent interventions to improved knowledge and attitudes.
Our study as well as other systematic reviews has found the studies of EBM teaching generally
weak (Harris et al., 2011, Flores-Mateo and Argimon, 2007, Ebbert et al., 2001, Parkes et al.,
2001, Taylor et al., 2000b). However, the lack of high-quality evidence is not merely confined to
the teaching of EBM, but it is a universal dilemma for the teaching of various sciences .(Hatala
and Guyatt, 2002). Therefore, we should not under-value the teaching of EBM to undergraduate
medical students due to the lack of insufficient robust evidence. Instead, we should focus on
providing robust evidence by the conduct of the future studies in higher qualities with a focus on
the skills and behaviors as well as the long-term educational effects. Only two of our included
studies measured behaviors following the educational interventions (Sastre et al., 2011, Landry
et al., 1994).
Although we did not aim to appraise the validity of the assessment tools of our included studies,
another review (Shaneyfelt et al., 2006) found that only seven of our included studies have high-
Ahmadi et al
15
Teaching EBM to Medical Students
quality assessment tools (West et al., 2011, Aronoff et al., 2010, Lai and Teng, 2009, Bradley et
al., 2005, Weberschock et al., 2005, Fritsche et al., 2002, Bennett et al., 1987). When we
attempted to pool the results of the included studies, we ended up with no more than five studies
(West et al., 2011, Lai and Teng, 2009, Aronoff et al., 2010, Fritsche et al., 2002, Weberschock
et al., 2005) since the other included studies had used miscellaneous tools rather than established
validated tools such as Fresno Test or Berlin Questionnaire. Using similar assessment tools
would enable the researchers to quantitatively pool the results together in meta-analyses, which
yields larger statistical powers and improved generalizability. Selection of the assessment tools
should be based on not only their quality, but also their purpose. As an example, although the
Berlin Questionnaire and the Fresno Test are both established assessment tools, the former is
designed to test applied knowledge through its multiple-choice format, thus it should be avoided
in evaluating the skills of asking or acquiring. On the other hand, the latter is suitable to test the
knowledge and skills across the four steps of EBM (West et al., 2011).
Our systematic review also calls for more robust studies of the clinically integrated methods.
Notably, our included studies examined no ideal “on foot” EBM teaching as described elsewhere
(Richardson, 2005). In addition, our included studies of the clinically integrated methods were
inconsistent and of low quality.
A systematic review (Coomarasamy and Khan, 2004) has found that in postgraduate medical
professionals, clinically integrated methods would improve their EBM knowledge, attitudes,
skills, and behaviors, while standalone methods would improve only knowledge and possibly
skills. In contrast, our results have indicated that for undergraduate students, standalone methods
are able to improve not only the EBM knowledge but also the attitudes and skills. This may be
because students are hypothetically driven by external factors such as the curriculum and the
Ahmadi et al
16
Teaching EBM to Medical Students
assessments. Such factors are possibly addressed by either stand-alone or clinically integrated
methods. In contrast, postgraduates are usually driven by self-motivation and relevance to
clinical practice, which are properly addressed by clinically integrated methods only
(Coomarasamy and Khan, 2004). This argument is supported by another systematic review in
which standalone instructions in critical appraisal improved the knowledge of undergraduate
students, but such instructions yielded limited knowledge gain in residents (Norman and
Shannon, 1998).
A robust systematic review by Hartling et al. drew no net conclusion about the effectiveness of
problem-based learning for undergraduate medical education because of the inconsistencies in
the included studies (Hartling et al., 2010). Another systematic review linked problem-based
learning in medical school to post-graduation improvements, but mainly in social and cognitive
competencies rather than in clinical knowledge and skills (Koh et al., 2008). Our single
identified study of teaching EBM by problem-based learning methods found it less favorable
than usual teaching (Johnston et al., 2009). However, this study was in Hong Kong where
didactic teaching is culturally dominant and the successful delivery of interactive approaches is
challenging (Khan and Coomarasamy, 2006). In addition, this study was brief while effective
problem-based learning methods usually need substantial student-educator interactions (Koh et
al., 2008). Researchers inferred that students may need to initially learn the basics and
subsequently receive problem-based learning in order to successfully grasp the skills to apply
their knowledge (Khan and Coomarasamy, 2006). Considering the above argument, we were
unable to draw net conclusions regarding the true effects of problem-based learning.
A systematic review by Cook et al. found the internet-based learning strategies as effective as
traditional teaching methods (Cook et al., 2008). Their findings are in line with the findings of
Ahmadi et al
17
Teaching EBM to Medical Students
our included studies, particularly the three high-quality studies of computer assisted sessions.
Since e-learning can provide a wide spectrum of teaching strategies, it may make learning more
exciting, effective, and likely to be retained (Greenhalgh, 2001). However, our included studies
of e-learning assessed no behavior, and only one study assessed skills (of acquiring and
applying) (Schilling et al., 2006). This is possibly because higher order impacts (such as
improved behaviors) result from interactive rather than deductive interventions (Greenhalgh,
2001) while none of our included studies of e-learning adopted an ideally interactive e-learning
strategy. In addition, the best EBM teaching models occur at the bedside (Richardson, 2005),
which e-learning cannot easily recreate it. Therefore, e-learning should be considered as a
complement for – rather than a substitute of – clinically integrated bedside models to teach EBM.
Straus et al. have previously distinguished the “using mode” from the “doing mode” of
practicing EBM (Straus et al., 2010). In the “using mode”, physicians search within preappraised sources, thus they bypass the time-consuming appraising step. Although the previous
studies of the EBM education are majorly focused on the “doing mode” and particularly “critical
appraisal” (Hatala and Guyatt, 2002), we found three studies emphasizing the “using mode” and
the “searching within pre-appraised sources” (Sastre et al., 2011, Schilling et al., 2006, Fritsche
et al., 2002). Moreover, another included study showed the positive effects of accessing
“InfoRetriever” through PDAs (Leung et al., 2003). Since the use of such sources is linked to
better clinical decision making (Alper et al., 2005), we would call for more robust studies of
teaching the using mode and the skills to use pre-appraised sources.
One included study reported a successful EBM teaching by medical students (Weberschock et
al., 2005). Trained students have been also reported to be as good as faculty educators in
teaching clinical principles and skills (Haist et al., 1998, Tolsgaard et al., 2007, Graziano, 2011).
Ahmadi et al
18
Teaching EBM to Medical Students
These observations can inspire a model for EBM education, particularly for institutions with
limited faculty educators. However, the current evidence supporting such a model is still
insufficient.
Our systematic review had a number of limitations: We had no access to EMBASE while conducting
this review, thus we could not search it. In addition, despite using comprehensive search
strategies, we used no abbreviated term such as EBM or EBP in our search queries. Moreover, to
identify the unpublished studies, we only searched the Current Controlled Trials that includes
few educational studies. Furthermore, only one investigator decided upon including each study
due to the limited time and resources of the team. Thus, we cannot exclude potential biases in the
identification of the including studies. Finally, we were rather strict in including only
comparative studies and in appraising our included studies based on meticulous quality criteria.
Ahmadi et al
19
Teaching EBM to Medical Students
CONCLUSIONS
Implications for practice:
Teaching EBM has the potential to improve knowledge, attitudes and skills in undergraduate
medical students. However, there is still insufficient evidence to support the statement that EBM
teaching either improves students’ behaviors or yields a long-term mastery of EBM. Evidence[l3]
supporting the use of clinically integrated methods (i.e. educational activities integrated into
clinical practice) and stand-alone short instructions (i.e. brief educational activities conducted in
no real clinical practice context) are currently insufficient. However, high quality evidence has
supported that computer-assisted instructions are as effective as traditional educational strategies
in improving EBM knowledge and attitudes. Nevertheless, their effects on the students’ skills
and behaviors are unclear. We have also drawn no net conclusion about the effectiveness of
problem-based learning of EBM since only one high-quality study examined it. Finally, the
effects of other multicomponent interventions were heterogeneous and inconclusive.
Implications for research:
We suggest future studies of teaching EBM to medical students to focus on: A) Reporting the
participants, interventions, outcomes, and results in sufficient details in order to allow
replication; B) Examining the effects of EBM teaching on long-term skills and behaviors using
robust assessment tools; C) Evaluating appropriate “on foot”, real world clinically integrated
methods, problem-based learning, interactive e-learning strategies, and short courses and
instructions; D) Comparing the teaching of the using and the doing modes of practicing EBM;
and E) Studying the student educator model to test whether trained students are able to teach
EBM effectively.[l4]
Ahmadi et al
20
Teaching EBM to Medical Students
PRACTICE POINTS

Although several systematic reviews have explored various aspects of evidence-based
medicine (EBM), no prior study has attempted to systematically review the effectiveness of
teaching EBM to undergraduate medical students.

We systematically reviewed the studies of clinically integrated methods of teaching EBM,
short courses and instructions, e-learning, problem-based learning, and other multicomponent
interventions. However, we drew no net conclusion since the included studies were either
weak, few, or inconsistent.

In general, teaching EBM has the potential to improve knowledge, attitudes and skills in
undergraduate medical students. However, evidence supporting the effect of EBM teaching
on students’ behaviors is currently insufficient.

We suggest future studies to focus on assessing long-term higher-order mastery of EBM and
use robust methods and high-quality assessment tools.
Ahmadi et al
21
Teaching EBM to Medical Students
NOTES ON CONTRIBUTORS
SFA was a Postdoctoral Fellow at Center for Educational Research in Medical Sciences, Iran
University of Medical Sciences, at the time of this study; he is currently a postdoctoral fellow at
Department of Medicine, University of California Irvine.
HRB is an Associate Professor of Clinical Epidemiology and the Deputy of Research at Center
for Educational Research in Medical Sciences, Iran University of Medical Sciences.
EA was a Postdoctoral Fellow at Digestive Diseases Research Institute, Tehran University of
Medical Sciences, at the time of this study; he is currently a Postdoctoral Fellow at Department
of Radiology, Massachusetts General Hospital, Harvard Medical School.
Ahmadi et al
22
Teaching EBM to Medical Students
ACKNOWLEDGMENTS
We would like to thank Prof. Hammick for all her thoughtful assistance in writing this systematic
review.
Ahmadi et al
23
Teaching EBM to Medical Students
REFERENCES
ALPER, B. S. & VINSON, D. C. 2005. Experiential curriculum improves medical students'
ability to answer clinical questions using the internet. Fam Med, 37, 565-9.
ALPER, B. S., WHITE, D. S. & GE, B. 2005. Physicians answer more clinical questions and
change clinical decisions more often with synthesized evidence: a randomized trial in
primary care. Ann Fam Med, 3, 507-13.
ARONOFF, S. C., EVANS, B., FLEECE, D., LYONS, P., KAPLAN, L. & ROJAS, R. 2010.
Integrating evidence based medicine into undergraduate medical education: combining
online instruction with clinical clerkships. Teach Learn Med, 22, 219-23.
BENNETT, K. J., SACKETT, D. L., HAYNES, R. B., NEUFELD, V. R., TUGWELL, P. &
ROBERTS, R. 1987. A controlled trial of teaching critical appraisal of the clinical
literature to medical students. JAMA, 257, 2451-4.
BOLBOACA, S. D. & JANTSCHI, L. 2006a. Assessment of a computer based curriculum in
evidence-based medicine.
BOLBOACA, S. D. & JANTSCHI, L. Assessment of a computer based curriculum in evidencebased medicine. In: CALLAOS, N., LESSO, W., TREMANTE, A., BARALT, J. &
REBIELAK, J., eds. Wmsci 2006: 10th World Multi-Conference on Systemics,
Cybernetics and Informatics, Vol Vii, Proceedings, 2006b. 405-409.
BRADLEY, P., OTERHOLT, C., HERRIN, J., NORDHEIM, L. & BJORNDAL, A. 2005.
Comparison of directed and self-directed learning in evidence-based medicine: a
randomised controlled trial. Med Educ, 39, 1027-35.
Ahmadi et al
24
Teaching EBM to Medical Students
CASPI, O., MCKNIGHT, P., KRUSE, L., CUNNINGHAM, V., FIGUEREDO, A. J. &
SECHREST, L. 2006. Evidence-based medicine: discrepancy between perceived
competence and actual performance among graduating medical students. Medical
Teacher, 28, 318-325.
COLLEGES., F. P. C. O. T. A. O. M. R. 2007. Curriculum for the foundation years in
postgraduate education and training. [Online]. UK: Departments of Health. Available:
http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndG
uidance/DH_4107594 2011].
COOK, D. A., LEVINSON, A. J., GARSIDE, S., DUPRAS, D. M., ERWIN, P. J. &
MONTORI, V. M. 2008. Internet-based learning in the health professions: a metaanalysis. JAMA, 300, 1181-96.
COOMARASAMY, A. & KHAN, K. S. 2004. What is the evidence that postgraduate teaching
in evidence based medicine changes anything? A systematic review. BMJ, 329, 1017.
CRILLY, M., GLASZIOU, P., HENEGHAN, C., MEATS, E. & BURLS, A. 2009. Does the
current version of 'Tomorrow's Doctors' adequately support the role of evidence-based
medicine in the undergraduate curriculum? Medical Teacher, 31, 938-944.
DAVIS, J., CHRYSSAFIDOU, E., ZAMORA, J., DAVIES, D., KHAN, K. &
COOMARASAMY, A. 2007. Computer-based teaching is as good as face to face lecturebased teaching of evidence based medicine: a randomised controlled trial. BMC Med
Educ, 7, 23.
DAVIS, J., CRABB, S., ROGERS, E., ZAMORA, J. & KHAN, K. 2008. Computer-based
teaching is as good as face to face lecture-based teaching of evidence based medicine: a
randomized controlled trial. Med Teach, 30, 302-7.
Ahmadi et al
25
Teaching EBM to Medical Students
DAWES, M., SUMMERSKILL, W., GLASZIOU, P., CARTABELLOTTA, A., MARTIN, J.,
HOPAYIAN, K., PORZSOLT, F., BURLS, A., OSBORNE, J., SECOND
INTERNATIONAL CONFERENCE OF EVIDENCE-BASED HEALTH CARE, T. &
DEVELOPERS 2005. Sicily statement on evidence-based practice. BMC Med Educ, 5, 1.
DEL MAR, C., GLASZIOU, P. & MAYER, D. 2004. Teaching evidence based medicine. BMJ,
329, 989-90.
DIZON, J. M. R., GRIMMER-SOMERS, K. A. & KUMAR, S. 2012. Current evidence on
evidence-based practice training in allied health: a systematic review of the literature. Int
J Evid Based Healthc, 10, 347-60.
DORSCH, J. L., AIYER, M. K. & MEYER, L. E. 2004. Impact of an evidence-based medicine
curriculum on medical students' attitudes and skills. J Med Libr Assoc, 92, 397-406.
EBBERT, J. O., MONTORI, V. M. & SCHULTZ, H. J. 2001. The journal club in postgraduate
medical education: a systematic review. Medical Teacher, 23, 455-461.
FLORES-MATEO, G. & ARGIMON, J. M. 2007. Evidence based practice in postgraduate
healthcare education: a systematic review. BMC Health Serv Res, 7, 119.
FRANK, J. R., JABBOUR, M. & AL., E. 2005. The CanMEDS 2005 Physician Competency
Framework: Better standards, better physicians, better care. [Online]. Royal College of
Physicians and Surgeons of Canada. Available:
http://rcpsc.medical.org/canmeds/CanMEDS2005/index.php 2011].
FRASCA, M. A., DORSCH, J. L., ALDAG, J. C. & CHRISTIANSEN, R. G. 1992. A
multidisciplinary approach to information management and critical appraisal instruction:
a controlled study. Bull Med Libr Assoc, 80, 23-8.
Ahmadi et al
26
Teaching EBM to Medical Students
FRITSCHE, L., GREENHALGH, T., FALCK-YTTER, Y., NEUMAYER, H. H. & KUNZ, R.
2002. Do short courses in evidence based medicine improve knowledge and skills?
Validation of Berlin questionnaire and before and after study of courses in evidence
based medicine. BMJ, 325, 1338-41.
GRAZIANO, S. C. 2011. Randomized surgical training for medical students: resident versus
peer-led teaching. Am J Obstet Gynecol, 204, 542 e1-4.
GREENHALGH, T. 2001. Computer assisted learning in undergraduate medical education.
BMJ, 322, 40-44.
GRUPPEN, L. D., RANA, G. K. & ARNDT, T. S. 2005. A controlled comparison study of the
efficacy of training medical students in evidence-based medicine literature searching
skills. Acad Med, 80, 940-4.
HADLEY, J., KULIER, R., ZAMORA, J., COPPUS, S. F., WEINBRENNER, S.,
MEYERROSE, B., DECSI, T., HORVATH, A. R., NAGY, E., EMPARANZA, J. I.,
ARVANITIS, T. N., BURLS, A., CABELLO, J. B., KACZOR, M., ZANREI, G.,
PIERER, K., KUNZ, R., WILKIE, V., WALL, D., MOL, B. W. & KHAN, K. S. 2010.
Effectiveness of an e-learning course in evidence-based medicine for foundation
(internship) training. J R Soc Med, 103, 288-94.
HAIST, S. A., WILSON, J. F., BRIGHAM, N. L., FOSSON, S. E. & BLUE, A. V. 1998.
Comparing fourth-year medical students with faculty in the teaching of physical
examination skills to first-year students. Acad Med, 73, 198-200.
HARRIS, J., KEARLEY, K., HENEGHAN, C., MEATS, E., ROBERTS, N., PERERA, R. &
KEARLEY-SHIERS, K. 2011. Are journal clubs effective in supporting evidence-based
decision making? A systematic review. BEME Guide No. 16. Medical Teacher, 33, 9-23.
Ahmadi et al
27
Teaching EBM to Medical Students
HARTLING, L., SPOONER, C., TJOSVOLD, L. & OSWALD, A. 2010. Problem-based
learning in pre-clinical medical education: 22 years of outcome research. Med Teach, 32,
28-35.
HATALA, R. & GUYATT, G. 2002. Evaluating the teaching of evidence-based medicine.
JAMA, 288, 1110-2.
JOHNSTON, J. M., SCHOOLING, C. M. & LEUNG, G. M. 2009. A randomised-controlled trial
of two educational modes for undergraduate evidence-based medicine learning in Asia.
BMC Med Educ, 9, 63.
KHAN, K. S., AWONUGA, A. O., DWARAKANATH, L. S. & TAYLOR, R. 2001.
Assessments in evidence-based medicine workshops: loose connection between
perception of knowledge and its objective assessment. Med Teach, 23, 92-94.
KHAN, K. S. & COOMARASAMY, A. 2006. A hierarchy of effective teaching and learning to
acquire competence in evidenced-based medicine. BMC Med Educ, 6, 59.
KOH, G. C., KHOO, H. E., WONG, M. L. & KOH, D. 2008. The effects of problem-based
learning during medical school on physician competency: a systematic review. CMAJ,
178, 34-41.
KRUEGER, P. M. 2006. Teaching critical appraisal: a pilot randomized controlled outcomes
trial in undergraduate osteopathic medical education. J Am Osteopath Assoc, 106, 65862.
LAI, N. M. & NALLIAH, S. 2010. Information-seeking practices of senior medical students: the
impact of an evidence-based medicine training programme. Educ Health (Abingdon), 23,
151.
Ahmadi et al
28
Teaching EBM to Medical Students
LAI, N. M. & TENG, C. L. 2009. Competence in evidence-based medicine of senior medical
students following a clinically integrated training programme. Hong Kong Med J, 15,
332-8.
LANDRY, F. J., PANGARO, L., KROENKE, K., LUCEY, C. & HERBERS, J. 1994. A
controlled trial of a seminar to improve medical student attitudes toward, knowledge
about, and use of the medical literature. J Gen Intern Med, 9, 436-9.
LEE, A., JOYNT, G. M., HO, A. M. H., GIN, T. & HAZLETT, C. B. 2007. Effect of an
integrated teaching intervention on clinical decision analysis: a randomized, controlled
study of undergraduate medical students. Medical Teacher, 29, 231-236.
LEUNG, G. M., JOHNSTON, J. M., TIN, K. Y., WONG, I. O., HO, L. M., LAM, W. W. &
LAM, T. H. 2003. Randomised controlled trial of clinical decision support tools to
improve learning of evidence based medicine in medical students. BMJ, 327, 1090.
NORMAN, G. R. & SHANNON, S. I. 1998. Effectiveness of instruction in critical appraisal
(evidence-based medicine) skills: a critical appraisal. Cmaj, 158, 177-81.
PARKES, J., HYDE, C., DEEKS, J. & MILNE, R. 2001. Teaching critical appraisal skills in
health care settings. Cochrane Database Syst Rev, CD001270.
RADACK, K. L. & VALANIS, B. 1986. Teaching critical appraisal and application of medical
literature to clinical problem-solving. J Med Educ, 61, 329-31.
REED, D., PRICE, E. G., WINDISH, D. M., WRIGHT, S. M., GOZU, A., HSU, E. B., BEACH,
M. C., KERN, D. & BASS, E. B. 2005. Challenges in systematic reviews of educational
intervention studies. Ann Intern Med, 142, 1080-9.
RICHARDSON, W. S. 2005. Teaching evidence-based practice on foot. Evidence Based
Medicine, 10, 98-101.
Ahmadi et al
29
Teaching EBM to Medical Students
ROSENBERG, W. M., DEEKS, J., LUSHER, A., SNOWBALL, R., DOOLEY, G. &
SACKETT, D. 1998. Improving searching skills and evidence retrieval. J R Coll
Physicians Lond, 32, 557-63.
SACKETT, D. L., ROSENBERG, W. M., GRAY, J. A., HAYNES, R. B. & RICHARDSON, W.
S. 1996. Evidence based medicine: what it is and what it isn't. Bmj, 312, 71-2.
SANCHEZ-MENDIOLA, M. 2004. Evidence-based medicine teaching in the Mexican Army
Medical School. Med Teach, 26, 661-3.
SASTRE, E. A., DENNY, J. C., MCCOY, J. A., MCCOY, A. B. & SPICKARD, A. 2011.
Teaching evidence-based medicine: Impact on students' literature use and inpatient
clinical documentation. Med Teach, 33, e306-12.
SCHILLING, K., WIECHA, J., POLINENI, D. & KHALIL, S. 2006. An interactive web-based
curriculum on evidence-based medicine: design and effectiveness. Fam Med, 38, 126-32.
SHANEYFELT, T., BAUM, K. D., BELL, D., FELDSTEIN, D., HOUSTON, T. K., KAATZ,
S., WHELAN, C. & GREEN, M. 2006. Instruments for evaluating education in evidencebased practice: a systematic review. JAMA, 296, 1116-27.
STEWART, M. G. 2001. ACGME Core Competencies. [Online]. Accreditation Council for
Graduate Medical Education. Available:
http://www.acgme.org/acwebsite/RRC_280/280_corecomp.asp 2011].
STRAUS, S. E., RICHARDSON, W. S., GLASZIOU, P. & HAYNES, R. B. 2010. Evidencebased Medicine: How to Practice and Teach EBM, Edinburgh, Churchill Livingstone.
TAHERI, H., MIRMOHAMADSADEGHI, M., ADIBI, I., ASHORION, V., SADEGHIZADE,
A. & ADIBI, P. 2008. Evidence-based medicine (EBM) for undergraduate medical
students. Ann Acad Med Singapore, 37, 764-8.
Ahmadi et al
30
Teaching EBM to Medical Students
TAYLOR, R., REEVES, B., EWINGS, P., BINNS, S., KEAST, J. & MEARS, R. 2000a. A
systematic review of the effectiveness of critical appraisal skills training for clinicians.
Med Educ, 34, 120-5.
TAYLOR, R., REEVES, B., EWINGS, P., BINNS, S., KEAST, J. & MEARS, R. 2000b. A
systematic review of the effectiveness of critical appraisal skills training for clinicians.
Medical Education, 34, 120-125.
TOLSGAARD, M. G., GUSTAFSSON, A., RASMUSSEN, M. B., HOIBY, P., MULLER, C. G.
& RINGSTED, C. 2007. Student teachers can be as good as associate professors in
teaching clinical skills. Med Teach, 29, 553-7.
VAN DIJK, N., HOOFT, L. & WIERINGA-DE WAARD, M. 2010. What are the barriers to
residents' practicing evidence-based medicine? A systematic review. Acad Med, 85,
1163-70.
WALCZAK, J., KALETA, A., GABRYS, E., KLOC, K., THANGARATINAM, S.,
BARNFIELD, G., WEINBRENNER, S., MEYERROSE, B., ARVANITIS, T. N.,
HORVATH, A. R., ZANREI, G., KUNZ, R., SUTER, K., BURNAND, B., ARDITI, C.,
RENGERINK, K. O., HARRY, G., MOL, B. W. J. & KHAN, K. S. 2010. How are
"teaching the teachers" courses in evidence based medicine evaluated? A systematic
review. Bmc Medical Education, 10.
WEBERSCHOCK, T. B., GINN, T. C., REINHOLD, J., STRAMETZ, R., KRUG, D.,
BERGOLD, M. & SCHULZE, J. 2005. Change in knowledge and skills of Year 3
undergraduates in evidence-based medicine seminars. Med Educ, 39, 665-71.
Ahmadi et al
31
Teaching EBM to Medical Students
WEST, C. P., JAEGER, T. M. & MCDONALD, F. S. 2011. Extended evaluation of a
longitudinal medical school evidence-based medicine curriculum. J Gen Intern Med, 26,
611-5.
Ahmadi et al
32
Teaching EBM to Medical Students
TABLES
Table 1: Characteristics of included studies.
Design, setting and
Study ID
Interventions (Teaching methods)
Outcomes (Assessment)
Key Results
participants
(West et al.,
SCT1 in Mayo Medical
Integrated Method: I. 20 to 22-hour
Validated: Survey with Berlin
Knowledge and skills
2011)
School, USA (2006-
lectures plus small-group discussions
Questionnaire and/or Fresno Test
of EBM: improved
2008) on 99 2nd-year
in EBM principles and appraisal of
prior to and after short course,
medical students
various study types in 2nd year, II.
and upon completion of 3rd year.
Electronic feedback on developing
CATs2 from real patients in 6-7
rotations of 3rd year.
(Sastre et al., SCT in Vanderbilt
Short instruction: 3-hour workshop
Partly validated3: Pre- and post-
Behaviors of acquiring
2011)
Medical School, USA
in asking, acquiring, and pros and
workshop analysis of computer
and applying and
(2007-2008) on 100 3rd-
cons of pre-appraised sources.
log data of searches, expert
Attitudes toward EBM
scoring of EBM content in
and acquiring:
clerkship notes, and survey to
improved
year medical students
Ahmadi et al
33
Teaching EBM to Medical Students
assess attitudes towards EBM and
acquiring.
(Lai and
SCT in International
Integrated Method: I. 2 sessions in
Validated: Pre- and post-course
Attitudes toward
Nalliah,
Medical University of
EBM principles, acquiring, and
questionnaire to assess preferred
acquiring: unchanged
2010)
Malaysia (2005-2006) on
appraising, II. Electronic exploratory
information sources.
65 Final-year medical
notes, III. 6 x 2-hour small-group
students
bedside sessions to exercise asking,
IV. Self-searching, V. Presenting
CATs in journal clubs, VI. EBM
reports in portfolios.
(Hadley et
Cluster RCT4 in seven
e-learning: Intervention A: 3-hour
Validated: Module-specific
Knowledge of asking,
al., 2010)
teaching hospitals in UK
session with modules in asking,
multiple-choice questions to
acquiring and
West Midlands (2007) on
acquiring, and appraising;
assess knowledge before and after appraising: comparable
237 Interns (Foundation
Intervention B: 6-week web access to each module.
year 2 doctors)
e-learning modules in the same topics.
SCT in Temple
Integrated Method: I. 18-week
(Aronoff et
Validated: Pre- and post-course
Skills of EBM:
Ahmadi et al
al., 2010)
34
University, USA (2005)
access to 6 online modules, plus
on 153 3rd-year medical
supervised assignments in asking,
students
acquiring, appraising various study
Teaching EBM to Medical Students
Fresno Test.
improved
types, and applying, II. Completion of
4 CATs from real patients.
(Lai and
SCT in International
Integrated Method: I. 2 sessions in
Validated: Pre- and post-course
Skills of EBM:
Teng, 2009)
Medical University of
EBM resources and appraising plus
Modified Fresno Test.
improved
Malaysia (2006) on 72
electronic exploratory notes, II. 6 x 2-
Final-year medical
hour small-group bedside sessions to
students
exercise asking, III. Self-searching,
IV. Presenting CATs in journal clubs,
V. Developing EBM reports in
portfolios.
(Johnston et
Cross-over RCT in
Problem-based learning:
Validated: Before, post-phase 1,
Knowledge of and
al., 2009)
University of Hong Kong
Intervention A: 4 x 4-hour usual
and post-phase 2 KAB
attitudes toward EBM:
(2007) on 129 2nd-year
teaching sessions to practice asking,
Questionnaire to assess EBM
favor usual teaching.
Ahmadi et al
medical students
35
Teaching EBM to Medical Students
acquiring, appraising, and applying;
knowledge, personal application
Intervention B: 4 x 4-hour problem-
and current and future use of
based learning sessions to practice the
EBM, and attitudes towards
same steps.
EBM.
(Taheri et
SCT in Isfahan
Short instruction: 4-day workshop
Validated: Pre- and post-
Knowledge of asking
al., 2008)
University of Medical
(each day: 2-hour lecture + 1-hour
workshop questionnaire to assess
and Skills of acquiring:
Sciences, Iran (2005) on
small-group session) in asking,
knowledge of asking, plus expert
improved
24 5th- and 6th-year
acquiring, appraising, and applying.
evaluation of acquiring.
medical students
(Davis et al.,
RCT in University of
e-learning: Intervention A: 40-
Validated: Pre- and post-session
Knowledge of EBM:
2008)
Birmingham, UK (2006)
minute computer based session to
questionnaire to measure EBM
comparable; Attitudes
on 229 1st-year medical
teach asking, acquiring, appraising,
knowledge and attitudes.
toward EBM: mostly
students
and applying; Intervention B: 40minute lecture based session to teach
the same topics.
comparable
Ahmadi et al
36
Teaching EBM to Medical Students
(Lee et al.,
RCT in Chinese
Multicomponent: Intervention:
Non-validated: Pre- and post-
Knowledge of apply
2007)
University of Hong Kong
teaching about performing and
course questionnaire to assess
subsets: partly
(2005) on 155 5th-year
appraising decision, sensitivity, and
knowledge of decision analysis,
improved
medical students
cost-effectiveness analyses through: I.
sensitivity analysis graphs, and
reading a handbook, II. 3 x 40-minute
cost-effectiveness ratios.
lectures, III. 1-hour small-group
session, IV. home-appraising, V. 1hour workshop on using software.
Control: none.
(Davis et al.,
RCT in five teaching
e-learning: Intervention: 40-minute
Validated: Pre- and post-session
Knowledge of and
2007)
hospitals in UK West
computer based session in asking,
questionnaire to measure EBM
attitudes toward EBM:
Midlands (2005) on 55
acquiring, appraising and applying;
knowledge and attitudes.
comparable
Interns (Foundation year
Control: 40-minute lecture based
1 doctors)
session in the same topics.
(Schilling et
RCT in Boston
e-learning: Intervention: 4-week
Partly validated: Post-course
Skills of acquiring and
al., 2006)
University, USA, on 238
web access to four online modules in
expert evaluation of captured
a minor skill of
Ahmadi et al
37
Teaching EBM to Medical Students
3rd-year medical
MEDLINE, pre-appraised sources,
OVID MEDLINE searches,
students
Study designs, and NNT calculation;
retrieval of high-quality
Control: none.
evidence, correct calculation of
applying: improved
NNT, and number of MEDLINE
searches.
(Krueger,
RCT in University of
Integrated Method: Intervention: I.
Non-validated: Post-course
Knowledge/ skills of
2006)
Medicine and Dentistry
Lecture in EBM, II. 2 small-group
multiple-choice question Critical
appraising: improved
of New Jersey, USA
discussions in appraising, III.
Appraisal Examination to assess
(1998-1999) on 77 3rd-
Reading materials in applying, IV.
appraising knowledge/ skills.
year students of
Journal club, V. Instruction in
osteopathic medicine
Cochrane Library, IV. EBM
assignment; Control: Lectures in
other topics.
(Bolboaca and
SCT in IuliuHatieganu
e-learning: 3-month access to a CD-
Non-validated: Pre- and post-e-
Knowledge of EBM:
Jantschi,
University of Medicine
ROM e-course consisting of: I. 14
course Test of EBM knowledge.
improved
2006a)
and Pharmacy, Romania
tutorials in EBM steps and appraising
Ahmadi et al
38
(2005) on 40 4th- to 6th-
various study types plus self-
year medical students
evaluation tests, II. Supplementary
Teaching EBM to Medical Students
material including glossary,
Romanian guidelines, and relevant
papers and software.
(Weberschock
SCT in Johann
Short instruction: 4 x 3-hour EBM
Validated: Pre- and post-course
et al., 2005)
Wolfgang Goethe
lectures and small-group discussions
Question papers (2 sets) to assess of EBM: improved
University, Germany
in EBM principles and asking,
application of principles, and
(2003-2004) on 132 3rd-
acquiring through MEDLINE, and
Berlin questionnaire.
year medical students
applying therapy and diagnosis
Knowledge and skills
studies.
(Gruppen et
NCT5 in University of
Short instruction: Intervention: 2-
Non-validated: Pre-session and
Skills of acquiring:
al., 2005)
Michigan, USA (2001-
hour additional session in using Ovid
1-month post-session expert
improved
2003) on 92 4th-year
MEDLINE, consisting of a brief
scoring of the quality of a
medical students
lecture and guided hand-on practice
particular search using a pre-
during an EBM course; Control:
developed scoring sheet.
Ahmadi et al
39
Teaching EBM to Medical Students
EBM course alone.
(Bradley et
RCT in University of
e-learning: Intervention A:
Validated: 18 weeks post
Knowledge of
al., 2005)
Oslo, Norway (2002-
Workshop (directed learning) of 5 x
intervention questionnaire to
acquiring, Knowledge
2003) on 175 10th-
3-hour sessions in asking, acquiring,
assess acquiring and appraising
and skills of
semester medical
appraising and applying;
knowledge, expert scoring of
appraising and
students
Intervention B: Computer-assisted
participant-developed CATs to
Attitudes toward
modules (self-directed learning) in
assess appraising skills, and 1 to
EBM: comparable
the same topics.
17 weeks post intervention
questionnaire to assess attitudes
toward EBM.
(Alper and
SCT in University of
Integrated Method: I. 90-minute
Non-validated: Pre- and post-
Skills of acquiring:
Vinson, 2005)
Missouri-Columbia,
computer lab session in acquiring, II.
intervention student recorded
improved
USA, on 90 3rd-year
Access to free internet portal, III.
time-to-answer and number of
medical students
Handout of practical points for
searched sites to answer 3
acquiring and applying, IV. 2
clinical questions, and expert
acquiring assignments from real
evaluation of the quality of
Ahmadi et al
40
patients, V. Follow-up 90-minute
Teaching EBM to Medical Students
answers.
computer lab session.
(Sanchez-
NCT in Mexican Army
Short instruction: Intervention: 14
Validated: Post-course
Attitudes toward
Mendiola,
Medical School (2001-
x 2-hour sessions in EBM;Control:
questionnaire to assess attitudes
acquiring: Improved;
2004)
2002) on 131 5th- and
none.
towards acquiring and self-
Knowledge of EBM:
6th-year medical
reported preferred information
unchanged
students
sources, plus knowledge of
EBM.
(Dorsch et al.,
SCT in University of
Integrated Method: I. 8 x 1-hour
Non-validated: Pre- and post-
Attitudes toward
2004)
Illinois, USA (2000-
weekly seminars plus pre-session
seminar self-reported frequently
acquiring and Skills of
2001) on 36 3rd-year
reading materials in EBM principles
used information sources, and
asking and acquiring:
medical students
and asking, acquiring, and appraising
expert scoring of written test of
minimally changed;
diagnosis, therapy, and meta-analysis
EBM steps.
Skills of
studies; II. Developing and
appraising:partly
presenting CATs from real patients in
changed
3 sessions.
Ahmadi et al
(Leung et al.,
Cross-over RCT in
2003)
41
Multicomponent: Intervention "P"
Teaching EBM to Medical Students
Validated: Baseline and post-
Attitudes toward
University of Hong Kong (Pocket Card): 2 x 2-hour sessions
phase 1 to 3 questionnaires to
EBM: improved by
(2001) on 169 4th-year
in EBM principles, asking, acquiring,
assess personal applicationas
Pocket Card, further
medical students
and applying; using pocket card of
well as current and future use of
improved by
EBM guides; and supervised
EBM.
InfoRetriever.
practicing of EBM steps;
Intervention "I" (InfoRetriever):
similar to "P", but PDA with
InfoRetriever and digital pocket card
was also provided; Control (C):
none.
(Fritsche et
SCT in various short
Short instruction: 3-day course in
Validated: 0 to 4 weeks pre- and
Knowledge and skills
al., 2002)
EBM courses in
EBM principles and in asking,
post-workshop Berlin
of EBM: improved
Germany (1999-2001) on appraising, and applying plus using
203 3rd-year medical
pre-appraised evidence and
students
estimating risk, benefit, and harm.
Questionnaire.
Ahmadi et al
42
Teaching EBM to Medical Students
(Rosenberg et
RCT in Oxford
Short instruction: Intervention: 3-
Non-validated: Pre- (in
Skills of acquiring:
al., 1998)
University, UK, on 108
hour small-group session in asking
intervention group) and post-
improved
1st-year clinical students
and acquiring through WinSpirs
session expert scoring of search
MEDLINE; Control: none.
strategies and number and
quality of retrieved citations.
(Landry et
NCT in 4 army
Short instruction: Intervention: 2 x
Non-validated: 1-week pre-
Behavior of acquiring:
al., 1994)
universities in DC
90-min seminars in types of medical
seminars and 5-weeks post-
unchanged
(Intervention), and
literature and study design, and in
seminars expert scoring of
Maryland, Ohio, and
appraising diagnostic test and therapy
literature use in patient write-ups.
Texas (Control), on 146
articles; Control: none.
3rd-year clinical clerks
(Frasca et
NCT in 2 campuses of
Short instruction: Intervention: I.
Non-validated: Post-course
Skills of acquiring and
al., 1992)
University of Illinois,
10 x 1.5-hour sessions in acquiring
questionnaire to assess acquiring
Knowledge of
USA, on 92 3rd-year
and appraising diagnostic test,
skills and appraising knowledge.
appraising: improved
clinical clerks
prognosis, etiology or causation, and
therapy effectiveness studies, II.
Ahmadi et al
43
Teaching EBM to Medical Students
Supervised development of a CAT;
Control: none.
(Bennett et
NCT in McMaster
Short instruction: Intervention: 8 x
Non-validated: Pre- and post-
Skills of appraising:
al., 1987)
University, Canada, on
2-hour small-group sessions to teach
sessions expert scoring of
improved
92 Final-year clinical
appraising diagnostic test and therapy
appraising a diagnostic test and 2
clerks
effectiveness studies;Control: none.
therapy effectiveness studies.
(Radack and
NCT in University of
Short instruction: Intervention: 5 x
Non-validated: Pre- and post-
Skills of appraising:
Valanis,
Cincinnati, USA (1984-
50-min small-group sessions in
sessions unidentified test of
unchanged
1986)
1985) on 34 4th-year
appraising clinical measurement,
appraising diagnosis and therapy.
clinical clerks
diagnostic testing, and therapeutic
efficacy;Control: none.
1
SCT: Self-controlled trial;
2
CAT: Critically appraised topic;3 Partly validated: Asubset (and not all) of outcomes were
assessed using validated assessment tools4 RCT: Randomized controlled trial; 5 NCT: Non-randomized controlled trial.
Study ID
(West et al.,
(Sastre et al.,
(Lai and
(Hadley et al.,
(Aronoff et
al., 2010)
Validated assessment tools
No other risks of bias
No selective outcome reporting
Protection against contamination
Blinding of the outcome assessors
Addressing incomplete outcome data
Similar baseline characteristics
Similar baseline outcome measurements
Allocation concealment
Generation of randomization sequence
Overall
risk of
bias
Reporting results in adequate details1
Appropriate statistical tests
Power analysis for sample size
calculation
Assessment of long-term effects
Description of assessment methods
Appraisal checklist for risk of bias assessment
Description of teaching methods
Description of teaching objectives and
content
44
Description of learner characteristics
Ahmadi et al
Teaching EBM to Medical Students
Table 2: Risk of bias assessment and ancillary quality criteria.
Ancillary quality criteria
2011)
2011)
Nalliah, 2010)
2010)
N2
N
N
N
Y3
U4
N
Y
Y
Y
High
Y
Y
Y
Y
N
N
Y
Y
N
N
N
N
Y
Y
N
Y
Y
Y
High
Y
Y
Y
Y
N
N
Y
Y
N
N
N
N
Y
Y
N
Y
Y
Y
High
Y
Y
Y
Y
N
N
Y
Y
U
U
Y
N
N
Y
Y
Y
Y
Y
Moderate
Y
Y
Y
Y
N
Y
Y
Y
N
N
N
N
Y
U
N
Y
Y
Y
High
Y
N
Y
Y
N
N
Y
Y
Ahmadi et al
(Lai and
45
Teaching EBM to Medical Students
N
N
N
N
N
U
N
Y
Y
Y
High
Y
Y
Y
Y
N
Y
Y
Y
U
Y
Y
Y
Y
Y
N
Y
Y
Y
Low
Y
Y
Y
Y
N
N
Y
Y
N
N
N
N
Y
Y
N
Y
Y
Y
High
Y
Y
Y
Y
N
N
Y
Y
Y
Y
Y
Y
N
Y
Y
Y
Y
Y
Low
Y
N
Y
Y
N
Y
Y
Y
Y
Y
Y
U
Y
Y
U
Y
Y
N
Moderate
Y
Y
Y
Y
N
N
Y
N
Y
Y
Y
Y
Y
U
Y
Y
Y
Y
Low
Y
Y
Y
Y
N
Y
Y
Y
Y
U
U
U
Y
Y
Y
Y
Y
N
Moderate
Y
Y
Y
Y
N
N
Y
Y
U
U
U
Y
Y
Y
U
Y
Y
N
Moderate
Y
Y
Y
N
N
N
Y
N
N
N
N
N
Y
U
N
Y
Y
N
High
Y
Y
Y
Y
N
N
Y
N
N
N
N
N
Y
Y
N
Y
Y
Y
High
Y
Y
Y
Y
N
Y
Y
N
Teng, 2009)
(Johnston et
al., 2009)
(Taheri et al.,
2008)
(Davis et al.,
2008)
(Lee et al.,
2007)
(Davis et al.,
2007)
(Schilling et
al., 2006)
(Krueger,
2006)
(Bolboaca and
Jantschi,
2006a)
(Weberschock
Ahmadi et al
46
Teaching EBM to Medical Students
et al., 2005)
(Gruppen et
N
N
Y
U
N
U
N
Y
Y
N
High
Y
Y
Y
Y
N
N
Y
Y
Y
Y
U
N
Y
Y
U
Y
Y
Y
Low
Y
N
Y
Y
N
Y
Y
Y
N
N
N
N
Y
Y
N
Y
Y
N
High
Y
Y
Y
Y
N
N
Y
N
N
N
U
U
U
Y
U
Y
Y
Y
High
Y
N
N
Y
N
N
Y
N
N
N
N
N
U
N
N
Y
Y
N
High
Y
Y
Y
Y
N
N
Y
Y
Y
Y
Y
Y
Y
Y
U
Y
Y
Y
Low
Y
Y
Y
Y
N
Y
Y
Y
N
N
N
N
N
N
N
Y
Y
Y
High
Y
Y
Y
Y
N
N
Y
N
Y
U
U
U
Y
Y
Y
Y
Y
N
Moderate
Y
Y
Y
Y
N
N
Y
Y
N
N
U
N
Y
N
Y
Y
Y
N
High
Y
Y
Y
Y
N
Y
U
N
al., 2005)
(Bradley et
al., 2005)
(Alper and
Vinson, 2005)
(SanchezMendiola,
2004)
(Dorsch et al.,
2004)
(Leung et al.,
2003)
(Fritsche et
al., 2002)
(Rosenberg et
al., 1998)
(Landry et al.,
1994)
Ahmadi et al
(Frasca et al.,
47
Teaching EBM to Medical Students
N
N
Y
N
U
U
Y
Y
Y
N
High
Y
Y
Y
Y
N
N
Y
N
N
N
N
U
Y
Y
Y
Y
Y
N
High
Y
Y
Y
Y
N
N
Y
Y
N
N
U
U
N
U
Y
Y
Y
N
High
Y
Y
Y
N
N
N
U
N
1992)
(Bennett et al.,
1987)
(Radack and
Valanis, 1986)
1
Reporting results in adequate details: Reporting numerical results for educational significance/effect size and measures of
dispersion/confidence intervals/P values; 2 N: No; 3 Y: Yes; 4 U: Unclear.
Ahmadi et al
48
Teaching EBM to Medical Students
FIGURES
Figure 1: Flow diagram of study selection.
8,708 Records identified through
database searching
1,513 Medline
645 ERIC
3,514 Scopus
817 CINAHL
2,166 Web of Science 53 Current Controlled Trials
1,403 Additional records identified through
other sources
346 Reference checking of relevant reviews
629 Reference checking of included studies
428 Citation checking of included studies
10,111 Records before removing duplicated records
4,638 Duplicated records removed
5,473 studies screened
by title/abstract
634 studies screened
by full-text article
27 Studies included in
systematic review
4,839 studies excluded
Due to not reporting teaching EBM to
undergraduate medical students
607 studies excluded
Due to:
1 duplicated results
7 no available full-text article
228 no report of an original study
221 not recruited medical students
80 not executed relevant teaching
70 not assessed objective outcomes