Blog Three – Reading Research – Application – Due: Saturday, January 26th
QUANTITATIVE RESEARCH
The Reader’s Companion Worksheet
Article Title:
A Controlled Comparison Study of the Efficacy of Training Medical Students in Evidence-Based Medicine
Literature Searching Skills:
Author:
Larry D. Gruppen, PhD., Gurpreet K. Rana, MLIS, and Theresa S. Arndt, MLIS
Title:
Topic of Interest:
Yes
Method of Interest:
Yes
Population of Interest: Yes
Abstract:
Results useful?
Yes
INTRODUCTION:
Why was the study done?
Done to study the effect of training medical students in evidence based medicine literature searching
skills.
What is the purpose of the study or what questions is the investigator trying to answer?
Purpose is to teach EBM searching skills to student, so they are able to efficiently search research
literature and to compare the intervention (effect) on students who had training to those who did not.
What are the central concepts and variables?
It is a comparison study to take 92 fourth year medical students from the University of Michigan Medical
School and see the effect of training searching skills on a certain number of students and see how it
effects the quality of their searches compared to students who do not have training.
The experimental (independent) variable is defined as:
A group of fourth year medical students from University of Michigan Medical School. Group of 92
students.
2
The Outcome variable is defined as:
Comparison study to see effective of training searching skills to not training searching skills to medical
students in fourth year.
Other variables that researcher has not thought about that might influence the results are:
1.
2.
3.
4.
5.
6.
Technical problems with the records of the literature
Incomplete or problems with the preintervention assessment
Mistakes by the user
Lack of appropriate limits, words
Failure to use Boolean operators correctly
Missing concepts
Note that sometimes there are many factors included in descriptive surveys and not a primary or
secondary outcome.
The students had to generate five clinical questions based on their own interests or experiences and
then had to investigate those questions the EBM techniques taught in the class. Each question was to
reflect a different type of research literature.
Are most of the references, recent (less than 5 years old)?
No
If not, is this a classic/groundbreaking reference or that has re-emerged in importance?
One that has remembered in importance – as technology increases and the use of online researching is
more available – it is important for students to know how to be literate in this regard.
Are experts cited?
No.
METHODS:
Design:
What is the research design? (e.g., survey, case-control, cohort study)?
A Controlled Comparison Study over a three year time period (2001, 2002, 2003)
3
Sample:
What are the characteristic of the participants who were included and excluded from the study (e.g.,
health status, age, education, gender, ethnicity, occupation, geographical residence, socioeconomic
status)?
Fourth year medical students at University of Michigan Medical School – one intervention group and
one control group.
Excluded:
Doesn’t say that any group was excluded from survey; however, the focus was on one study group only
(fourth year medical students who participated in a four week elective course in EBM)– no other
medical students in any other medical year were asked to participate.
Are the participants similar to those in your setting?
Yes/No/Somewhat
This question doesn’t apply.
Do you think the methods used to select participants for the study biased the results?
Selection Bias: Yes
If yes, why:
The methods were fair for selection. Students got to chose to be in the elective. However, some
students who elected to be involved in the comparison may have had lower levels of skill and
performance while others just wanted to improve their research skills. Also, some may already be skilled
in the proper research process.
Were there many refusals, withdrawals, dropouts, or deaths?
Participation bias:
Yes
There were some withdrawals of students’ strategies due to technical and preintervention assessment.
RESEARCH ETHICS CONCERNS
Was informed consent obtained?
Yes
Were the participants reasonably able to take part:
Yes
Was the study potentially/actually harmful to participants?
No
List any ethical issues that are of concern to you as a reader and potential user of the research.
4
None.
Setting:
What was the environment in which the data was collected?
Classroom setting.
Experiment:
No special treatment was received. The participants were aware of what they were doing in the class.
What methods, if any, were used to “blind” the participants, staff, and data collectors form
knowledge about the study that might influence the results?
None
Was there any contamination or mixing of treatments across the groups?
Not applicable
No other factors played a role.
DATA COLLECTION:
What was the data collection method?
4 Week elective course in EBM.
Were methods used to ensure that data were reliably collected (e.g., differences between raters,
differences between times of measurement)?
Yes. Two librarians evaluated student’s search results. They assigned scores through the application of
search strategy analysis instrument developed by librarians at the University of Michigan and based on a
template designed by librarians at the University of Rochester. The librarians developed search criteria
that were relevant to the clinical scenario and that reflected the instruction provided during the Medline
training session. Points were awarded for correct search strategies and points were deducted for using
inappropriate search limits and using incorrect search term (Gruppen, and et al 940-944).
Were methods used to ensure the validity of the collected data (e.g., expert review, comparison with
other measures)?
Yes. Librarians collected the data in accordance with a survey produced by the University of Rochester.
Do you think that the measurement methods biased the results?
No.
5
DATA ANALYSIS:
What statistical methods were used to analyze the data?
Points were collected for right answer and points were deducted for wrong answers based on the search
criteria. Scores were used to analyze the effectiveness of the comparison taking into account the errors
and correctness of the students searches.
RESULTS:
Was the response rate satisfactory? Yes
What are the main findings of the study?
Data for 30 intervention and 40 control students could be analyzed. Intervention students had fewer
search errors and higher quality searches than did control students. Educational intervention accounted
for about 8% of the variance in both of these outcomes. Most common search errors were a lack of
MESH explosion, missing MESH terms, lack of appropriate limits, failure to search for best evidence and
inappropriate combination of all search concepts (Gruppen, et al 940-944).
What information is presented in tables, figures, or graphs? Are these easy to understand or
confusing?
Data is set out in tables (Table 1 and Table 2).
Table 1 - Compared the results of the intervention group with the control group on a number of
different errors that occurred in the study. Then the mean difference was calculated to reflect the
variance of the results.
Also, in Table 2 it showed the effect of training in Evidence Based Medicine Search Strategies for both
the control group and intervention group pre and post intervention results and the overall change in the
effectiveness.
Were any of the findings statistically significant?
Yes– it said that there was an 8% variance of students who were trained properly would have better
search skills.
Were there clinically meaningful results/trends?
No.
DISCUSSION:
Do you agree with the investigator’s opinions? Yes
6
Opinion was that a single bride training session can have a marked beneficial effect on the quality of
short term search outcomes.
Are the results consistent with those of past research?
Findings are relevant to the many programs that seek to teach literature searching skills.
Does the interpretation make sense theoretically?
Yes.
Does the interpretation make sense clinically?
Not applicable.
OVERALL IMPRESSIONS:
Overall is the article important or significant for the practice to health care?
Somewhat.
Do the findings resonate (seem correct and familiar) with you and your practice?
Not applicable.
Major limitations of the study:
1. The selection of people used for comparison.
2. The amount of people that signed up for the study.
3. The length of time of the training.
Major strengths of the study:
1. Training does improve search skills in medical students.
2. The investigators are two librarians who are skilled in the search process and can analyze and
assess criteria.
3. The study was taken over a three year period allowing for better and more detailed results.
Are the results applicable or relevant to other settings, populations, or disciplines?
Yes. As stated in the results, “These findings are relevant to the many programs that seek to teach
literature searching skills to students and residents.” (Gruppen, and et al 940-944)
Gruppen, Larry D., et al. "A Controlled Comparison Study of the Efficacy of Training Medical Students in EvidenceBased Medicine Literature Searching Skills." Academic Medicine. Vol. 80.No. 10 (2005): 940-944. Web. 22 Jan.
2014.