You can do medical
education research!
Rachel Ellaway Ph.D.
Lisa Graves, MD, CCFP, FCFP
Assistant Dean Curriculum and Planning
Associate Dean Undergraduate Medical Education
Conflict of Interest
We have no financial interests,
arrangements, or affiliations that constitute a
direct or indirect conflict of interest in the
context or content of the subject of this
workshop.
Conflict of Interest
•
•
•
•
•
•
•
•
Intros 5
What is MER? 10
A1: Opportunities 15
Getting started 10
A2: Ask a good question 15
Designing a study 10
A3: sketch a study 15
Close 10
Teaching objectives
By the end of the session participants will be able to:
1. Differentiate between the many forms of
contemporary medical education research.
2. Identify needs and opportunities for undertaking
medical education research in their own practice.
3. Appraise the support and resources needed to
conduct a medical education research project and
how to obtain them.
4. Prepare a simple medical education research
plan, implement it and report on it.
Learning objectives
• Up to you but …
What is medical
education research?
Medical Education Research
“any investigation related to the education of
medical professionals, including research
related to undergraduate (medical school),
graduate (residency), and continuing medical
education”
Collins J (2008)
Medical Education
Research:
Challenges and
Opportunities.
Radiology, 240,
pp639-647
What is the focus?
Collins J (2008)
Medical Education
Research:
Challenges and
Opportunities.
Radiology, 240,
pp639-647
“medical education research can focus on any
number of topics, including curriculum
development, teaching methods, student
evaluation, teacher evaluation, course
evaluation, faculty development, admission
and preparation of candidates for medical
training, factors influencing career choice,
research methodology, and use of technology
in education”
What is researched?
• curriculum and teaching issues
• skills and attitudes relevant to the structure
of the profession
• individual characteristics of medical
students
• the evaluation of students and residents
• … and more
Acad Med. 2004 Oct;79(10):939-47. Trends in medical education
research. Regehr G.
Medical Education Research
• There are many forms of contemporary
medical education research, many schools
of thought
• Still a relatively underdeveloped field
• Critical to medical education
Why do it?
•
•
•
•
•
•
•
•
•
Evaluation
Quality assurance
Quality improvement
Increased effectiveness
Increased efficiency
Professional responsibility
Curiosity
Ultimately improving patient care
Accountability
Accountability
“medical education is accountable
through its research arm, where, in
principle, research evidence provides
the rationale for practices”
Bleakley, A, Bligh, J and Browne, J (2011). Medical Education for the Future:
Identity, Power and Location. Dordrecht, Germany, Springer.
Evidence Based Medical
Education
Best Evidence Medical Education (BEME) is
defined as "the implementation by teachers
and educational bodies in their practice, of
methods and approaches to education based
on the best evidence available”
BEME collaboration generating systematic
reviews
CEMESTR
http://www.bemecollaboration.org
Boyer's model of
scholarship
Boyer E. (1990).
Scholarship
Reconsidered: priorities
of the professoriate. The
Carnegie Foundation for
the Advancement of
Teaching, New York, NY.
1.
The scholarship of discovery that includes
original research that advances knowledge.
2.
The scholarship of integration -synthesis of
information across disciplines, across topics
within a discipline, or across time.
3.
The scholarship of application that goes
beyond the service duties of a faculty
4.
The scholarship of teaching and learning the systematic study of teaching and learning
processes
Needs and opportunities
Activity 1: needs and opportunities
Identify at least 1 need you have for medical
education research
Identify at least 1 opportunity you have to
conduct medical education research
Getting started
What is done?
• Interventions – efficacy and effectiveness
• Assessment – validation, reliability,
psychometrics
• Psychosocial dimensions
• Social dimensions
• Cultural dimensions
• Education systems
• Systematic reviews – BEME, CEMESTR
How is it done?
• Intervention vs no intervention •
•
•
•
•
Intervention #1 vs intervention #2
Pre-post
Psychometrics
Performance
Kirkpatrick
Kirkpatrick
1: Did the learners like the learning process?
2: What did they learn? How much did they
learn?
3: What performance changes resulted from
the learning process?
4: How has the learning process reduced cost,
improved quality, increased efficiency etc?
After: Kirkpatrick DL
(1994). Evaluating
Training Programs.
Berrett-Koehler
Publishers, Inc.
Kirkpatrick in MER
Care-focused impact:
Tochel C, Haig A,
Hesketh A, Cadzow
A, Beggs K, Colthart I
and Peacock H.
(1999) The
Effectiveness of
Portfolios for PostGraduate
Assessment and
Education: BEME
Guide No 12. Medical
Teacher, 31(4):299318.
K1
Participation
K2a
Modification of attitudes / perceptions
K2b
Modification of knowledge / skills
K3
Behavioural change
K4a
Change in organizational practice
K4b
Benefits to patient / clients
Kirkpatrick
Education-focused impact:
Steinert Y, Mann K,
Centeno A, Dolmans
D, Spencer J, Gelula
M and Prideaux D.
(2006) A systematic
review of faculty
development
initiatives designed to
improve teaching
effectiveness in
medical education:
BEME Guide No 8.
Medical Teacher,
28(6):497-526.
K1
Participants’ views
K2a
Change in attitudes
K2b
Modification of knowledge or skills
K3
Transfer of learning to the workplace
K4a
wider changes in the organization,
attributable to the educational program
K4b
improvement in student or resident
learning/performance
Question development
FINER criteria for a good research question:
F
I
N
Farrugia P, Petrisor
BA, Farrokhyar Fand
Bhandari M. Research
questions, hypotheses
and objectives. Can J
Surg. 2010 August;
53(4): 278–281
E
R
Feasible: adequate number of subjects,
adequate technical expertise, affordable in time
and money, manageable in scope
Interesting: getting the answer intrigues
investigator, peers and community
Novel: confirms, refutes or extends previous
findings
Ethical: amenable to a study that institutional
research ethics board will approve
Relevant: to scientific knowledge, to clinical and
health policy, to future research
Question format
PICOT criteria:
P
Population: what specific learner/teacher
population are you interested in?
I
Intervention (for intervention studies only)
What is your investigational intervention?
C
Comparison group: what is the main alternative
to compare with the intervention?
O
Outcome of interest: what do you intend to
accomplish, measure, improve or affect?
T
Time: what is the appropriate follow-up time to
assess outcome
Activity 2: ask a good question
Take your need or opportunity from the first
activity and create a good research question
to address it
Designing a study
The process of empiricism
•
•
•
•
•
Gorham, G
(2009) The
Philosophy of
Science: A
Beginner's
Guide. Oneworld.
Experience
Classification
Quantification
Discovery of relationships
Approximation to the truth
“the ultimate arbiter is not faith or utility or
logic, or even truth, but the empirical world
itself” Gorham, p52
Progression of enquiry
•
•
•
•
Observe the world
Identify possible phenomena
Describe in increasing breadth and depth
Identify relationships and differences within
the phenomena and with other phenomena
• Build models and develop theories and test
them
• Refine models and theories to become
predictive and to account for the phenomena
• … doesn’t necessarily start with numbers
Methods
• Quantitative – proofs, measurements
– Experimental – pre-post, trials, parallel
– Epidemiologic – RCTs, cohorts, randomization
– Psychometric – measuring individual knowledge,
abilities, study reliability, validity
– Correlational – questionnaires, surveys
Norman and Eva
(2010) Quantitative
Rsearch Methods in
medical education.
In Swanwick (Ed)
Understanding
Medical Education:
Evidence, Theory
and Practice. Wiley.
• Individuals vs groups
• Efficacy vs effectiveness
Methods
• Qualitative
–
–
–
–
–
–
–
–
Themes
Motives
Attitudes
Behaviours
Beliefs
Cultures
Ethics
Phenomenology
Methods
• Mixed methods
– QuanQuan, QualQual, QuanQual
– Different methods = different
questions/answers
– Holism and triangulation
– Sequences
• from what to why – survey > interview
• from why to what – interview > expt.
– Lenses, theories and more
Study design
•
•
•
•
Design the study
Create a protocol
Specific questions and rationale
Methods and participants follow questions
(and reality)
• Ethical limitations to educational research
• REB forms can help you design a protocol
Ethical review
•
•
•
•
Research Ethics Boards (REBs)
Not just about getting published
TCPS2 – research involving human subjects
… including autoethnography and
secondary data
• Lakehead ± Laurentian … others
• Using the REB process to your advantage
Activity 3: sketch a study
Take your research question and sketch out a
study to address it
Where next?
Resources?
• Monthly NOSM medical education journal
club – contact jdepatie@nosm.ca
• Conferences: AMEE, AAMC (Generalists,
RIME), CCME
• Calls for funding …
• Research Support Group
• Collaborations … let’s talk