Educational Research:
Getting Started
Steven B. Goldin
Making Life Better
through Research, Education & Healthcare
TM
Chief of Hepatobiliary and Pancreatic Surgery
Associate Professor of Surgery
Vice Chair of Surgical Education
Chief of Hepatobiliary and Pancreatic Surgery
NO DISCLOSURES EXCEPT:
Yes, they really are my photos
Outline
1.
2.
3.
4.
5.
6.
7.
Research: Why do it?
What is educational research
Research: How to get started
Features of rigorous studies
Pitfalls – validity and reliability
Surveys
Ethics, funding, resources
Research – why do it?
ASE – “No discipline can call itself a
profession that does not give top priority
to research. More specifically, the status
of a given profession depends on the
adequacy of the research activities, and
the application of research findings to
routine professional activities.” (Mouly
1978)
Research- why do it?
Faculty advancement
Teaching and research separate academic
surgeons from those in community
practice
Develop a national prominence/reputation
May be a funding source
Introduction
Importance of clinical productivity – how do you find the
time to do a research project?
Do research with your daily activities
Data collection is common for all courses and rotations.
medical students
residents
faculty
Innovation – what makes an academic center academic.
What additional data would need to be collected to begin
a research project?
Introduction
Innovation and evaluation form the backbone of
research ideas and projects
Many research projects are done with students,
residents, and faculty
Steady supply of potential participants
Introduction
Traditionally empirical research (bench top) was viewed as
the most valid method. Other forms of research (historical,
phenomenological, introspective, qualitative, etc.)
contribute to “educational research”
Empirical research is characterized by direct observation,
objectivity, falsification, replicability, parsimony and
quantification (scientific method)
Parsimony is defined as the adoption of the simplest
assumption in the formulation of a theory or in the
interpretation of data, especially in accordance with the
rule of Ockham's razor
Introduction
Occam’s razor (Ockham’s razor) - “entities should not be
multiplied unnecessarily.” The explanation of any
phenomenon should make as few assumptions as possible,
eliminating those that make no difference in the observable
predictions of the explanatory hypothesis or theory
When competing hypotheses are equal in other respects,
select the hypothesis that introduces the fewest
assumptions and postulates the fewest entities while
sufficiently answering the question
Evidence Based Medicine
Evidence based medicine – interventions should be guided
by data from rigorous studies was described at least 15
years ago
Evidence-Based Medicine (EBM) – excellent framework for
evaluating the quality of clinical research
EBM is not focused on educational research, but provides
useful criteria for different categories of research.
(Treatment efficacy, accuracy of diagnostic tests, clinical
guidelines, prognosis and risk, overviews and metaanalysis)
Evidence Based Medicine
Many EBM concepts are applicable to
educational research and should be
familiar to all educational researchers
Medical educators, likewise, should teach
based on evidence
Introduction
Educational research is driven by the process of
documenting the steps of a project
This process is the same for all studies including bench
research, studying teaching methods, or student attitudes
Educational research focuses on describing and
documenting the transmission of knowledge, competency,
and understanding about which teaching methods are
best and can lead to better practice
Making it Scholarly
All research involves – asking a question and obtaining an
answer in a systematic, thoughtful way
Scholarly teaching involves routinely asking questions
about daily educational activities
Research is answering these questions and scholarship is
the dissemination of the research results
Types of Studies
Descriptive: study describes what is going on or what exists
Relational: study looks at the relationships between two or
more variables
Causal: study is designed to determine whether one or
more variables causes or affects one or more outcome
variable
Research Steps
1. Ask the question – state clear goals (hypothesis
and alternate hypothesis)
2. Prepare (literature review, discussions,
conferences)
3. Specify the techniques or interventions
4. Test methods and ways of measuring impact
5. Pilot test the process and refine process
6. Select the best method to answer the question
Research Steps
7. Do the study
8. Capture and analyze data
9. Interpret findings and determine significance of
the outcome
10.Negative results can be meaningful depending
on the question
11.Reflective critique (thoughtful discussion, think
about the implications of the findings and develop
the next steps)
12.Disseminate the findings (article, abstract,
presentation, workshop)
Features of Rigorous Studies
Large numbers of student participants
Multi-institutional enrollment of participants
Having a control or appropriate comparison group
Measuring an objective outcome
Measuring some outcome at least 1 month after the intervention
Conduct the intervention more than once
Estimate statistical power
Features of Rigorous Studies
Measure objective outcomes including any evaluation other than selfassessment by students including:
Tests of knowledge
Course grades
Objective structured clinical examinations or observations of
standardized patient
Interaction assessment of performance with a real patient such as
graded observation of patient interactions or patient feedback
Objective clinical outcomes such as blood pressure control
Performance with mannequins, computer simulations, or
laboratory animals
Psychological inventories
Foundation
Language
Types of questions (descriptive, relational, causal)
Time in research (cross-sectional vs. longitudinal with
repeated measures or time series)
Types of relationships (correlational vs. causal)
Patterns of relationships (none, positive, negative)
Variables and attributes (independent and dependent,
exhaustive, mutually exclusive)
Hypothesis and null hypothesis
Types of data (quantitative vs. qualitative)
The Hypothesis
Theoretically significant research questions are different from
other types of questions.
A research question is not a question:
of fact. (How many residents passed the in-service or
their qualifying examinations?)
about a specific sample of subjects. (Do my students
perform better with open-ended rather than multiple choice
questions?)
about things that can not be measured. (Do lucky
students have more success than unlucky students?)
A good research question should:
1. seek new knowledge
2. use the scientific method
3. build upon existing scientific knowledge
4. pick up where existing theories and data end
5. lead to specific hypotheses which can be tested
6. have an answer deducible from the results and
provide information applicable not only to the
subjects studied, but to a broad, definable
population of subjects and settings
Reliability and Validity
Education research quality is linked to reliability
and validity, which are the two major criteria for
the quality of the measurement
Reliability involves the quality of measurement and
the consistency or repeatability of the measures
Theories of validity involve each step which are
interdependent
Reliability and Validity
Studies may be
reliable but not valid
valid but not reliable
neither reliable nor valid
both reliable and valid
Consulting with a research design and
statistics expert to avoid issues that may
impact study reliability or validity
General Surgeon Shortage
Surveys - References
Sage Publications, Inc; 2nd edition (November 12, 2002), 10 volumes, 1434 pages
Gall, Borg, and Gall. Educational Research: An Introduction (8th edition). Allyn
and Bacon. 2006
Surveys
One of the most important areas of measurement in applied
social research.
Two types
Questionnaires
Interviews
Construction of the Survey
Types of questions
Decisions about question content
Decisions about question wording
Decisions about response format
Question placement and sequence
Surveys
Often done because users think they are simple and quick ways to
publications
Survey design has fallen into disrepute among editors, reviewers and
consumers of medical research because they have been:
poorly designed
poorly developed
poorly administered
poorly analyzed
Reinforces the reconceived notion that surveys are not worth doing
Like experimental research, good survey research is very systematic and
rigorous
Surveys
The purpose of the study must be supported by a review of the
literature and understanding of the problem. Then develop the
research question, which will guide the development of the content.
If an item does not help you answer a research question, do not
include it on the survey
What information do you want from respondents?
attitudes/beliefs?
knowledge?
behavior?
attributes (demographic characteristics)?
Surveys
Structure the questions carefully
Unless the sole purpose of the survey is to solicit
suggestions, probe memories, clarify positions and or
vent frustrations, limit the number of open-ended
questions
Open ended questions make it difficult to construct
meaningful variables for statistical analysis, are difficult
to collate, and difficult to decipher
Surveys
A questionnaire should:
1. look attractive/appealing to complete
2. have questions laid out and be organized in such a manner as to make
them as easy to complete as possible
3. be organized with items appearing in some logical sequence
4. begin with those questions that the respondent is likely to see as useful
to the purpose of the study, or with those that are easy to answer and
non-threatening
5. place important items that are key to the study near the beginning
Items that are interesting and clearly relevant to the study will increase
response rate. Length has a small effect on response rate.
Pilot the instrument
Ethics
Principle of voluntary participation
Informed consent
Risk of harm – physical or psychological
Confidentiality
Principle of anonymity
Person’s right to service (no treatment control group)
Institutional Review Board (IRB)
Students and residents are considered a vulnerable
population
Social and Behavioral Institutional IRB vs. Biomedical IRB
Funding
Most surgical education research is conducted without the
benefit of research grants
ASE Foundation – CESERT program
(www.surgicaleducation.com)
National Board of Medical Examiners
Emergency Medicine Foundation
Agency for Health Care Research and Quality
Sages
American College of Surgeons (www.facs.org) lists funding
agencies that may be willing to fund educational research
Resources
ASE Educational Research Committee
Institutional Department of Medical Education
Departments of psychology, education, sociology, and
public health
Biostatistician
ASE – Surgical Education Research Fellowship Program
(SERF)
Conclusions
Clinical demands are taking on larger levels of importance
Protected time is a luxury
Research is required to advance academically
Educational research is a viable option even when faced
with high clinical demands
Conclusions
Many educational research projects have significant limitations
Insufficient sample sizes
Lack of generalizability when interventions are assessed
at only 1 institution or are conducted only once
Lack of appropriate control groups
Use of subjective and non-validated
instruments to assess outcomes
Assessing only short-term outcomes
Conclusions
Educational research challenges often relate to
measurement issues, bias and individual differences
An understanding of key educational concepts in
psychology, sociology, anthropology, psychometrics,
sociometrics, and statistics. Collaboration is key!
Assuming a significant question has been clearly
articulated, the quality of the study will depend upon the
research design, data analysis, interpretation and results
reporting
References
Research Methods Knowledge Base. William M.K. Trochim.
http://www.socialresearchmethods.net. Free on the
internet
Surgical Educators’ Handbook. Curriculum Committee and
Members of the Association for Surgical Education. ASE
website. http://www.surgicaleducation.com/mc/page.do
Educational Research:
Getting Started
Steven B. Goldin
Making Life Better
through Research, Education & Healthcare
TM
Chief of Hepatobiliary and Pancreatic Surgery
Associate Professor of Surgery
Vice Chair of Surgical Education
Chief of Hepatobiliary and Pancreatic Surgery