Formulating Research
Questions and Designing
Studies
2 March 2011
Facilitator: Larry Gruppen, Ph.D.
University of Michigan
Development team: Carol Hodgson, LuAnn Wilkerson, David Irby, Judy Shea,
Clair Kuykendall, Larry Gruppen
Preview
• Moving from general issues to specific
research questions
• Using the literature to refine the question
• Measuring the variables
• Selecting the right research design
The Research Process
Define Research Question
No hole in
the literature
Conduct Literature Review
Refine Question
Operationalize variables
Design Study
Obtain IRB Approval
Collect & Analyze Data
Write and Report Results
Hole in
literature
Recognizing & Choosing Among
Research Opportunities
• What situations or problems tend
to puzzle, fascinate, challenge,
or interest you?
• List as many research questions
as you can as quickly as
possible
• Identify which are:
• - Most interesting
(I)
• - Feasible
(F)
• - Fundable
($)
• - Best overall
(*)
• Rewrite your best research
question at least twice
The Research Question
• Based on literature/theory
• Includes sample description (e.g., 4th-year
•
•
•
•
medical students)
Includes study design (e.g., relationship,
difference between groups, etc.)
Includes the independent & dependent variables
Is measurable
Stated as a question or hypothesis
Definitions
•
•
•
•
•
Variable
Independent and dependent variables
Covariates
Operationalization and measurement
Sample and population
Example Research Question
• Does training in airway management on a
simulator improve resident performance
(correct selection of equipment, # of
errors, time to intubation), compared with
standard didactic/demonstration
instruction.
Example Research Question
•
•
•
•
•
What is the independent variable?
What is the dependent variable?
What are possible covariates?
What is the population and sample?
What is your design?
Example Research Question
•
•
•
•
•
Do first-year medical students who complete a
student-run anatomy review course score higher
on the anatomy final exam than students who do
not complete the review course?
What is the independent variable?
What is the dependent variable?
What are possible control variables?
What is the sample?
What is your design?
The Research Process
Define Research Question
No hole in
the literature
Conduct Literature Review
Refine Question
Operationalize variables
Design Study
Obtain IRB Approval
Collect & Analyze Data
Write and Report Results
Hole in
literature
Searching for Related Work
• Identify possible sources of information
• Colleagues and librarians
• Databases, PUBMED, PSYCLIT, Science
Citation Index
• Journals, chapters, books, publications
• Read critically and summarize
• Citations referenced
• Sample size
• Study design and limitations
• Overall conclusions
What to Look For
•
•
•
•
•
•
Conceptualization and theory
How variables are defined & measured
Sample size
Study design and limitations
Overall conclusions
Citations referenced
The Research Process
Define Research Question
No hole in
the literature
Conduct Literature Review
Refine Question
Operationalize variables
Design Study
Obtain IRB Approval
Collect & Analyze Data
Write and Report Results
Hole in
literature
Refining the Research
Question
• Share your best question
• Critique each other’s research questions
• Determine which is the best/most
interesting question
• Refine this group question based on
critique
• Write final version of question
Critique Your Question
•
•
•
•
•
•
Clearly stated?
Stated as a question?
Testable?
Defines variables to be studied?
Defines sample to be studied?
Describes the setting for the study?
The Research Process
Define Research Question
No hole in
the literature
Conduct Literature Review
Refine Question
Operationalize variables
Design Study
Obtain IRB Approval
Collect & Analyze Data
Write and Report Results
Hole in
literature
Operationalization and
Measurement
• Three basic questions
• What do you measure?
• How do you measure?
• How well do you measure?
What do you measure?
• Defined by research question
• Dependent, independent variables
• Covariates
How do You Measure?
• Operationalization is essential to the
conduct of the study
• Counting events
• Measuring time and
physical quantities
• Externalizing internal
(psychological) states,
events, and processes
Measurement Instruments
• Knowledge
• Tests (MCQ, essay, oral)
• Attitudes
• Questionnaires, surveys
• Behavior or performance (skills)
• Clinical exams (OSCE), standardized
patients, direct observation
• Checklists, ratings, error rates
How Well do You Measure?
• Reliability — score accuracy or stability
• Would the score be reproduced if tested
again?
• Would the score be reproduced by
different raters?
• Validity — score meaning
• Does the score measure what you
intend to measure?
The Research Process
Define Research Question
No hole in
the literature
Conduct Literature Review
Refine Question
Operationalize variables
Design Study
Obtain IRB Approval
Collect & Analyze Data
Write and Report Results
Hole in
literature
Qualitative Research
What is the phenomenon?
Use When
• Focus on meaning
and context
• In-depth recording
and triangulation
• Inductively derived
interpretation
Methods
• Interview
• Observation
• Think aloud,
stimulated recall
• Chart review
• Surveys
Correlational Research
Use When
• Predictors can’t be
randomized
• Subjects &/or
treatments not
controllable
• Control groups not
available
Methods
• Surveys
• Chart review
• Archived data
All data is
confidential
Experimental and QuasiExperimental Research
Use When
• Specific intervention
• Temporal relationship
• Feasible explanatory
mechanism
• Subjects and
treatments controlled
Methods
• Control over
treatment and
measurement
• Randomization
• Control Groups
Defining Your Sample
• Target population
• Accessible population
• Intended sample
• Inclusion and exclusion criteria
• Availability
• Time frame
• Willingness to participate
• Intended variables
• Actual sample
Research Designs
Quasi Experimental
• One shot case study
X -----O
• One group Pretest-Posttest
O-----X-----O
(O=Observation X=Treatment)
Research Designs
Quasi-Experimental
• Post-test only control group
X---------O
-----------O
• Pre-test/Post-test Control Group
O--------X-------O
O-----------------O
Research Designs
Experimental
• Post-test only control group (Randomized)
X---------O
-----------O
• Pre-test/Post-test Control Group (Randomized)
O--------X-------O
O-----------------O
• Solomon Four Group Design (Randomized)
(1) O------X------O
(2) O--------------O
(3) ------X------O
(4) --------------O
R = randomization
R
R
R
Threats to Internal Validity
•
•
•
•
•
•
•
History
Maturation
Repeated measurement
Statistical regression
Selection
Loss of Subjects/mortality
Investigator bias
External Validity
• Is the sample representative of the
population? Can the study be generalized
to the population?
• Are the conditions the same? For
example, laboratory setting versus natural
setting.
• Did the subjects act differently because
they were subjects in the study
(Hawthorne Effect)?
The Research Process
Define Research Question
No hole in
the literature
Conduct Literature Review
Refine Question
Operationalize variables
Design Study
Obtain IRB Approval
Collect & Analyze Data
Write and Report Results
Hole in
literature
Is Your Study Research?
• Research means a systematic
investigation including research,
development, testing, and evaluation to
develop or contribute to generalizable
knowledge
• If you might publish the results, it’s
research AND NEEDS IRB REVIEW
• What about quality or evaluation studies?
Issues for the IRB
•
•
•
•
•
•
Data normally collected in your course?
Data collected specifically for your study?
Will the data be anonymous?
Will you use an any identifier? Is it linked?
Do you need an informed consent?
What kind of review? Exempt, Expedited,
Full
The Research Process
Define Research Question
No hole in
the literature
Conduct Literature Review
Refine Question
Operationalize variables
Design Study
Obtain IRB Approval
Collect & Analyze Data
Write and Report Results
Hole in
literature
Analyzing Data
• Get a consultation on the analysis
• Even if you know biostatistics - ed
research uses some different methods
• Do it before you start the study!
Analyzing Data
• Where to look for consultations
• Department - colleagues who have done
this before
• Medical school - Dean’s office, ed
research unit
• University - school of ed, public health,
psychology, educational support centers
The Research Process
Define Research Question
No hole in
the literature
Conduct Literature Review
Refine Question
Operationalize variables
Design Study
Obtain IRB Approval
Collect & Analyze Data
Write and Report Results
Hole in
literature
Publishing Med Ed Research
• Emergency Medicine journals & meetings
• General Medical Education journals & meetings
• Academic Medicine, Medical Education,
Teaching and Learning in Medicine,
Advances in Health Sciences Education,
Medical Teacher, …
• Assoc. of American Medical Colleges, Ottawa
Conference, Assoc. for Medical Education in
Europe, ACGME, and many more
Research That Makes a
Difference
• Investigates important questions
• Connects study to prior research and to
theory
• Uses appropriate research design & data
analysis procedures
• Disseminates results
Planning Next Steps
• Create a project plan:
•
•
•
•
•
tasks & deadlines
Find collaborators,
mentors, consultants
Search for funding
Protect time for
research
Keep a research
journal
Have fun
References
• Bass, Dunn, Norton, Stewart, & Tudiver. (1993).
Conducting Research in the Practice Setting.
Newbury Park, CA: Sage.
• Campbell & Stanley (1963). Experimental and
Quasi-experimental Designs for Research.
Dallas: Houghton Mifflin.
• Glesne & Peshkin (1992). Becoming Qualitative
Researcher: An Introduction. Longman.
• Hulley & Cummings (1988). Designing Clinical
Research. Baltimore: Williams & Wilkins.