Formulating Research Questions and Designing Studies 2 March 2011 Facilitator: Larry Gruppen, Ph.D. University of Michigan Development team: Carol Hodgson, LuAnn Wilkerson, David Irby, Judy Shea, Clair Kuykendall, Larry Gruppen Preview • Moving from general issues to specific research questions • Using the literature to refine the question • Measuring the variables • Selecting the right research design The Research Process Define Research Question No hole in the literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results Hole in literature Recognizing & Choosing Among Research Opportunities • What situations or problems tend to puzzle, fascinate, challenge, or interest you? • List as many research questions as you can as quickly as possible • Identify which are: • - Most interesting (I) • - Feasible (F) • - Fundable ($) • - Best overall (*) • Rewrite your best research question at least twice The Research Question • Based on literature/theory • Includes sample description (e.g., 4th-year • • • • medical students) Includes study design (e.g., relationship, difference between groups, etc.) Includes the independent & dependent variables Is measurable Stated as a question or hypothesis Definitions • • • • • Variable Independent and dependent variables Covariates Operationalization and measurement Sample and population Example Research Question • Does training in airway management on a simulator improve resident performance (correct selection of equipment, # of errors, time to intubation), compared with standard didactic/demonstration instruction. Example Research Question • • • • • What is the independent variable? What is the dependent variable? What are possible covariates? What is the population and sample? What is your design? Example Research Question • • • • • Do first-year medical students who complete a student-run anatomy review course score higher on the anatomy final exam than students who do not complete the review course? What is the independent variable? What is the dependent variable? What are possible control variables? What is the sample? What is your design? The Research Process Define Research Question No hole in the literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results Hole in literature Searching for Related Work • Identify possible sources of information • Colleagues and librarians • Databases, PUBMED, PSYCLIT, Science Citation Index • Journals, chapters, books, publications • Read critically and summarize • Citations referenced • Sample size • Study design and limitations • Overall conclusions What to Look For • • • • • • Conceptualization and theory How variables are defined & measured Sample size Study design and limitations Overall conclusions Citations referenced The Research Process Define Research Question No hole in the literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results Hole in literature Refining the Research Question • Share your best question • Critique each other’s research questions • Determine which is the best/most interesting question • Refine this group question based on critique • Write final version of question Critique Your Question • • • • • • Clearly stated? Stated as a question? Testable? Defines variables to be studied? Defines sample to be studied? Describes the setting for the study? The Research Process Define Research Question No hole in the literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results Hole in literature Operationalization and Measurement • Three basic questions • What do you measure? • How do you measure? • How well do you measure? What do you measure? • Defined by research question • Dependent, independent variables • Covariates How do You Measure? • Operationalization is essential to the conduct of the study • Counting events • Measuring time and physical quantities • Externalizing internal (psychological) states, events, and processes Measurement Instruments • Knowledge • Tests (MCQ, essay, oral) • Attitudes • Questionnaires, surveys • Behavior or performance (skills) • Clinical exams (OSCE), standardized patients, direct observation • Checklists, ratings, error rates How Well do You Measure? • Reliability — score accuracy or stability • Would the score be reproduced if tested again? • Would the score be reproduced by different raters? • Validity — score meaning • Does the score measure what you intend to measure? The Research Process Define Research Question No hole in the literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results Hole in literature Qualitative Research What is the phenomenon? Use When • Focus on meaning and context • In-depth recording and triangulation • Inductively derived interpretation Methods • Interview • Observation • Think aloud, stimulated recall • Chart review • Surveys Correlational Research Use When • Predictors can’t be randomized • Subjects &/or treatments not controllable • Control groups not available Methods • Surveys • Chart review • Archived data All data is confidential Experimental and QuasiExperimental Research Use When • Specific intervention • Temporal relationship • Feasible explanatory mechanism • Subjects and treatments controlled Methods • Control over treatment and measurement • Randomization • Control Groups Defining Your Sample • Target population • Accessible population • Intended sample • Inclusion and exclusion criteria • Availability • Time frame • Willingness to participate • Intended variables • Actual sample Research Designs Quasi Experimental • One shot case study X -----O • One group Pretest-Posttest O-----X-----O (O=Observation X=Treatment) Research Designs Quasi-Experimental • Post-test only control group X---------O -----------O • Pre-test/Post-test Control Group O--------X-------O O-----------------O Research Designs Experimental • Post-test only control group (Randomized) X---------O -----------O • Pre-test/Post-test Control Group (Randomized) O--------X-------O O-----------------O • Solomon Four Group Design (Randomized) (1) O------X------O (2) O--------------O (3) ------X------O (4) --------------O R = randomization R R R Threats to Internal Validity • • • • • • • History Maturation Repeated measurement Statistical regression Selection Loss of Subjects/mortality Investigator bias External Validity • Is the sample representative of the population? Can the study be generalized to the population? • Are the conditions the same? For example, laboratory setting versus natural setting. • Did the subjects act differently because they were subjects in the study (Hawthorne Effect)? The Research Process Define Research Question No hole in the literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results Hole in literature Is Your Study Research? • Research means a systematic investigation including research, development, testing, and evaluation to develop or contribute to generalizable knowledge • If you might publish the results, it’s research AND NEEDS IRB REVIEW • What about quality or evaluation studies? Issues for the IRB • • • • • • Data normally collected in your course? Data collected specifically for your study? Will the data be anonymous? Will you use an any identifier? Is it linked? Do you need an informed consent? What kind of review? Exempt, Expedited, Full The Research Process Define Research Question No hole in the literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results Hole in literature Analyzing Data • Get a consultation on the analysis • Even if you know biostatistics - ed research uses some different methods • Do it before you start the study! Analyzing Data • Where to look for consultations • Department - colleagues who have done this before • Medical school - Dean’s office, ed research unit • University - school of ed, public health, psychology, educational support centers The Research Process Define Research Question No hole in the literature Conduct Literature Review Refine Question Operationalize variables Design Study Obtain IRB Approval Collect & Analyze Data Write and Report Results Hole in literature Publishing Med Ed Research • Emergency Medicine journals & meetings • General Medical Education journals & meetings • Academic Medicine, Medical Education, Teaching and Learning in Medicine, Advances in Health Sciences Education, Medical Teacher, … • Assoc. of American Medical Colleges, Ottawa Conference, Assoc. for Medical Education in Europe, ACGME, and many more Research That Makes a Difference • Investigates important questions • Connects study to prior research and to theory • Uses appropriate research design & data analysis procedures • Disseminates results Planning Next Steps • Create a project plan: • • • • • tasks & deadlines Find collaborators, mentors, consultants Search for funding Protect time for research Keep a research journal Have fun References • Bass, Dunn, Norton, Stewart, & Tudiver. (1993). Conducting Research in the Practice Setting. Newbury Park, CA: Sage. • Campbell & Stanley (1963). Experimental and Quasi-experimental Designs for Research. Dallas: Houghton Mifflin. • Glesne & Peshkin (1992). Becoming Qualitative Researcher: An Introduction. Longman. • Hulley & Cummings (1988). Designing Clinical Research. Baltimore: Williams & Wilkins.