Developing a Validated
Responsible Conduct of Research
Test Bank
Holly Phernetton, MPH, MS
James DuBois, PhD, DSc
Saint Louis University
Planned Collaboration with CITI
 With ORI funding, DuBois, Mumford,
Phernetton, Antes have developed a RCR
test bank:
 >125 multiple-choice items aimed at assessing
basic knowledge of RCR
 50 “pick 2” ethical decision-making items that
test ability to choose optimal responses to
ethical problems described in vignettes
Planned Collaboration with CITI
 All items will be available for CITI to use at
no cost
 This presentation describes how we have
already validated the test items and how we
propose to work with CITI for final
construct validation using classical test
theory
Background
 Office of Research Integrity (ORI)
identified 9 core areas that RCR courses
should address (2000)
 However, there was no guidance or consensus
on what content should be taught within these
core areas, nor what the overarching objectives
of RCR training should be assessed
 Further, there is no standard validated test that
measures RCR education objectives
Process for Test Bank
Development
 3 Stages of Test Development
1. Delphi Survey to validate objectives and
content areas
2. Development of the Items
 Multiple choice items
 Pick-2 items
3. Pilot-testing of items and calculation of
psychometric properties
Validation Process
 Content Validity
 Delphi survey: identify areas to assess
 Item development: evidence-based answers
 Expert review and revision
 Construct Validity
 Pilot-test items
 Classical test theory
Content Validity
 Level 1: Delphi survey
 Define objectives and determine important RCR
content to be taught and knowledge assessed by
consensus panel
 Level 2: Item development
 According to assessment recommendations from Delphi
panel
 Evidence-based answers from well-recognized RCR
curricula and text books
 Level 3: Expert review
Delphi Survey
 Funded by ORI
 2006-2007: four panels of experts convened
to develop a consensus: identify RCR
overarching goals for instruction and
assessment
 Delphi process: anonymous process
 Minimizes groupthink and domineering group
members
 Maximizes benefits of group decision-making
Delphi Survey
 Formed 4 separate panels: each worked independently
and simultaneously
 Survey conducted online in 3 rounds
 Round 1: Open-ended items asking what should be the
objectives and content of RCR training
 Round 2: Presented lists of edited responses from
round 1—participants asked to rate the importance of
teaching each topic
 Round 3: Presented lists of edited responses from
round 2—participants asked to rate importance of
teaching and assessing each area
Delphi Survey
 4 Panels
 1. Objectives (n=18)
 Overarching objectives and assessed objectives
 2. Scientific Data (n=13)
 Data management and research misconduct sections
 3. Scientific Relationships (n=14)
 Mentor-trainee responsibilities, collaborative science, conflicts
of interest and commitment
 4. Publication and Peer Review (n=13)
 Publication practices/responsible authorship and peer review
Delphi Survey Results
 7 Core Areas of RCR instruction examined
 Panel Consensus: 2/3 of panelists supporting
a rating of “important” or “very important”
 Consensus on teaching 43 main topics
 Consensus on assessing learning in 21 of the 43
main topics
Content Validity
 Level 1: Delphi survey
 Define objectives and determine important RCR content to be
taught and knowledge assessed by consensus panel
 Level 2: Item development
 According to assessment recommendations from Delphi panel
 Evidence-based answers from well-recognized RCR curricula and
text books
 Level 3: Expert review
Item Development
 Multiple Choice Questions (MCQ)
 Developed by Saint Louis University
(Dr. James DuBois, Holly Phernetton)
 Stem and lead-in question: 1 correct answer and 3
distractors
 Pick 2 Items (P2)
 Developed by University of Oklahoma
(Dr. Michael Mumford, Allison Antes)
 Stem: vignette presenting ethical problem
 Answer: pick 2 “optimal responses” from list of 8
options
MCQ Item Development
 Follow National Board of Medical
Examiner Guidelines
 Rationale for NBME guidelines. Good test
items:
 Discriminate between those who know and
those who do not know material
 Are constructed to reduce correct guessing by
“test savvy” participants
 Avoid being “tricky”
Rationale Behind the NBME
Guidelines
 Items contain only critically important content
 Must have 1 clearly best answer
 No negative questions such as
 “each of the following is correct except”
 “which of the following is NOT correct”
 No absolute terms such as “never” or “always”
 Avoid “all of the above” and “none of the
above answers”
 All responses are of roughly equal length and
form
Item Example: MCQ
A researcher commits research misconduct at a large
university which receives federal funding for research
from the Public Health Services. Which of the
following is correct regarding institutional obligations
in such cases?
A. Academic institutions are legally obligated to report all allegations of
misconduct to the Office of Research Integrity prior to investigation.
B. Institutions must have policies in place for the inquiry, investigation, and
adjudication of allegations of misconduct.
C. The federal government has to be immediately contacted in every case of
alleged misconduct.
D. Researchers found guilty of misconduct can never receive PHS funding
again.
Item Example: P2
Vignette:
Dr. Davis has been culturing a strain of Escherichia coli in an
oxygenated broth. For 12 years he has been gradually reducing
glucose concentration in the medium. As a mid-career
evolutionary biologist, he is interested in the possible evolution
of the strain over thousands of generations. He is targeting an
extremely unlikely adaptation-the ability to take up and
metabolize citrate, something that E. coli strains are notably
unable to do. After about 30,000 generations, the adaptation
appears. Indeed, the bacteria, which also shows some
morphological changes, can thrive in a totally glucose-free
medium, and phenotypes with the trait rapidly dominate the
population.
Item Example: P2I
Vignette, cont’d
An investigation revealed a new set of genes in Davis’ E. coli
culture consistent with contamination by the common soil
bacterium, Serratia marcascens. Yet, analysis could not identify
a single living S. marcescens in the culture. S. marcescens
readily metabolizes citrate, and the almost certain conclusion is
that his E. coli acquired the adaptive trait through lateral gene
transfer. This, itself, is an exciting finding, and Davis writes up
and distributes a report for prepublication review by the
research team. Two members object to Davis’ procedure
section where he says that the E. coli culture was
experimentally inoculated with S. marcescens. How should
Davis respond? Choose two from the following:
Item Example: P2I
 A. Remind them that this is his laboratory and he alone makes the final
decisions about the content of publications from his laboratory.
 B. Agree that the students have a good and valid point, but that the machinery
of science is intolerant of accidental findings.
 C. Explain to his students that the intention behind a finding is irrelevant to the
fact being presented.
 D. Revise the paper’s method section to explain how S. marcescens found its
way into the culture.
 E. Explain that full disclosure would only taint the reputation of the laboratory
by admitting the breach in original procedure.
 F. Listen to the objections of his students, accept that he is outvoted, and
change the procedure section of the paper.
 G. Do not offer any explanation other than a reminder that solid publications
are what make a career and that if they are uncomfortable with this, they
should consider leaving the lab.
 H. Acknowledge to the students that they are probably correct and use the data
as a basis for a follow-up study on lateral gene transfer under more tightly
controlled conditions.
Content Validity
 Level 1: Delphi survey
 Define objectives and determine important RCR content to be
taught and knowledge assessed by consensus panel
 Level 2: Item development
 According to assessment recommendations from Delphi panel
 Evidence-based answers from well-recognized RCR curricula and
text books
 Level 3: Expert review
Expert Review
 3 Experts to review RCR test items; provide
revision, comments
 MCQ (all three experts to review)
 P2 (one expert to review each area)
 Biological Sciences
 Social Science
 Health Science
Construct Validity
 Pilot-test Items
 CITI Program:
 Implement test bank
 Each item will be completed by a minimum of 200
CITI participants
 A participant will complete only 20% of total battery of
items: 5 MCQ per 7 core areas, 2-3 P2 vignettes with 3
items each (41-44 items total)
 Total of 1,000 CITI participants needed
CITI & Construct Validation
 CITI program pilot-testing
 All items will be pilot-tested on the CITI training
program
 While participants can re-take test until they pass, only
their initial response will be considered in construct
validation
 Will use classical test theory to validate items:
 Will identify top 20% and bottom 20% of test performers
 Any items missed more frequently by the top 20% than the
bottom 20% will be treated as invalid. Such items will be
discarded or revised
CITI Logistics
 CITI will upload test items using our Excel file
 Program test items to appear in random subsets
 After participants have completed all test items,
CITI will send de-identified data for statistical
analysis
 Test bank team will revise items and send CITI a
finalized test bank for use by Jan 2010
CITI—Ethical Issues
 Participants can re-take the test until passing score
achieved
 Most items tested are covered by CITI educational
content
 A few test items will be piloted of which are not
directly covered in the CITI training program
 These will not be scored, but feedback will be
provided
 This will ensure fairness and will provide
educational value
CITI—Ethical Issues
 Will obtain IRB permission
 CITI will provide participants with
introductory statement and request for
permission to use test data
 Our approach distinguishes between:
 Consent to take test (not needed, because testing is
a required component of training and items are
already content validated)
 Consent to use data (will be requested) in any
activities that extend beyond normal QI
Introductory Statement for
Participants
“Most of the test items you will see address material
directly covered in our training program. However,
some items also test your ability to generalize to
new ethical situations, and still others test your
ethical problem-solving skills. In each case, you
will be told the correct answer after you complete
an item and will have the chance to re-take the test
until you achieve a passing score.”
Upon Completion of the Test
“CITI is committed to quality improvement and
sometimes analyzes de-identified test and survey
data to identify ways to improve test items and
course content. We may also describe our quality
improvement efforts in publications or at
conferences using analyses of de-identified test data.
Do we have your permission to use your deidentified test data in this manner?
Yes
No