KNR 497
KNR 497: Practice Questions – Controlling for Threats to Validity
Below are a set of research scenarios. Go through each scenario to discuss the potential threats to validity, and
discuss how the study might be better designed to help control for these threats.
1. A researcher wants to examine hyperactivity among children. The researcher randomly establishes a
control group and an experimental group and tests each participant for hyperactivity in the morning
and in the afternoon. This continues daily for three months.
History –
Selection –
Statistical regression –
Pre-testing Instrumentation Attrition Experimenter Subject effect 2. An undergrad class taught by the researcher is split into 3 groups of 30. One third is assigned to a
plyometric strength-training program, 1/3 to an isometric program, and 1/3 do nothing. Before
assigning them, the researcher makes sure to tell the entire class about the purpose of the research,
and explains we are doing it to see if the researcher’s suspicions about plyometrics are correct.
History –
Selection –
Statistical regression –
Pre-testing Instrumentation Attrition Experimenter Subject effect -
KNR 497
3. A teacher wants to examine his students’ level of improvement during a three-week basketball unit. He
places them in two groups (high-skilled and low skilled) based upon their performance during the first
day of the unit. He then measures their performance against a pre-existing rubric each day of the unit.
History –
Selection –
Statistical regression –
Pre-testing Instrumentation Attrition Experimenter Subject effect 4. A researcher wants to examine the fitness levels of college students. She chooses to use the
students enrolled in a fitness class she teaches as her participants. Students are required to
participate in a set of fitness tests on the first day of the semester, in the middle of the semester,
and on the last day of the semester.
History –
Selection –
Statistical regression –
Pre-testing Instrumentation Attrition Experimenter Subject effect 5. A researcher wants to study the educational achievements of children as they progress from
kindergarten through twelfth grade. Students are placed into groups based upon the
socioeconomic status of their parents or guardians. Scores from standardized tests are obtained
and analyzed for each student during this time period.
History –
Selection –
Statistical regression –
Pre-testing Instrumentation Attrition Experimenter Subject effect -
KNR 497
Answers
KNR 497: Practice Questions – Controlling for Threats to Validity
Below are a set of research scenarios. Go through each scenario to discuss the potential threats to validity, and
discuss how the study might be better designed to help control for these threats.
1. A researcher wants to examine hyperactivity among children. The researcher randomly establishes a
control group and an experimental group and tests each participant for hyperactivity in the morning
and in the afternoon. This continues daily for three months.
History – Testing every day for 3 months; reduce # of times subjects are tested
Selection – Good that it is randomly selected
Statistical regression – Could occur. Taking enough measure to get it back to the mean
Pre-testing – no pre-test done
Instrumentation – need to be consistent in use of instrument. If observations are made, need consistency
Attrition – possible over 3 months
Experimenter – A relationship will build. If testing can be biased, control for this.
Subject effect – Can happen, but 3 months will likely reduce this factor.
2. An introductory wellness undergrad class taught by the researcher is split into 3 groups of 30. One
third is assigned to a plyometric strength-training program, 1/3 to an isometric program, and 1/3 do
nothing. Before assigning them, the researcher makes sure to tell the entire class about the purpose of
the research, and explains we are doing it to see if the researcher’s suspicions about plyometrics are
correct. Measures of their changes will be taken once a week for 6 weeks.
History – Illness, injury could influence results; pull them from the study if this happens
Selection – Need to randomly select people to the group. This class may attract people who are physically
active & unwilling to do nothing for 6 weeks if they are selected to that group.
Statistical regression – Will even out high/low testers
Pre-testing - NA
Instrumentation – Must ensure calibration of tests are accurate; researchers stay consistent
Attrition – 6 weeks may lose people; some may not complete the required exercises.
Experimenter – Researcher is their instructor.
Subject effect – Wellness class and the up-front purpose & suspicions of the researcher were made known.
People may seek to fulfill suspicions.
KNR 497
3. A teacher wants to examine his students’ level of improvement during a three-week basketball unit. He
places them in two groups (high-skilled and low skilled) based upon their performance during the first
day of the unit. He then measures their performance against a pre-existing rubric each day of the unit.
History – 3 week time span;
Selection – random sampling would increase validity but is prohibitive;
Statistical regression – Don’t place in groups after the first test
Pre-testing – maybe, depends on the test
Instrumentation – keep instrument consistent
Attrition - 3 weeks tolose people
Experimenter – teacher is measuring
Subject effect – May increase effect
4. A researcher wants to examine the fitness levels of college students. She chooses to use the
students enrolled in a fitness class she teaches as her participants. Students are required to
participate in a set of fitness tests on the first day of the semester, in the middle of the semester,
and on the last day of the semester.
History – Data collected over a semester;
Selection – random sampling would increase validity but is prohibitive;
Statistical regression – Could occur. Taking enough measure to get it back to the mean
Pre-testing - won’t improve from practicing test
Instrumentation – Keep instrument consistent
Attrition – Potential, 16 weeks of a class
Experimenter – Teacher is measuring, get someone else to help measure
Subject effect - Potential
5. A researcher wants to study the educational achievements of children as they progress from
kindergarten through twelfth grade. Students are placed into groups based upon the
socioeconomic status of their parents or guardians. Scores from standardized tests are obtained
and analyzed for each student during this time period.
History – High
Selection – random sampling would increase validity but is prohibitive;
Statistical regression – Could occur. Taking enough measure to get it back to the mean
Pre-testing - NA
Instrumentation – Standardized test will increase the validity
Attrition – High – 12 years;
Experimenter - NA
Subject effect - NA