Evaluating Tests --- Some Questions to Ask
General Information --• What is the purpose of the test?
• For what population is the designed? Is this population relevant to the
people who will take your test?
• What special qualifications are needed (if any) to acquire the test?
Test Content and Scoring --• How many questions exist on the test? What is the format of the test
questions? Is the format and number of questions sufficient for your needs?
Is the difficulty level and nature of the questions appropriate for your
sample?
• What scores are yielded by the test?
• How are test results reported?
• What materials are available to aid in interpreting test results (e.g., narrative
summaries, graphs, charts)?
Evaluating Tests --- Some Questions to Ask (cont.)
Test Administration ---
• How is the test administered?
• Are the test instructions clear?
• Are the scoring procedures clear and appropriate for your use?
• What time constraints (if any) exist in the administration of the test?
• Are the test materials (e.g., test booklet, manual, appropriateness for
examinees) sufficient for your purposes?
• What special training is needed (if any) to administer the test?
Evaluating Tests --- Some Questions to Ask (cont.)
Norm/Validation Sample --• What is the composition of the sample(s) used in validating and
establishing norms for the test (e.g., race, age, gender, SES)? Is the
composition of the sample relevant to your test takers?
• What data is available for the normative sample (e.g., percentiles, standard
scores)?
• How many people were in the validation and normative sample(s)? Is the
sample size large enough to yield accurate data and interpretations (e.g.,
standard error of measurement?
Evaluating Tests --- Some Questions to Ask (cont.)
Test Reliability --• What is the reliability of the test and any subscales? What reliability
techniques were used? Were these techniques appropriately applied?
• Do reliability estimates exist (and are different) for any sub-groups?
• Is the reliability of the test high enough for the purpose in which the
test is going to be used being used?
Evaluating Tests --- Some Questions to Ask (cont.)
Test Validity: Content-Related Validity --• What is the domain (e.g., skills, ability, traits) that is being assessed by the
test? How was the test content collected and analyzed? What data was
used to determine the appropriateness of the test content?
• What was the composition of SMEs (if any) used in the selection of test
content? How was the data from theses experts collected and analyzed?
•
Is the content of the test related to the qualifications needed by sample to
which the test is being given?
Evaluating Tests --- Some Questions to Ask (cont.)
Test Validity: Criterion-Related Validity --• What criterion measure (e.g., performance measure) was used to assess the
concurrent or predictive validity of the test? Why was the criterion chosen?
• What was the size and nature of the sample used in the validation of the
test? Does restriction of range exist in the criterion measure (s)?
• What is the criterion-related validity of the test (e.g., the correlation
between test and criterion scores? Is the level of this correlation sufficient
for your purposes?
Test Validity: Construct Validity --• Is the conceptual framework for the test stated clearly? Is the construct, as
defined and measured, relevant for your purposes?
Evaluating Tests --- Some Questions to Ask (cont.)
Test/Item Bias --1. Were the items analyzed statistically for possible sub-group bias? What
method(s) was used? How were items selected for inclusion in the final
version of the test?
2. Was the test analyzed for differential validity across groups? How was this
analysis conducted?