Assessment Literacy
Series
-Module 6Quality Assurance & Form
Reviews
1
Objectives
SAM-L Development participants will:
1.
2.
3.
Apply a quality assurance checklist to the
developed materials to ensure all documents
meet the requirements.
Verify the item is measuring the intended
content and standard.
Ensure the scoring key/rubrics are correct.
2
Helpful Tools
Participants may wish to reference:
Guides
• Handout #10 – Quality Assurance Checklist
• QA Screening Tool
• Performance Measure Rubric-Teacher
Other “Stuff”
•
“Smart Book”
3
Outline of Module 6
Module 6:
Form Reviews
Content
Review
Alignment
Rigor
Fairness,
Sensitivity,
Bias
Apply
Checklist
to all
forms
4
Form Review
Reviewing forms involves the following
components:






Content Review
Bias
Fairness
Sensitivity
Accessibility
Alignment
Note: Refer to “Smart Book” for detailed explanations of
these components.
5
Content Review
Determine if each item/task clearly aligns to the
targeted content standard.
 Evaluate all items for accuracy.
 Judge if each item is age and grade appropriate
for students.
• Reading level is appropriate.
• Vocabulary is appropriate.
• Required reasoning skills are appropriate.
 Review each item stem and answer options to be
sure they are clear and understandable.
• Student response expectations are clear.
6

Item Bias

Bias is the presence of some
characteristic of an item that results in
the differential performance of two
individuals with the same ability but
from different subgroups.

Bias is not the same as stereotyping.

Bias-free items/tasks provide an equal
opportunity for all students to
demonstrate their knowledge and skills.
7
Fairness and Sensitivity
Review to ensure:

Items are sensitive to different cultures,
religions, ethnic and socio-economic groups,
and disabilities.

There is a balance of gender roles.

Use of positive language, situations, and
images.

Items avoid text that may elicit strong emotions
in specific groups of students and prevent those
students from accurately demonstrating their
skills and knowledge.
8
Sensitive Topics to Avoid
•
•
•
•
•
•
•
•
Abortion
Birth control
Child abuse/neglect
Creationism
Divorce
Incest
Illegal activities
Occult/witchcraft
• Questioning parental
authority
• Rape
• Religious doctrine
• Sex/sexuality
• Sexual orientation
• Weight
• Suicide
• Sexually transmitted diseases
*Note: The above list does not contain every sensitive topic that item
developers should avoid and does not consider test-taker’s
age/maturity.
8
What is Accessibility?

Test accessibility is defined as the extent to which a test eliminates
barriers and permits the test-taker to demonstrate his or her
knowledge of the tested content.

All items should be reviewed to ensure they are accessible to the
entire population of students.

Item reviews must consider:
•
Readability
•
Syntax complexity
•
Item presentation
•
Font size
•
Clarity of images, graphs, tables
•
Amount of work space
10
Alignment

Categorical Concurrence
◦ The same categories of the content standards are included in the assessment
◦ Items might be aligned to more than one standard

Source of Challenge
◦ Students give a correct or incorrect response for the wrong reason (bias)

Balance of Representation
◦ Ensures there is an even distribution of the standards across the test

Range of Knowledge
◦ The extent of knowledge required to answer correctly parallels the knowledge
the standard requires

Depth of Knowledge
◦ The cognitive demands in the standard must align to the cognitive
demands in the test item
Source: Webb, N. L. (1997). Research Monograph No. 6: Criteria for alignment of expectations and
assessments in mathematics and science education. Washington, DC: Council of Chief State School
Officers
11
Quality Assurance
 QA Screening Tool
◦ Use to examine first draft materials
◦ Outlines key criteria
 Quality Assurance Checklist
[Handout #10]
◦ Use with final drafts
◦ Examines each developed component of the
performance measure.
 Performance Measure Rubric-Teachers
◦ Comprehensive self-assessment of teachermade performance measures.
12
Process Steps
1.
Apply the Quality Assurance Checklist [Handout #10] to the
performance measure and supporting documents. Apply the
checklist to each individual item /task, and then evaluate the
measure in its totality.
2.
Make notes directly on the paper copies of the documents.
3.
Review areas that may need additional focus.
4.
Print and conduct a final team review of all documents,
particularly the operational form and scoring keys.
5.
Turn your packet in to the facilitator, and destroy draft
material.
13
Summary
Module 6: Form Reviews
•
Evaluated the documents created by the
performance measure development process, and
ensured the measures were of high-quality, rigorous,
and aligned to the targeted content standards. All
documents went through a quality assurance
process prior to finalization.
14