Update on the Revisions to the Standards
for Educational and Psychological Testing
Antonio E. Puente, Ph.D.
University of North Carolina Wilmington
Joint Committee for the Revision to the Standards for
Educational and Psychological Testing
www.apa.org/science/programs/testing/standards.aspx
Mission & Brief History
• “The purpose for publishing the
•
standards is to provide criteria for the
evaluation tests, testing practices, and
the effects of tests use”. (1999)
Established in 1954; revised in 1955,
1966, 1999 and currently (2009-2011)
• American Psychological Association,
American Educational Research Association,
National Council on Measurement in
Education
Update on Revisions to the Test
4/9/2015
Standards
2
Update on the Revisions to the Standards
for Educational and Psychological Testing:
Overview of the Revision Process
Scope of the Revision
• Based on comments each organization
•
received from invitation to comment
Summarized by the Management
Committee in consultation with the CoChairs
• Wayne Camara, Chair, APA
• Suzanne Lane, AERA
• David Frisbie, NCME
4/9/2015
Update on Revisions to the Test
Standards
4
Joint Committee Members
• Lauress Wise, Co Chair, HumRRO
• Barbara Plake, Co Chair, University of
Neb.
• Linda Cook, ETS
• Fritz Drasgow, University of Illinois
• Brian Gong, NCIEA
• Laura Hamilton, Rand Corporation
• Jo-Ida Hansen, University on MN
• Joan Herman, UCLA
4/9/2015
Update on Revisions to the Test
Standards
5
Joint Committee Members
• Michael Kane, ETS
• Michael Kolen, University of Iowa
• Antonio E. Puente, UNC-Wilmington
• Paul Sackett, University of MN
• Nancy Tippins, Valtera Corporation
• Walter (Denny) Way, Pearson
• Frank Worrell, Univ of CA- Berkeley
4/9/2015
Update on Revisions to the Test
Standards
6
Five Identified Areas for the
Revisions
• Access/Fairness
• Accountability
• Technology
• Workplace
• Format issues
4/9/2015
Update on Revisions to the Test
Standards
7
Timeline
• First meeting January, 2009
• Three year process for completing text of
•
•
•
revision
Release of draft revision following
December 2010 JC meeting
Open comment/Organization reviews
Projected publication Summer, 2012
4/9/2015
Update on Revisions to the Test
Standards
8
Anticipated Review Process
• Draft released December 2010
• 90 Day (+/-) Review Period
• Public Portal for Submitting Comments
• Sponsoring Organizations with Publicize
•
Opportunity for Comments
Committee will Summarize Key Issues
from Reviews and Provide Response(s)
4/9/2015
Update on Revisions to the Test
Standards
9
Publication Options
• Management Committee responsibility
• Goal is for electronic access
• Pursuing options for Kindle, etc.
• Concerns about retaining integrity and
financial support for future revision
efforts
May 1, 2010
4/9/2015
Update on Revisions to the Test
Standards
10
Update on the Revisions to the Standards
for Educational and Psychological Testing:
Fairness for Diverse Populations
1999 Approach
• Standards related to fairness appear
•
throughout many chapters
Concentrated attention in:
• Chapter 7: Fairness in Testing and Test Use
• Chapter 8: Rights and Responsibilities of
Test Takers
• Chapter 9: Testing Individuals of Diverse
Linguistic Backgrounds
• Chapter 10: Testing Individuals with
Disabilities
4/9/2015
Update on Revisions to the Test
Standards
12
Committee Charge
•
•
•
Five elements of the charge focused on
accommodations/modifications
•
•
•
•
•
4/9/2015
Impact/differentiation of accommodation and
modification
Appropriate selection/use for English learners (ELs)
and examinees with disabilities (EwD)
Attention to other groups, e.g., pre-K, older
populations
Flagging
Comparability/validity of accommodated scores
One element focused on adequacy and
comparability of translations
One element focused on Universal Design
Update on Revisions to the Test
Standards
13
Revision Response
•
•
•
•
•
•
•
4/9/2015
Fairness is fundamental to test validity: include
as foundation chapter
Fairness and access are inseparable
Same principles of fairness and access apply
to all individuals and regardless of specific
subgroup
From three chapters to a single chapter that
describes core principles and standards
Examples drawn from ELs, EwD, and other groups
(young children, aging adults adults, etc)
Comments point to applications for specific groups
More particular standards retained where
appropriate (e.g., test translations)
Update on Revisions to the Test
Standards
14
Overview to Fairness Chapter
• Section I: General Views of Fairness
• Section II: Threats to the Fair and Valid
Interpretations of Test Scores
• Section III: Minimizing Construct Irrelevant
Components Through the Use of Test Design
and Testing Adaptations
• Section IV: The Standards
4/9/2015
Update on Revisions to the Test
Standards
15
Four Clusters of Standards
1.
2.
3.
4.
4/9/2015
Use test design, development administration and
scoring procedures that minimize barriers to valid
test interpretations for all individuals.
Conduct studies to examine the validity of test score
inferences for the intended examinee population.
Provide appropriate accommodations to remove
barriers to the accessibility of the construct
measured by the assessment and to the valid
interpretation of the assessment scores.
Guard against inappropriate interpretations, use,
and/or unintended consequences of test results for
individuals or subgroups.
Update on Revisions to the Test
Standards
16
Update on the Revisions to the Standards
for Educational and Psychological Testing:
Other Issues
Other Issues in Committee Charge
• Increased use of tests for accountability
• Impact of technology on testing
• Format and style issues
4/9/2015
Update on Revisions to the Test
Standards
18
Increased Use of Testing for
Accountability
•
Use of tests for accountability has expanded
•
•
•
•
Most notably in education but also in other areas such as
behavioral health
Facilitated by increasing availability of data and analysis tools
Recent and impending federal and state initiatives will likely
lead to further expansion
Use of test scores in policy and program evaluations
continues to be widespread
•
May 1, 2010
4/9/2015
Reinforced by groups that fund and evaluate research (e.g.,
IES, What Works Clearinghouse)
Update on Revisions to the Test
Standards
19
Organization of Accountability
Material
• Chapter on policy uses of tests focuses on use
of aggregate scores for accountability and
policy
• Chapter on educational testing addresses
student-level accountability (e.g., promotional
gates, high school exit exams) and interim
assessment
• Validity, reliability, and fairness standards in
earlier chapters apply to accountability testing
as well
May 1, 2010
4/9/2015
Update on Revisions to the Test
Standards
20
Impact of Technology
• Technological advances are changing the way
•
•
tests are delivered, scored, interpreted and in
some cases, the nature of the tests
themselves
The Joint Committee has been charged with
considering how technological advances
should impact revisions to the Standards
As with the other themes, comments on the
standards that related to technology were
compiled by the Management Committee and
summarized in their charge to the Joint
Committee
4/9/2015
Update on Revisions to the Test
Standards
21
Key Technology Issues Included in
our Charge
•
•
•
Reliability & validity of innovative item formats
Validity issues associated with the use of:
•
•
•
4/9/2015
Automated scoring algorithms
Automated score reports and interpretations
Security issues for tests delivered over the
internet
Issues with web-accessible data, including
data warehousing
Update on Revisions to the Test
Standards
22
Reliability & Validity of Innovative
Item Formats
•
•
•
What special issues exist for innovative items with
respect to access and elimination of bias against
particular groups? How might the standards reflect
these issues?
What steps should the standards suggest with regards
to “usability” of innovative items?
What issues will emerge over the next five years
related to innovative items/test formats that need to be
addressed by the standards?
4/9/2015
Update on Revisions to the Test
Standards
23
Automated Scoring Algorithms
•
•
•
What level of documentation/disclosure is appropriate
and tolerable for automated scoring
developers/vendors?
What sorts of evidence seem most important for
demonstrating the validity and “reliability” of
automated scoring systems?
What issues will emerge over the next five years
related to automated scoring systems that need to be
addressed by the standards?
4/9/2015
Update on Revisions to the Test
Standards
24
Automated Score Reports and
Interpretation
•
•
Use of computer for score interpretation
“Actionable” reports (e.g., routing students and
teachers to instructional materials and lesson
plans based on test results)
•
•
4/9/2015
Documentation of rationale
Supporting validity evidence
Update on Revisions to the Test
Standards
25
Format Issues
• Consideration of ways to identify of “Priority
•
•
Standards”
More parallelism between chapter
• Tone
• Complexity
• Technical language
Organization of Chapters
May 1, 2010
4/9/2015
Update on Revisions to the Test
Standards
26
Possible Ways to Identify “Priority
Standards”
• Clustering of Standards into thematic
•
•
topics
Over-arching Standards/ Guiding
Principles
Application Chapters
• Connection of standards to previous
standards
May 1, 2010
4/9/2015
Update on Revisions to the Test
Standards
27
Organization of Chapters
• 1999 Testing Standards
• Three sections
• Foundation: Validity, Reliability, Test Development,
•
•
May 1, 2010
4/9/2015
Scaling & Equating, Administration & Scoring,
Documentation
Fairness: Fairness, Test Takers Rights and
Responsibilities, Disabilities, Linguistic Minorities
Applications: Test Users, Psychological,
Educational, Workplace, Policy
Update on Revisions to the Test
Standards
28
Possible New Organization
Section 1:
Core Principles
Validity, Reliability, Fairness
Testing Operations
Test Design and Development, Scaling &
Equating, Test Administration & Scoring,
Documentation, Test Takers, Test Users
Section 2
: Applications
Psychological, Educational, Workplace,
Update on Revisions to the Test
Policy
and
Accountability
May 1, 2010
4/9/2015
Standards
29
Additional Important Information
• Documentation
• Third Party Observers
• Standardized Administration/Scoring
• Technicians
• Supervision
• Diverse Populations
• Norms
4/9/2015
Update on Revisions to the Test
Standards
30
Feedback Requested
• Feedback Information
• Web-based
• Necessary due to its impact
• Feedback Timetable
• Comment period• Revision period-
Publication expected- 2012 (APA)
4/9/2015
Update on Revisions to the Test
Standards
31