C RE SS T/U C LA
Usable Assessment
Knowledge: A Design
Problem
Eva L. Baker
UCLA Graduate School of Education & Information Studies
Center for the Study of Evaluation
National Center for Research on Evaluation, Standards, and Student Testing
With support from the Education Institute of Science,
U.S. Department of Education
International Congress for School Effectiveness and Improvement
January 6, 2003
Sydney, Australia
C RE SS T/U C LA
Knowledge to Support
Educational Improvement
Unusable
Plentiful
Usable
Scarce
Useful
Rare
C RE SS T/U C LA
Usable and Useful Knowledge
Usable Knowledge
 In a form that can be understood
 In a form that can be applied
 Timed appropriately
 May cause rethinking of the problem
Useful Knowledge
 Rethinking indicates a new solution path

Adapted to situation
 Sufficient to guide solution
 Improved outcomes occur as a result
C RE SS T/U C LA
Why Are Some Schools
Successful in Using
Knowledge?
 Focus on learning (students and adults)
 Constant use of appropriate information
(formal and informal)
 Focus on feedback and change
 Public display and exchange
 Pride in outcomes of students and place
 Select knowledge to foster these ends
C RE SS T/U C LA
Assessment Knowledge
in the Service of Reform
Usable?
Useful?
C RE SS T/U C LA
Assessment: Historical
Transformations
1. Assessment = measurement and
interpretation of a sample of performance
in the desired area of learning
(Useful)
2. Assessment (test) score = learning
(Usable)
 Activities not contributing should be
dropped
 Coincidence = causality
3. Assessment is the best intervention
 Cost/ Effectiveness
(Unusable)
C RE SS T/U C LA
Types of Assessment
Knowledge
 Purposes for assessment (Why? For whom?)
 What to assess
 Whom to assess
 How to assess (design and procedures)
 How to interpret and report results
 How to determine if results are trustworthy
(validity) for the purpose(s)
C RE SS T/U C LA
Key Design Principles for
Useful Assessment
 Assessment systems that start with
thinking skills and apply them to
content domains support
 Coherent, sustained learning
 Spiral teaching
 Transfer (application to new or unforeseen
situations)
C RE SS T/U C LA
CRESST Assessment Models
 Research-based
 Focus on cognition and learning
 Reusable and cost-sensitive
 Operationalized in models and templates
C RE SS T/U C LA
Model-Based Assessment
Families of Cognitive
Demands
Content
Understanding
Teamwork and
Collaboration
Communication
Learning
Problem
Solving
Metacognition
C RE SS T/U C LA
Usable Model-Based
Assessment Design
 Specifications and “Templates” for assessment
protocols, including scoring
 Templates that allow common design
approaches to be used, e.g., primary sources
 Two template examples for the model of deep
understanding of content
 Explanation
 Graphical representation of relationships
C RE SS T/U C LA
Specifications of Model for
Content Understanding
 Primary source materials in each
domain
 Student required to integrate prior
knowledge and principles to succeed
 Scored by using expert model in
subject matter
C RE SS T/U C LA
Content Understanding
Template #1
Explanation
 An array of primary source materials
 A prompt that asks for an explanation
in context
 Constructed (written) answer
 Evaluated by means of a scoring rubric
C RE SS T/U C LA
Hawaiian History Writing
Assignment: Bayonet
Constitution
Imagine you are in a class that has been studying Hawaiian history. One of
your friends, who is a new student in the class, has missed all the classes.
Recently, your class began studying the Bayonet Constitution. Your friend is
very interested in this topic and asks you to explain everything that you have
learned about it.
Write an essay explaining the most important ideas you want your friend to
understand. Include what you have already learned in class about Hawaiian
history, and what you have learned from the texts you have just read. While
you write, think about what Thurston and Liliuokalani said about the Bayonet
Constitution, and what is shown in the other materials.
Your essay should be based on two major sources:
1. The general concepts and specific facts you know about Hawaiian history,
and especially what you know about the period of the Bayonet Constitution.
2. What you have learned from the readings yesterday.
Be sure to show the relationships among your ideas and facts.
C RE SS T/U C LA
EXCERPTS from HAWAIIAN HISTORY
PRIMARY SOURCE DOCUMENTS
LILIUOKALANI
For many years our sovereigns had welcomed the advice of American
residents who had established industries on the Islands. As they became
wealthy, their greed and their love of power increased. Although settled
among us, and drawing their wealth from resources, they were alien to us
in their customs and ideas, and desired above all things to secure their own
personal benefit.
Kalakaua valued the commercial and industrial prosperity of his kingdom
highly. He sought honestly to secure it for every class of people, alien or
native. Kalakaua’s highest desire was to be a true sovereign, the chief
servant of a happy, prosperous, and progressive people.
And now, without any provocation on the part of the king, having matured
their plans in secret, the men of foreign birth rose one day en masse, called
a public meeting, and forced the king to sign a constitution of their own
preparation, a document which deprived [him] of all power and practically
took away the franchise from the Hawaiian race.
C RE SS T/U C LA
History Explanation
Scoring Rubric
 General Impression of Content Quality
 Principles or Concepts
 Prior Knowledge
 Use of Available Resources
 Misconceptions (negative)
 Argumentation (domain appropriate)
 English Mechanics
C RE SS T/U C LA
Mathematics Explanation
Task (9-year-olds)
Imagine a person from a television station has asked you to give a
demonstration on TV. You will be on a show to help other students learn
about math. You are asked to explain everything students your age should
know about fractions.
Below are some questions you should try to answer. These are
questions that students in the TV audience will ask you.
For each question you should draw as many pictures as you can to
show what you mean. Then write down what you would say about your
pictures on TV. Use as many words and pictures as you need.
What is a fraction? Why are there two numbers in a fraction? How
many fractions are there between 0 and 1? How many fractions are equal
to 1/2? What other important ideas should students know about fractions?
Show how you would explain these ideas. Use as many pictures and words
as you need.
C RE SS T/U C LA
Template #2
Knowledge Representation
 Key aspects of ideas, supporting facts
and views and their relationships
 Relationship is explicit
 Organizational options
 Core and peripheral
 Hierarchical
 Cause and effect
 Chronological
 Expert scoring
C RE SS T/U C LA
History
C RE SS T/U C LA
Genetics
C RE SS T/U C LA
Bicycle Pump
C RE SS T/U C LA
Key Design Principles from
Knowledge Requirements
 Knowing why
 Knowing what to assess: content plus
cognitive demands (problem solving,
communication, learning to learn,
teamwork, content knowledge)
 Knowing how: to make tests that
measure and support transfer
(application to other topics and
situations)
C RE SS T/U C LA
To Make Assessment
Knowledge Useful for
Educators
 Timing
 Having sufficient content knowledge
 Having sufficient knowledge of students
 Knowing how to combine results and other
sources of information
 Knowing where to find help and resources
 Knowing what to do (more than one option)
C RE SS T/U C LA
Data Interpretation—Design
Principles
 How to interpret results (in comparison
to what?)
 How to understand data from sources
other than classroom testing
 How to integrate sources of test results
and form ideas about what to do next
C RE SS T/U C LA
Decision Support System
: allows the integration of
coherent information
: allows the identification of
conflicting or discordant data
: requires the judgment of the
user to assign value
C RE SS T/U C LA
C RE SS T/U C LA
QSP
 Creates longitudinal records from external
sources
 Easy to read icons
 Grade books for teachers
 Digitized examples of student work
 Easy to use queries about relationships
 Parent conferencing material
C RE SS T/U C LA
Functions
Groups - for disaggregation
Reports - for displaying information graphically
Goals - for monitoring improvement
Gradebook - for keeping tabs on student progress
Digital Portfolio - for examples of student work
Resource Kit - to gather locally important information
(bottom-up)
C RE SS T/U C LA
Communication
The teacher decided to
send a QSP Progress
Report to the student’s
parents.
C RE SS T/U C LA
Using Assessments and
Reflecting on Data Is Hard
 Teachers: Fundamental shift from
chronological organization to
functional organization
 From what am I doing?
 To what should each learner be doing
now? Time and collaboration
 Administration: Taking a chance
on change
C RE SS T/U C LA
Context for Success of
Knowledge-Based Reform
 Local ownership of knowledge
 Infrastructure and stability
 Capacity to investigate
 Learning
 Congruence or peace with
external mandates
C RE SS T/U C LA
Usable Knowledge and
Support May Get to Useful
Knowledge
 For assessment knowledge to be
useful, it depends upon the context,
capacity, and communication of the
teaching system
 For assessment knowledge to be useful
to students, it must go to the heart of
why, what, and how they learn