C RE SS T/U C LA
Technology-Based
Assessment for
High-Performance
Learning
Eva L. Baker
UCLA Graduate School of Education & Information Studies
Center for the Study of Evaluation
National Center for Research on Evaluation, Standards, and Student Testing
International Congress for School Effectiveness and Improvement
January 2003
Sydney, Australia
C RE SS T/U C LA
Premises and Goals
Assessment is central to the effectiveness of
classroom and distance learning and
accountability—some basics
What are advances in assessment? How does
technology fit?
How can technology weave instructional and external
testing into a coherent system?
Can assessment be cost-sensitive and valid?
Many examples throughout
C RE SS T/U C LA
Order of Topics
Basics about assessment
CRESST research-based models
 Attributes, benefits
Examples
 Template 1: Paper and pencil
 Template 2: Computer
 Template 3: Authoring
Technology benefits and requirements
C RE SS T/U C LA
About Assessment
Role of external tests—accountability, evaluation,
and system monitoring
Instructional uses of tests in classrooms and
schools: diagnosis, modeling, formative
assessment as a core teaching strategy
Variations in curriculum require strands supporting
coherence among goals (standards), content,
cognitive demands (what thinking skills are
required?)
Coherence perceived from student, teacher,
administrator, and expert views
C RE SS T/U C LA
External
Exams
Instructional
Assessments
 Motivated
performance
 Embedded in
learning
 Time-sensitive
 Adapted to learners
 Standardized
 Extended time
 Shallow sampling
 Opportunity to revise
 Stand-alone
 Contextualized
results
C RE SS T/U C LA
External and Instructional
Assessments Must Be More
Coherent
Horizontally and vertically
Conceptual and psychological linkage
For accountability systems, classroom
measures may be good supplements
C RE SS T/U C LA
CRESST Assessment Models
Do not start with content
Focus on aspects of learning and assessment that
transfer from content to content area
Multi purpose: both formative learning and
outcomes
 Emphasize student-constructed answers
Expert performance defines scoring
Research substantiates these models across
different subjects and people
C RE SS T/U C LA
Families of Cognitive
Demands:
Model-Based Assessment
Content
Understanding
Teamwork and
Collaboration
Communication
Learning
Problem
Solving
Metacognition
C RE SS T/U C LA
CRESST Assessment Models
Research-based, and sites are exclusively
classrooms, schools, and systems
Focus on cognition and learning
Combine content-independent and contentdependent knowledge and strategies
Reusable and cost-sensitive
C RE SS T/U C LA
Examples in Knowledge
Understanding and Problem
Solving
Paper-pencil templates
Technology-based administration,
scoring, and reporting
Technology-supported authoring
templates and menus
For teachers, curriculum experts, test
makers
C RE SS T/U C LA
Assessment of
Understanding
Deep understanding of primary source
materials or key processes
Standard reading as part of task
Standard directions
Standard scoring rubrics based on
experts’ performance
C RE SS T/U C LA
Hawaiian History Assessment Task:
Bayonet Constitution
Imagine you are in a class that has been studying Hawaiian history. One of
your friends, who is a new student in the class, has missed all the classes.
Recently, your class began studying the Bayonet Constitution. Your friend is
very interested in this topic and asks you to explain everything that you have
learned about it.
Write an essay explaining the most important ideas you want your friend to
understand. Include what you have already learned in class about Hawaiian
history, and what you have learned from the texts you have just read. While
you write, think about what Thurston and Liliuokalani said about the Bayonet
Constitution, and what is shown in the other materials.
Your essay should be based on two major sources:
1. The general concepts and specific facts you know about Hawaiian history,
and especially what you know about the period of the Bayonet Constitution.
2. What you have learned from the readings yesterday.
Be sure to show the relationships among your ideas and facts.
C RE SS T/U C LA
Excerpts from Hawaiian History
Primary Source Documents
LILIUOKALANI
For many years our sovereigns had welcomed the advice of American
residents who had established industries on the Islands. As they became
wealthy, their greed and their love of power increased. Although settled
among us, and drawing their wealth from resources, they were alien to us
in their customs and ideas, and desired above all things to secure their own
personal benefit.
Kalakaua valued the commercial and industrial prosperity of his kingdom
highly. He sought honestly to secure it for every class of people, alien or
native. Kalakaua’s highest desire was to be a true sovereign, the chief
servant of a happy, prosperous, and progressive people.
And now, without any provocation on the part of the king, having matured
their plans in secret, the men of foreign birth rose one day en masse, called
a public meeting, and forced the king to sign a constitution of their own
preparation, a document which deprived [him] of all power and practically
took away the franchise from the Hawaiian race.
C RE SS T/U C LA
History Explanation
Scoring Rubric
1.
2.
3.
4.
5.
6.
7.
General impression of content quality
Principles or concepts (DD)
Prior knowledge (DD)
Use of available resources (DD)
Misconceptions (DD)
Argument (DD?)
English mechanics (DI)
C RE SS T/U C LA
Mathematics Explanation
Task (4th or 5th Grade)
Imagine a person from a television station has asked you to give a
demonstration on TV. You will be on a show to help other students learn
about maths. You are asked to explain everything 10-year-old students
should know about fractions.
Below are some questions you should try to answer. These are
questions that students in the TV audience will ask you.
For each question you should draw as many pictures as you can to
show what you mean. Then write down what you would say about your
pictures on TV. Use as many words and pictures as you need.
What is a fraction? Why are there two numbers in a fraction? How
many fractions are there between 0 and 1? How many fractions are equal
to 1/2? What other important ideas should students know about fractions?
Show how you would explain these ideas. Use as many pictures and words
as you need.
C RE SS T/U C LA
Scoring for Maths Task
Principles
Prior knowledge
Resources
Misconceptions
Argument/explanation
C RE SS T/U C LA
Technology for What?
Most available technology-based assessment
promotes efficiency rather than expands the
boundaries for measurement of learning
Tendency to limit attention to data processing
requirements
We should use technology to extend our
understanding of student accomplishment and
program and school quality
C RE SS T/U C LA
Template 2: Knowledge
Representation to Assess
Content Knowledge and
Problem Solving
Same tasks for content knowledge
Responses not essay but representation
of relationships, hierarchies
Scored by expert maps
Efficient but expands breadth and
depth of knowledge measurement
C RE SS T/U C LA
History: U.S. Depression (CK)
C RE SS T/U C LA
Genetics: High Performance (CK)
C RE SS T/U C LA
Bicycle Pump—High
Performance (PS)
C RE SS T/U C LA
Integrating Knowledge Map
with Web Search Strategy
C RE SS T/U C LA
MAIN Story: Integrating
Formative and External
Assessments
Authoring systems (computer-supported
guidance) to help teachers create and
share various sorts of assessments
intended to measure goals included, not
covered by, or that need deeper
attention than given in external
measures
C RE SS T/U C LA
Strategy to Link External
and Formative Assessments
Cognitive demands (domain independent)
and content knowledge and strategies
(domain dependent)
Authoring system allows teachers to
assess goals not externally measured, or
to connect their assessments to external
measures but in a more contextualized
setting
C RE SS T/U C LA
How It Works
Teacher will get guidance, using research-based
templates
What is the purpose of the test? What is to be
measured? What content? What conditions? What
intellectual skills?
Scoring rubrics (based on expert performance)
will be provided but can be edited
Graphical representation, simulations, and essays
are current options
C RE SS T/U C LA
Assessment Authoring
Benefits
Common floor on assessments created by teachers
Systems start with easy, fixed formats related to
learning demands, and as teacher sophistication
develops, move to more choices (mix and match)
Will allow summaries of student work bubbling up
from teachers’ formative assessments to validate
the external scores
Supports collaboration across different teachers
C RE SS T/U C LA
Authoring Systems Issues
Scored work made public (within the school,
with privacy provisions, and among schools)
Success depends upon teacher subject matter
knowledge, access to needed information,
and sharing
Success may depend on the realistic link to
external examinations
Generation of paper- or computer-based tasks
C RE SS T/U C LA
Minimum Requirements
Infrastructure
Capacity—subject matter
Orientation to learning and to results
Congruence with external mandates
Availability of smart tools
Lead to a culture shift
C RE SS T/U C LA
Distance Learning
Applications
High-quality performance demanded
No bottleneck in scoring
Basis of comparing courses
Generates online assessments with
instant scoring, feedback to student
and instructor
Aggregates student work
C RE SS T/U C LA
Window on the Development of
Problem-Solving Template 2—
Author Screen
Assessment Purpose(s)
 Diagnostic, readiness monitoring, certification
Scenario
 Context, constraints, situation
Problem Characteristics
 Fix, change usual sequence, improvise step(s), combination
Problem Identification Menu
 Stated, embedded, multiply masked, barriers, inconsistent
data from multiple sources, time bound, partial identification,
prior knowledge requirements
C RE SS T/U C LA
Author Screen Template 2
(Cont’d)
Macro Planning Menu
 Explicit courses of action, problem subdivision, backup
strategies, help seeking, mix of domain-independent and
domain-dependent cognitive strategies
Trial and Feedback Menu
 Data capture of process, process feedback, help,
iteration
Solution(s)
 Convergent (right answer), multiple acceptable, partially
acceptable, divergent—with scoring criteria, sequential,
mixed
C RE SS T/U C LA
Sample Examinee Screen
Components
Scenario
Problem
Information acquisition
Macro strategy
Micro strategies (domain specific)
Solution trials
Report of performance
C RE SS T/U C LA
Sample Examiner
Information
Time spent on problem
Trials to criterion
Help access
Solution paths
Generalizability of solution
Acceptability of solution
Likely explanation for errors (e.g., lack of
prior knowledge)
Metric (standards for performance)
C RE SS T/U C LA
Summary
Technology-based assessment needs to
extend what we can do
Authoring systems can help teachers design
better, more sensitive tests and projects
Technology can help us share findings
Technology-based assessment requires the
same evidence of technical quality
Demand evidence, not business claims, before
you buy