Some Implications of Expertise Research
for Educational Assessment
Robert J. Mislevy, PhD
Professor of Measurement & Statistics
University of Maryland
Keynote address at the 34th International Association for Educational
Assessment (IAEA) Conference, Cambridge University, September 8, 2008.
September 8, 2008
IAEA 2008
Slide 1
Introduction

These are exciting times in
assessment.

Developments in psychology and
technology.

Insights from expertise research with
implications for assessment design.

Let’s start with a quiz.
September 8, 2008
IAEA 2008
Slide 2
What is this a picture of?
(http://www.optillusions.com)
September 8, 2008
IAEA 2008
Slide 3
Which cards need to be turned over?
Each card has a letter on one side and a number
on the other. Consider the rule “If there is a vowel
on one side, there is an even number on the
other.” Which cards do you need to turn over to
make sure the rule is not violated? (Wason,1966)
E
September 8, 2008
4
T
IAEA 2008
7
Slide 4
A Little Story
I will read a little story and ask you
some questions about it.
September 8, 2008
IAEA 2008
Slide 5
Was this sentence in the story?
1.
Mary gave Ed a stuffed hippo.
2.
Harold received ninety gifts.
3.
Ralph gave Tina a toy car.
September 8, 2008
IAEA 2008
Slide 6
There were five sentences in the story.
In order, what were the second-to-last
words in each of the sentences?
September 8, 2008
IAEA 2008
Slide 7
What is this a picture of?
(http://www.optillusions.com)
September 8, 2008
IAEA 2008
Slide 8
Which cards need to be turned over?
Each card has a letter on one side and a number
on the other. Consider the rule “If there is a vowel
on one side, there is an even number on the
other.” Which cards do you need to turn over to
make sure the rule is not violated? (Wason,1966)
E
September 8, 2008
4
T
IAEA 2008
7
Slide 9
Whose IDs do we need to check?
Each person has an age and a beverage.
Consider the rule “If you are under 21, your
beverage must be non-alcoholic.” Which people
do you need to check to make sure the rule is not
violated?
15
Year
old
September 8, 2008
Water
drinker
IAEA 2008
30
Year
old
Wine
drinker
Slide 10
Was this sentence in the story?
1.
Mary gave Ed a stuffed hippo.
YES
2.
Harold received ninety gifts.
3.
Ralph gave Tina a toy car.
NO
NO
Ralph gave a toy car to Tina.
September 8, 2008
IAEA 2008
Slide 11
There were five sentences in the story.
In order, what were the second-to-last
words in each of the sentences?
One, stuffed, to, four, holiday.
September 8, 2008
IAEA 2008
Slide 12
Limitations and Difficulties
Processing limitations
 Limited attention
 Limited working memory
Knowledge limitations

Not knowing what information is relevant

Don’t know how to integrate information

Not knowing what to expect

Not knowing what to do and when to do it

Lack of production proficiency
September 8, 2008
IAEA 2008
Slide 13
Capabilities





Reasoning in terms of patterns
Many patterns simultaneously (language)
Can make perception, procedures, strategies
automatic with practice
Can think about our thinking (metacognition)
Benefit from procedures, methods, tools, external
knowledge representations
Expertise as the circumvention of human
processing limitations (Salthouse, 1991)
September 8, 2008
IAEA 2008
Slide 14
A Closer Look at Cognition

Knowledge as patterns, at many levels…

Assembled to understand, to interact with, and to
create particular situations in the world

Developed, strengthened, modified by use

Associations of all kinds, including applicability,
affordances, procedures, strategies, affect
“The user’s knowledge of the language rules is
interlocked with his knowledge of when, where, and
with whom to use them” (R. Ellis, 1985)
September 8, 2008
IAEA 2008
Slide 15
Walter Kintsch’s CI Theory of
Reading Comprehension
Text
Text base
Context
LTM
Situation Model
Context1
More focused
research areas
within cognitive
psychology
today differ as
to their foci,
methods, and
levels of
explanation.
They include
perception and
attention,
language and
communication,
development of
expertise,
situated and
sociocultural
psychology, and
neurological
bases of
cognition.
September 8, 2008
•E.g., reading tasks in Occupational English Test
(OET; McNamara, 1996) call upon patterns re
language, but also genre, medical knowledge, use
of information in clinical settings
A relevant pattern from LTM may be activated in
contexts but not others (e.g., physics models, use of
conditionals).
If a pattern hasn’t been learned, it won’t be activated
(althoughIAEA
it may
get constructed in the interaction).
2008
Slide 16
Walter Kintsch’s CI Theory of
Reading Comprehension
Text
Text base
Context
LTM
Situation Model
Action
Context1
More focused
research areas
within cognitive
psychology
today differ as
to their foci,
methods, and
levels of
explanation.
They include
perception and
attention,
language and
communication,
development of
expertise,
situated and
sociocultural
psychology, and
neurological
bases of
cognition.
September 8, 2008
Context2
IAEA 2008
Slide 17
Walter Kintsch’s CI Theory of
Reading Comprehension
Text
Text base
Context
LTM
Situation Model
Action
Context1
More focused
research areas
within cognitive
psychology
today differ as
to their foci,
methods, and
levels of
explanation.
They include
perception and
attention,
language and
communication,
development of
expertise,
situated and
sociocultural
psychology, and
neurological
bases of
cognition.
September 8, 2008
Context2
IAEA 2008
Slide 18
Walter Kintsch’s CI Theory of
Reading Comprehension
Text
Text base
More focused
research areas
within cognitive
psychology
today differ as
to their foci,
methods, and
levels of
explanation.
They include
perception and
attention,
language and
communication,
development of
expertise,
situated and
sociocultural
psychology, and
neurological
bases of
cognition.
September 8, 2008
Context
LTM
Situation Model
Action
Context2
Context3
IAEA 2008
Slide 19
Expertise Research
Cognitive task analysis (e.g., Simon & Chase)

Compare experts & novices in replicable conditions

What knowledge is needed? How is it represented?
How is it used? What makes tasks hard?
Ethnographic research (e.g., Lave)


Expertise in situ
“Critical incident” studies (NBME)
Replication possible in simulations
 Flight simulators, football kick in video situations
September 8, 2008
IAEA 2008
Slide 20
Expertise Research
Experts organize their knowledge effectively

Perceive / understand / act in terms of
fundamental principles rather than surface
features (Chi, Feltovich, & Glaser)
Importance of interaction with situation

Cycles of “perceive / understand / act”
External knowledge representations (KRs)
Nexus of info-processing & sociocultural POV
 Supported cognition / distributed cognition

September 8, 2008
IAEA 2008
Slide 21
So…
How do you use this improved
understanding of the nature and
acquisition of expertise to design
and conduct assessments?
September 8, 2008
IAEA 2008
Slide 22
Assessment Arguments
What complex of knowledge, skills, or other
attributes should be assessed?
What behaviors or performances should
reveal those constructs?
What tasks or situations should elicit those
behaviors?
(Messick, 1994)
September 8, 2008
IAEA 2008
Slide 23
Examples

The Architectural Registration
Examination (ARE)
Architectural design; CAD-like environment

DISC simulator
Simulations for problem-solving in dental
hygiene

NetPASS (Cisco)
Computer network design & troubleshooting
September 8, 2008
IAEA 2008
Slide 24
ARE Example (Irv Katz, ETS)

To replace 10-hour hand-drawn design
problem

Reflects changing of profession to CAD

Premium on thinking, not drawing

Planning the firestation site
September 8, 2008
IAEA 2008
Slide 25
An Example of a Task Prompt for the ARE
September 8, 2008
IAEA 2008
Slide 26
AB
D
An Illustrative Base Diagram
for an ARE Task
IAEA 2008
September 8, 2008
Slide 27
September 8, 2008
A Sample Solution
to an ARE Task
IAEA 2008
Slide 28
NCARB Example (Irv Katz, ETS)
Differences between novices and experts
• Success rate: 98% vs. 88%
• Planning time & execution time
• Patterns of revision involving rework
Implications for task design: Constraints
• Number
• Variation in importance/difficulty
• Degree of conflict
• Implicit constraints? (c.f. writing expertise)
September 8, 2008
IAEA 2008
Slide 29
Design Patterns

“Design under constraints” is common to many
domains: e.g., engineering, assessment design,
wedding planning, apparel design

Could define a Design Pattern with these as foci

What are characteristics of performances (i.e.,
observables) that evidence these knowledge /
capabilities / attunements?

What are characteristics of situations that elicit these
observables?
September 8, 2008
IAEA 2008
Slide 30
Other design pattern possibilities

Troubleshooting finite systems

Medical diagnosis

Inquiry cycle

Model-based reasoning
September 8, 2008
IAEA 2008
Slide 31
DISC Example

The Dental Interactive Simulations
Corporation (DISC)

The DISC Simulator & Scoring Engine

Cognitive Task Analysis to support
design rationale
September 8, 2008
IAEA 2008
Slide 32
Goals of the DISC CTA

What are the kinds of knowledge and skills that
hygienists call upon to solve problems and make
decisions in dental hygiene?

What do they say and do that gives you evidence
about their use of this knowledge?

What kinds of situations call upon this knowledge?

How do you make re-usable schemas to evoke
evidence and construct ‘stories’ around these
recurring patterns?
September 8, 2008
IAEA 2008
Slide 33
Using Disparate Sources of Information
Expert
Tries to use all sources of information all the time.
Constructs a model of the patient, with each source
of information an imperfect and incomplete window
on some aspect of the total situation. Exhibits
movement back and forth between sources
(resources and personal knowledge/experience),
trying to fit the partial clues together into a unified
whole.
Novice
Uses single information sources in isolation.
Compartmentalized use of information and failure to
integrate information across sources.
September 8, 2008
IAEA 2008
Slide 34
Formulating Problems & Hypotheses
Expert
Forms problems and generates hypotheses
using efficient, focused, and targeted action.
Uses forward and deductive reasoning in
formulating problems. Thinks strategically and
functions within the problem space. Efficiently
generates and prunes search trees.
Novice
Uses forward reasoning but scope and depth of
knowledge to support forward reasoning is
limited. Generates a search tree but possesses
limited tools for pruning the search tree.
September 8, 2008
IAEA 2008
Slide 35
Vocabulary and Language Usage
Expert
Retrieves and uses appropriate, clear,
sophisticated, accurate, and precise
terminology. Uses spontaneous declaration and
can retrieve without effort. Creates messages
that are easily understood by the target
audience.
Novice
Strained, unreliable, and effortful retrieval of
terminology. Hesitant, delayed, and labored
responses due to time required to process and
lack of knowledge. Uses terminology
unsystematically. Prone to canned speeches
and explanations without awareness of
individual patients’ needs.
September 8, 2008
IAEA 2008
Slide 36
An example with the Cisco Learning
Institute: The NetPass Prototype

Create on-line performance assessment of
networking skills

Focus on learner feedback rather than highstakes testing

Expert/Novice studies ground design
September 8, 2008
IAEA 2008
Slide 37
The task starts with a scenario
and description of goals
September 8, 2008
IAEA 2008
Slide 38
To capture their mental model of the network, we ask
them to draw the network with a diagramming tool
September 8, 2008
IAEA 2008
Slide 39
The diagram is created by dragging
and dropping icons
September 8, 2008
IAEA 2008
Slide 40
Configuring the devices
September 8, 2008
IAEA 2008
Slide 41
When students are done, they press Submit”…
September 8, 2008
IAEA 2008
Slide 42
And the graphical representation is converted into
an text representation in XML format
September 8, 2008
IAEA 2008
Slide 43
The text file is scored following detailed rules, which
result in characterizations of the work
September 8, 2008
IAEA 2008
Slide 44
And feedback is created for the student…
both diagnostic and a summary profile.
September 8, 2008
IAEA 2008
Slide 45
Conclusion

Insights from expertise research can improve
the practice of assessment, and support
deeper learning.

Doing so requires a deeper understanding of
assessment design.
» More explicit arguments and representations
» Generativity, re-usability, and inter-operability

“Too many notes” ? (Emperor Joseph II)

Suitable conceptual frameworks, tools, and
exemplars are now IAEA
beginning
to appear. Slide 46
September 8, 2008
2008