Design Talk as a Way to
Assess Student Learning of
Practices
Janet L. Kolodner
Jennifer Holbrook, Jackie Gray,
Paul J. Camp
College of Computing
Georgia Institute of Technology
Science Practices

Understanding a problem what might need to
be investigated

Investigation with a purpose -- experimentation,
modeling, learning from cases, ...

Informed decision making, reporting on and
justifying conclusions

Iteration towards understanding

Explaining scientifically

Teamwork, collaboration across teams, giving
credit
Design Practices








Identifying criteria, constraints, problem
specification
“Messing about” with materials
Investigation for the purpose of application
Informed decision making, reporting on and
justifying design decisions
Tradeoff and optimization
Iteration towards a good enough solution
Explaining failures and refining solutions
Teamwork, collaboration across teams, giving
credit
Learning by Design™




A project-based inquiry approach to science
education for middle school
Students learn science and technology
concepts and practices in the context of
attempting to achieve design challenges.
Highly collaborative
A variety of “community rituals” are
embedded in the approach to promote
learning of science and design practices.
Our Units

Physical Science
– Apollo 13 – introduction to practices of design and
science
– Vehicles in Motion – motion and forces
– Machines that Help – simple machines and
mechanical advantage

Earth Science
– Digging In -- introduction to practices
– Managing Erosion – erosion and accretion
– Tunneling through Georgia – geology, rocks and
minerals, rock formations, underground water
A Typical Design Cycle









Challenge is presented
Students mess about to generate questions for
inquiry
Investigation to address questions followed by
presentation of results to class
Design planning
Pin-up session
Construction & testing
Gallery walk
Additional investigation, demo, reading,
discussion of content
Iteration over last three steps to solution
Two LBD™ community rituals

Gallery walks (explanation and presentation
of results)

Pin-up sessions (justification)

Ritualized public ways of participating in
science and design practices
Well-articulated expectations
Repeatedly practiced and publicly discussed


Research Goals




To what extent are students learning
targeted content, and what is responsible
for differences?
To what extent are students learning
targeted practices, and what is
responsible for differences?
How do student conceptions and
capabilities develop over time?
What practices by teachers are most
conducive to promoting learning?
WhatХs needed to make
qualitative/observational data trustworthy?
 Focus informed by theoretical perspective
 Consistency across environments and a way of
managing reliability
 Careful planning
 Analysis informed by expectations Р looking for
confirming and disconfirming evidence
 Well-controlled quantitative results to back up
observations
 Collection of qualitative data that can be analyzed
quantitatively using developmental coding rubrics
 Several sources of data that can be triangulated
Assessment Methodologies

Content Test Pre/Post
– multiple choice & some essay
– what they know and development of their understanding

20 Questions
– self-assessment of capabilities

Videotaped Performance Assessments
– design an experiment, run an experiment, analyze data

Videotaped Structured Interviews
– interactive design, experiment design, & performance

Ethnography & Informal Data
Performance Assessments





Ask students to carry out practices in the
context of an authentic, but short
performance task.
Have students work in teams so as to be able
to record their deliberations.
Video deliberations.
Code written products and team interactions
for evidence of participation in practices.
Repeat periodically throughout experimental
period.
1Group Members__ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ___
_____________________________________________________
Teacher
__ ____ ____ ____ ____ ____ ____ _Block/Period__ ____ ____ _
Group Number __ ____ Da te___________
Where the Rubber Meets the Road
Group Probl em Solvin g Activity
PART I: Design and Describ e a Sim p le Expe riment in 1 5 Minu tes or Less
Rac e cars and l arge truc ks ha ve very different ty pe s o f t ires. Different ki nds of rubber require
different a mo unts o f force to overcome slidi ng fricti on. In this acti vity , your group will try to
figure out and de scribe a simple experiment th at tests the amount of force needed to ove rcome
sli ding fricti on f or different types of rubber and u nder different road conditi ons. You onl y h ave
15 mi nutes from rightn ow to fini sh. Everyone read the instructi ons t ogethe r and begi n.
INSTRUCTIONS
Examin
e th
e ubber
r
block th
atyou weregiv
en. Ithas h
ar
d r
ubberon one si
deand s
oft
rubberon t
h eoth
ersi
de.
(2) W ork to
geth
eras a
group to es
d ig
n and d
escribea si
m pl
e xpe
e rim ent ha
t t om
c par
es the
am ountsofforce
nee
ded to overc
ome slid
ing fr
ictio
n forbot
h kin
ds o
f rubber
.
(3) Y ourexper
iment sh
oul
d o
cm pare
slidi
ng fri
cti
on for th
e tw o types o
frubberunderdif
f ere
nt
condit
ions th
atmightbefound o
n a roadsurf
ace.
(4) O n thenextpage, wr
it
e your exper
iment es
d cr
iptio
n:
(a
)Selecta pe
rson n
i your group to
writ
e down th
e xpe
e ri
m entdescri
pti
on. Thi
s person
shoulduse bl
ack p
en, an
d s
houldwrit
e neatly
. owever,
H
all
m em bers ofhetgroup are
responsibl
e or
f w
hat iswrit
te
n.
(3) O n thenextpage, wr
it
e th
e xpe
e ri
m entdescri
ptio
n.
(4) D o n
otwrit
e on t
h ebac
k of th
e sheet;ra
is
e your hand and request a
n ex
tr
a sheet
, nd
a writ
e
the nam es o
f y
our group and “page2”on h
tis age
p .
(5) G ive enough detail
so t
hatsom eone rea
di
ng w hatyou writ
e will
know:
1. W hat y
ou are tryi
ng t
o find o
ut
(1)
1 H olb
rook/
LBD
Page1 4/7
/01
PART I: Design and Describe a Simple Experiment in 15
Minutes
Race cars and large trucks have very different types of tires.
Different kinds of rubber require different amounts of force to
overcome sliding friction. In this activity, your group will try to
figure out and describe a simple experiment that tests the
amount of force needed to overcome sliding friction for
different types of rubber and under different road conditions.
You have 15 minutes. Read the instructions together and
begin.
INSTRUCTIONS
Examine the rubber block. It has hard rubber on one side
and soft rubber on the other side.
Work together to design and describe a simple experiment
that compares the amount of force needed to overcome sliding
friction for each kind of rubber.
Your experiment should look at sliding friction for the two
types of rubber under several different conditions that might be
found on a road surface.
Describe your experiment on the next page.
Coding Categories
Negotiation during collaboration
 Distribution of the task
 Access to prior knowledge
 Adequacy of prior knowledge
 Science/design talk
 Science practice (experiment design)
 Self checks

Scoring the Quality of
Practices

Characterizing the
practice
– 1 - didn’t do it
– 2 – recognized
need; minimal
ability
– 3 – novice level
– 4 – intermediate
level
– 5 – expert level

Characterizing
group behavior
– 1 – nobody did it
– 2 – one person
recognized need
and tried
– 3 – half the group
tried
– 4 – done well half
the time
– 5 – group fluency
Scoring the quality of
practices

A 5 -point likert scale for each coding
category reflects a quantitative
continuum.
– 1 = Not at all: no evidence of the quality to be
rated
– 2 = Some evidence that at least one episode or
one student exhibits the quality rated
– 3 = The quality is exhibited by half the group
– 4 = The quality is exhibited for more than half
the episodes
– 5 = The quality completely captures the nature
of the episodes
Performance Assessme nt tasks: Coding for science practice
Jack ie Gray, Paul C amp, Jenni fer Holbrook, Ba rbara Fasse, and Jane t Kolodne r
Addit ionalno tes are fi ne and canbe recorded on the cod ingsh eet.
Ple ase note whi cheven t segmen t is being coded for eachep isod e:
planningan exp erim ent; p roblem set up; exp erim enta l man ipula tion; respon se
to written qu estion s.
Ingen eral, the 5 -poin t likert scale refle cts the follo wing quan tit ativ e
con tinuum. Details for each item are also inc luded be low.
1 = Not at all : no eviden ce o f the qual ityto b e rated
2 = So me ev idenc e tha t at least on e epi sode o r one student exhibi ts the
qua lity rated
3 = The qu ali tyi s exhib ited h alf the time
4 = The qu ali tyi s exhib ited for more th anhal f t he ep isod es
5 = The qu ali tycomp letelyc aptu res the natu re of the episode s
Design an expe riment segme nt:
Within an epi sode , t he context o f the gro up i s charact erized by :
Negotiations
Not a t all
1
at least one o f the
members of t he
group s ugge sts a
compromis e abou t
some aspect of t he
pro cedu re
at least one o f the
members of t he
group s ugge sts that
compromis e or
deb at e is needed
fo r at least half t he
is sues thatr equi re
it
at least two of the
members of t he
group que stions
several as pectof
the proc edur e and
the group makes
the needed chang e
Most de cis ion s are
made about
pro cedu re by the
entire team
con tribu ting and
decis ion m aking is
con sensual
2
3
4
5
Prior knowledg e is defined a s s tudents referring to so me a spec t of the c urriculum unit
that relate s to the current p robl em; referr ing to some a spect o f a pe rs ona l expe rience
that seems to relate to the current p robl em; referr ing to some a spect o f the science
concept o r method a t issue that appea rs to com e from previous expo sure to t he co ncept
or sk ill.
Students show evi dence of using prio r knowledge to solv e the problem
Not a t all
1
at least one o f the
members of t he
group me ntions a
prior event or
con cept that rel ates
to the prob le m
at least half of the
team mentions a
prior event or
con cept that rel ates
to the prob le m
Seve ral even ts and
con cepts are
mentioned and
app li ed t o the
prob le m
The group
rou tinely recalls
events or con cepts
that as sist in their
coll abo rative
prob le m s olv ing
2
3
4
5
More than one
member of the
group m entions or
foll ows up on
events or con cepts
that are useful
Ev ery me ntion of
prior know ledg e is
directlyapp li cable
to the prob le m
4
5
Prior knowledge seems adequate
Not a t all
1
at least one o f the
mentions of prior
know ledg e is
foll owed up on and
is useful
2
At le ast hal f the
mentions of prior
know ledg e are
app ropr iate to the
prob le m
3
Students use science practice to d ecide on method/proc edures
Not a t all
1
at least one o f the
members of t he
group s ugge st a
method to t est at
leas t one variable
2
at least one o f the
members sugge st a
method and
indicates an
unde rs tanding of
fai r t esting
at least one o f the
members sugge st a
method and
indicates an
unde rs tanding of
fai r t esting and
con trolli ng for
vari ables
Most of t he team
agrees that the
method u sed will
fai rly test the
im por tant variable s
and t heir decisions
wou ld actually be
a reasonab le
exp erim ent
3
4
5
The episodes are characterized by gro up self-checks on proce du res
Not a t all
1
at least one o f the
members of t he
group que stions
some aspect of t he
pro cedu re
at least one o f the
members of t he
group que stions
some aspect of t he
pro cedu re and the
makes the needed
change
at least one o f the
members of t he
group que stions
severala spect of
the proc edur e and
the group makes
the needed chang e
More than one of
the mem ber s of the
group que stions
severala spect o f
the proc edur e and
the group makes
the needed chang e
2
3
4
5
Table 1 :
Meansa nd standard d eviations for ca tego ries fr om pe rformance a ss essment cod ing for
LBD students (typical a nd hono rs) and Co mpar ison students (typical a nd hono rs)
Coding
category
Means (SD)
LBD Typ ical
Means (SD)
LBD Honors
2.50 (1.00)
3.25 (.50) *
Means (SD)
Comparison
Typica l
1.50 (.58)
2.25 (.50)
4.50 (.58) ***
4.00 (1.15)
Means (SD)
Honors
Comparison
2.67 (.58)
3.00 (1.00)
Negot iations
Distribu ted
Effort/tasks
Prio r
knowledg e
Prio r
Know ledge
adequat e
Science terms
used
Science
practice skil ls
2.25 (.50)
`1.75 (.50)
3.75 (1.50)
3.0 (.00)
2.75 (.96)
1.50 (.58)
3.50 (1.00)
2.67 (1.15)
2.50 (1.29)
1.75 (.50)
3.50 (1.00)
2.67 (1.15)
2.75 (.96)
2.25 (.50)
4.75 (.50) ***
2.67 (.71)
Self-c heck s
3.00 (.82) **
1.50 (.58)
4.25 (.50) ***
2.33 (.58)
Significance levels: * = p < .03; ** = p < .02; *** = p < .01
The me ans are ba sed on the likert scale : 1 - 5
Putting it all together




Code videos of performance tasks so as to
document performance capabilities of students
over time.
Use documentation of teacher practices to
identify high, medium, and low-fidelity LBD
classes and the practices of particular teachers.
Analyze across high-, medium-, and low-fidelity
LBD classrooms and between LBD and non-LBD
classrooms to see differences.
Use observational data to try to explain those
differences in terms of the ways the classroom
practices and culture and teacher facilitation
differed.
Making Observations Consistent:
Observational Prompt Tool




Sections focus on
Individual, small-group, whole-class
activities -- what to look for in each
LBD rituals – gallery walks, messing
about, wb
what to look for in student and teacher
behavior when certain goals are active –
generating questions for inquiry, focusing
investigation, debugging, construction, …
OPT Sample Prompts

Questioning
– What are teacher
questions about?
– What are student
questions about?
– What question types are
being used?
– Purpose of teacher
questions?
– How does teacher deal
with off-topic questions?
– (each question has a menu
of types and a set of
examples associated)

Gallery Walks
–
–
–
–
–
Who initiates the session?
Who displays the artifact?
Who asks questions?
Who gives feedback?
In what ways is feedback
constructive?
– What comparisons are
made between groups or
to previous work of the
presenting group?
– To what extent are
students being explicit
about design decisions,
LBD’s Foundations

Case-based reasoning’s model of learning from
experience (Kolodner, Schank, Hammond, …)

Problem-Based Learning’s model of the
classroom (Barrows, …)

Communities of Learners (Brown, Campione),
Constructionism (Papert, Harel, Kafai, …),
Cognitive Apprenticeship (Collins, Brown, …),
architecture studio, Decision-Based Design
(Mistree, …), transfer, development, ...

A project-based inquiry approach with more.