Disciplinary Research Strategies for
Assessment of Learning
Diane Ebert-May
Department of Plant Biology
Michigan State University
www.first2.org
Question 1
Please respond on a scale of 0 - 100 in increments of 10:
How important is it to use multiple kinds
of data to assess student learning?
How important is it to use multiple kinds of
data to assess student learning?
Question 2
Please respond on a scale of 0 - 100 in increments of 10:
How often do you use data to make
instructional decisions?
How often do you use data to make
instructional decisions?
True or False?
Assessing student learning in science
parallels what scientists do as
researchers.
Assessment in Teaching
Parallels Assessment in Research







We ask questions and develop hypotheses to solve problems and
make predictions about learning.
Our questions are based on current knowledge and theories, are
creative, original and relevant to the investigator.
Research designs and methods we use to collect data are logical
processes to answer questions.
Instruments/techniques we use are valid, repeatable measures of
learning.
Assessment (results) help us understand student thinking.
Results drive our next questions and decisions about a course.
Our ideas are peer reviewed - informally or formally.
What is assessment?
Data collection with the purpose of answering
questions about…
» student understanding
» students’ attitudes
» students’ skills
» instructional design and implementation
Use data to make predictions about student learning
Graduate Education

Often excellent at preparing
individuals to design and carry out
disciplinary research.
Graduate Education


Often inadequate and haphazard in
preparing future faculty/professionals to
take on the increasingly complex demands
of the professoriate.
Teaching is not mentored, peer reviewed,
or based on accumulated knowledge.
Solution: IRD model

Intergenerational research development teams
(IRDs) in cooperative academic environments
» Who: senior faculty, junior faculty, postdoctoral
and graduate students.
» What: scholarship of science teaching and learning
is fully integrated into the professional culture
along with discipline-based activities.

Assessment is critical to both practices.
Collaborators









Janet Batzli - Plant Biology (Director-Biocore, U of Wisconsin)
Doug Luckie - Physiology (Associate Professor)
Scott Harrison - Microbiology (Graduate student)
Tammy Long - Plant Biology (Assistant Professor)
Jim Smith - Zoology (Associate Professor)
Deb Linton - Plant Biology (Postdoctoral Fellow)
Heejun Lim - Chemistry Education (Postdoctoral Fellow)
Duncan Sibley - Geology (Professor)
National Science Foundation, Hewlett Foundation
Recognizing and Rewarding
Evaluating and Improving Undergraduate
Teaching in Science, Technology,
Engineering, and Mathematics (2003)
»National Research Council
»www.nap.edu/catalog/10024.html
What are central questions about
learning?
1. What do we want our students to know
and be able to do?
2. What knowledge or misconceptions do our
students bring to the course?
3. What evidence will we accept that
students know and can do?
4. How does our instruction help learning?
What Type of Learning?
Bloom (1956)
 6 major categories in the
Cognitive Domain of Educational
Objectives
Cognitive Levels



Knowledge - remember
Understanding and Application - grasp
meaning, use, interpret
Critical Analysis - original thinking,
open-ended answers, whole to parts,
parts to whole, evaluation
What type of data do we gather?



Depends on the evidence we will accept that
students have learned what we want them to learn.
Data must be aligned with the course goals.
Measures of knowledge, attitudes, and skills.
» tests, extended responses, concept maps,
» research papers, teamwork, communication

Use Bloom’s as a tool to categorize cognitive
domains
Research Question
How can diagnostic assessment
questions help us understand
students’ prior understanding
and progressive thinking about
the carbon cycle over time?
Prediction
Diagnostic, robust questions about the
carbon cycle integrated into the
biology course instructional design will
provide the same results about
student learning regardless of the
teacher.
Theoretical Background

Conceptual change theory
»Force Concept Inventory
(David Hestenes, Physics Dept., ASU)
Carbon Cycle = Rich Problem
Why?




Integrates many biological concepts at multiple
scales - ecosystems to molecules.
Instruction can return to elements intrinsic in
the carbon cycle - bioenergetics, metabolism.
Several documented student misconceptions
associated with the carbon cycle.
Real-world applied consequences if students
continue to misunderstand.
Some Common Misconceptions about
Photosynthesis & Respiration
Concept 1: Matter disappears during decomposition of organisms in the soil.
Concept 2: Photosynthesis as Energy: Photosynthesis provides energy for uptake of
nutrients through roots which builds biomass. No biomass built through
photosynthesis alone.
Concept 3: Thin Air: CO2 and O2 are gases therefore, do not have mass and
therefore, can not add or take away mass from an organism.
Concept 4: Plant Altruism: CO2 is converted to O2 in plant leaves so that all
organisms can ‘breathe’.
Concept 5: All Green: Plants have chloroplasts instead of mitochondria so they can
not respire.
Instructional Design

Active, inquiry-based learning
» Cooperative groups
» Questions, group processing, large lecture sections
(2 class meetings @80 minutes), 2 discussion
sections, multi-week laboratory investigation
» Homework problems including web-based modules

Different faculty for two courses
» One graduate/8-10 undergraduate TAs per course
Experimental Design
Two introductory courses for majors:
» Bio 1 - organismal/population biology (faculty A)
» Bio 2 - cell and molecular biology (faculty B)
Three cohorts:
» Cohort 1
Bio 1
» Cohort 2
Bio1/Bio2
» Cohort 3
Other/Bio2
Assessment Design



Multiple iterations/versions of the carbon
cycle problem
Pretest, midterm, final with additional
formative assessments during class
Administered during instruction
» Semester 1 - pretest, midterm, final exam
» Semester 2 - final exam
Multiple choice question (pre-post)
The majority of actual weight (dry biomass) gained by plants
as they progress from seed to adult plant comes from
which one of the following substances?
a. Particle substances in soil that are take up by plant roots.
(15%).
b. Molecules in the air that enter through holes in the plant
leaves (4%).
c. Substances dissolved in water taken up directly by plant
roots. (28%).
d. Energy from the sun (29%).
N=138
Radish Problem (formative)



Experimental Setup:
Weighed out 3 batches of radish seeds each weighing 1.5 g.
Experimental treatments:
» 1. Seeds placed on moistened paper towels in LIGHT
» 2. Seeds placed on moistened paper
towels in DARK
» 3. Seeds not moistened (left DRY) placed in light
Radish problem (2)


After 1 week, all plant material was dried in an
oven overnight (no water left) and plant biomass
was measured in grams.
Predict the biomass of the plant material in the
various treatments.
» Water, light
» Water, dark
» No water, light
Results: Weight of Radish Plants
1.46 g
1.63 g
1.20 g
Write an explanation about the results.
Whale Problem (midterm Bio 1)
Two fundamental concepts in ecology
are “energy flows” and “matter cycles”.
In an Antarctic ecosystem with the
food web given above, how could a
carbon atom in the blubber of the
Minke whale become part of a
crabeater seal? Note: crabeater seals
do not eat Minke whales. In your
response include a drawing with arrows
showing the movement of the C atom.
In addition to your drawing, provide a written description of the steps the
carbon atom must take through each component of the ecosystem
Describe which biological processes are involved in the carbon cycle.
Grandma Johnson Problem
(final, Bio 1)
Hypothetical scenario: Grandma Johnson had very
sentimental feelings toward Johnson Canyon, Utah, where
she and her late husband had honeymooned long ago. Her
feelings toward this spot were such that upon her death
she requested to be buried under a creosote bush
overlooking the canyon. Trace the path of a carbon atom
from Grandma Johnson’s remains to where it could become
part of a coyote. NOTE: the coyote will not dig up
Grandma Johnson and consume any of her remains.
Spider Monkey Problem (final, Bio 2)
Deep within a remote forest of Guatemala, the remains of a spider
monkey have been buried under an enormous mahogany tree.
Although rare, jaguars have been spotted in this forest by local
farmers. Use coherently written sentences and clearly labeled
drawings to explain how a carbon atom in glucose contained within
muscle cells of the spider monkey might become part of a cell within
the stomach lining of a jaguar. (Note:The jaguar does not dig up the
monkey and eat the remains!) Include in your answer descriptions of
the key features (not complete biochemical pathways!) of the
organismal and cellular processes that explain how the carbon atom of
the monkey’s corpse could become a part of the jaguar’s body.
Analysis of Responses
Used same scoring rubric for all three problems - calibrated
by adding additional criteria when necessary, rescoring:
Examined two major concepts:
Concept 1: Decomposers respire CO2
Concept 2: Plants uptake of CO2
Explanations categorized into two groups:
Organisms (trophic levels)
Processes (metabolic)
Trace Carbon from Whale to Seal
(Bio1 students, n=141)
100
Organism
Process
80
%
60
40
20
Concept 1
Concept 2
Decomposers respire CO2
Plants uptake CO2
Photosynthesis
Glucose
Through Root
Through Air
Primary produces
Release CO2
Respiration
Decomposers
0
Cellular Respiration by Decomposers
(Bio1/Bio2 students, n=63)
100
80
%
60
40
20
0
Q1 Whale
Q2 Grandma J
Q3 Spider Monkey
Concept 1: Decomposers respire CO2
2(2) = 20.16, p < 0.01
Pathway of Carbon into Primary Producer
(Bio1/Bio2 students, n=63)
100
Air
Root
80
60
%
40
20
0
Q1 Whale
Q2 Grandma J
Q3 Spider Monkey
Concept 2: Plants uptake CO2
2(2) = 4.778, p = .092
Trace Carbon from Spider Monkey to Jaguar
100
Respiration
NA
80
60
%
40
20
0
Bio1/Bio2 (n=63)
0ther + Bio2 (n=40)
Concept 1: Decomposers respire CO2
2(1) = 14.59, p < .01
Pathway of Carbon into Primary Producer
100
Air
Root
NA
80
60
%
40
20
0
Bio1/Bio2 (n=63)
0ther + Bio2 (n=40)
Concept 2: Plants uptake CO2
2(1) = 8.89, p < 0.05
So What?
Problem sets about major concepts:
» Diagnostic re: what students
understand/misconceptions
» Methods: parallel to process in disciplinary research
» Learn what prior knowledge students bring to
course, what students gained
» Make predictions re: student responses about
difficult concepts
» Unveil new misconceptions
» Influenced our teaching for understanding
So What? (2)
Curricular changes:
»
»
»
»
Bacteria/Archaea metabolism - often omitted
Primary production - models in lab
Source/Sink and carbon flux
‘Spiral’ major concepts - over/over/over
Use of technology: CTOOLS
(concept mapping java applet ctools.msu.edu)
Assessment Gradient
low
Potential for Assessment of Learning
high
Multiple Choice … … Concept Maps … … Essay … … Interview
high
Ease of Assessment
Theoretical Framework
•Ausubel 1968; meaningful learning
•Novak 1998; visual representations
•King and Kitchner 1994; reflective judgement
•National Research Council 1999; theoretical
frameworks for assessment
low
The real world without C-TOOLs
The ideal world with C-TOOLS
Which scenario would you bet money on?
The Grandma Johnson Problem
Hypothetical Scenario: Grandma Johnson had very
sentimental feelings toward Johnson Canyon, Utah where she
and her late husband had honeymooned long ago. Her feelings
toward this spot were such that upon her death she
requested to be buried under a creosote bush overlooking the
canyon. She loved the idea that she'd become part of the
wonderful wilderness and live on through the wildlife that
lived there. Think to yourself and begin to trace the path of a
carbon atom from Grandma Johnson's (GJ) remains to where
it could become part of a coyote (NOTE: the coyote WILL
NOT dig up Grandma and consume any of her remains). What
fundamental pathways and processes of biology will be
involved in the transit of GJ's carbon atoms to that of the
wild coyote in Utah?
Task: Create a concept map that illustrates your understanding of the relationship
between these 10 concepts in the context of the Grandma Johnson problem. You may add
up to 5 extra concepts if you need them to explain the problem more clearly.
photosynthesis
respiration
carbon cycle
decomposers
primary producers
consumers
carbon dioxide
glucose
energy
oxygen
1. Work on the problem individually first, save it in CTOOLS, and print a hard copy.
2. Work on the problem with a partner. Both of you can retrieve your concept maps,
discuss, revise and produce the best final map to which both of you have contributed.
3. Submit all three maps - yours, your partner's and the FINAL MAP you completed
together - Please put the final map on top, with both of your names. Staple them together
(5 pts off if not stapled).
C-TOOLS Research
Question:
Is there a correlation between students’ concept map and
their written explanation of a problem?
Methods:
1. Develop a diagnostic problem
2. Build into instructional design
3. Student complete as homework
4. Develop coding scheme and analyze
The Plant Adaptation Problem
In this course, we have
claimed that land plants were the
first truly terrestrial organisms.
However, most biologists contend
that the saturated soils on land
were undoubtedly teaming with
bacteria, archaea, and protists
long before land plants evolved.
In light of this, what does the
phrase "truly terrestrial” mean?
To answer this question follow
these instructions:
Task: Make a concept map using the following concepts:
adaptation
dispersal
fitness
flowers
fruit
leaves
photosynthesis
reproduction
roots
seeds
vascular tissue
2. PRINT one copy for yourself then SUBMIT a copy electronically.
3. Then, using your concept map, write a short response to answer the question by
explaining the problems plants had to overcome to live on land and explain the adaptations
that allowed plants to overcome those problems.
4. After you finish you short response, print it out and use a highlighter to highlight the
statements (propositions) you used directly from your concept map. You should have
elaborated (explained) further upon these statements in your written response.
5. Hand in hard copy of (a) concept map and (b) written, highlighted answer.
Key Concepts - Plant Adaptation Problem
Key Concepts in Extended Response
Links on Concept Map
1. Adaptation increases fitness.
adaptation / fitness
2. Fitness “is” reproduction.
fitness / reproduction
3. Vascular tissue is an adaptation for
getting water on land.
vascular tissue / roots
4. Leaves are and adaptation for
photosynthesis on land.
leaves / photosynthesis
5. Flowers are an adaptation for
reproduction on land.
flowers / reproduction
6. Seeds are an adaptation for dispersal on
land.
seeds / dispersal
Coding Criteria
Each key concept coded
Each key concept counted as a






correct [C]
incorrect [I]
non-informative [N]
absent [A]
for both concept map
and extended
response.

Match:same coding on map
and writing or
No match: different coding
on map and writing
Matches Between Concept Map & Extended Response
Plant Adaptation Problem
Upper 25% of Students
40.9%
Match
59.1%
No Match
n=44
Concept Present: Both C-Map & Extended Response
Plant Adaptation Problem
Upper 25% of Students
Concept Map
Extended Response
100.0
% of Students Including
90.0
80.0
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
adaptation
increases fitness
fitness is
reproduction
vascular tissue for
water
leaves for
photosynthesis
flowers for
reproduction
seeds for dispersal
Concept
n=44
Non-matches: Concept Included on C-map or
Extended Response Only
Plant Adaptation Problem
Concept Map Only
40.4%
59.6%
Extended Response
Only
So Who?


IRD teams in the disciplines -Intergenerational Research Development
teams
Use the process they know best to gather
meaningful data about student learning to
guide the direction of undergraduate
education.