Supported by NSF grant DUE-1043443.
Conceptual Assessment:
What should your students learn?
And how do you know if they have
learned it?
Joel Michael, PhD
Dept of Molecular Biophysics and Physiology
Rush Medical College,Chicago, IL
jmichael40@gmail.com
Rosalind Franklin University Oct. 1, 2014
Members of the team
(in alphabetical order)
Bill Cliff, Niagara University
Jenny McFarland, Edmonds Community College
Joel Michael, Rush Medical College
Harold Modell, Bastyr University
Mary Pat Wenderoth, Univ. of Washington
Ann Wright, Canisius College
Rosalind Franklin University Oct. 1, 2014
Members of the team
Jeannie
Mary
Harold
Pat
Ann
Bill
Jenny
A long standing problem
Rosalind Franklin University Oct. 1, 2014
A long standing problem
I have been teaching physiology to 1st year
medical students for more years than I want to
think about.
Rosalind Franklin University Oct. 1, 2014
A long standing problem
I have been teaching physiology to 1st year
medical students for more years than I want to
think about.
It has always been the case that . . .
Rosalind Franklin University Oct. 1, 2014
A long standing problem
I have been teaching physiology to 1st year
medical students for more years than I want to
think about.
It has always been the case that . . .
WE KNOW MORE THAN WE CAN EXPECT
STUDENTS TO LEARN!
Rosalind Franklin University Oct. 1, 2014
To illustrate the problem
Rosalind Franklin University Oct. 1, 2014
To illustrate the problem
How many of you assign, require,
or recommend textbooks for use in
your course?
Rosalind Franklin University Oct. 1, 2014
To illustrate the problem
How many of you assign, require,
or recommend textbooks for use in
your course?
How big are these books? How
many pages?
Rosalind Franklin University Oct. 1, 2014
“SIZE” of some PHYSIOLOGY textbooks # of pages
Human Anatomy & Physiology
Saladin
Marieb and Hoehn
Martini
Undergraduate Physiology
Sherwood
Widmaier et al
Medical Physiology
Berne et al.
Boron and Boulpaep
Guyton and Hall
1,248
1,296
1,110
801
738
978
1,267
1,066
Michael et al., 2009
To illustrate the problem
How many of you expect the
students to learn EVERYTHING in
the textbooks you assign?
Rosalind Franklin University Oct. 1, 2014
The problem
• Too much is known and the amount known is
growing every day (the knowledge explosion)
Rosalind Franklin University Oct. 1, 2014
The problem
• Too much is known and the amount known is
growing every day (the knowledge explosion)
• We all recognize that students cannot, and
should not, be expected to know everything
Rosalind Franklin University Oct. 1, 2014
The problem
• Too much is known and the amount known is
growing every day (the knowledge explosion)
• We all recognize that students cannot, and
should not, be expected to know everything
• So, what should your students know? And
how should you go about determining that?
Rosalind Franklin University Oct. 1, 2014
The problem
• Too much is known and the amount known is
growing every day (the knowledge explosion)
• We all recognize that students cannot, and
should not, be expected to know everything
• So, what should your students know? And
how should you go about determining that?
• And how do you determine whether they do
know it?
Rosalind Franklin University Oct. 1, 2014
Today’s agenda
• Describe an approach to defining what
students should know
• Describe an approach to determining if they
do know it
• Provide you with a brief example of what has
been accomplished in physiology
Rosalind Franklin University Oct. 1, 2014
How do you determine what your
students should know?
Rosalind Franklin University Oct. 1, 2014
How do you determine what your
students should know?
• Ask yourself what’s important
Rosalind Franklin University Oct. 1, 2014
How do you determine what your
students should know?
• Ask yourself what’s important
• Ask your local colleagues
Rosalind Franklin University Oct. 1, 2014
How do you determine what your
students should know?
• Ask yourself what’s important
• Ask your local colleagues
• Ask a broader sampling of your colleagues
Rosalind Franklin University Oct. 1, 2014
How do you determine what your
students should know?
•
•
•
•
Ask yourself what’s important
Ask your local colleagues
Ask a broader sampling of your colleagues
Collectively reach consensus on what are the
important things
Rosalind Franklin University Oct. 1, 2014
“I want my students to learn . . .”
Rosalind Franklin University Oct. 1, 2014
“I want my students to learn . . .”
the “BIG IDEAS” that they will be able to use to
help understand physiology wherever they
encounter it, even if they don’t remember all
the details.
Rosalind Franklin University Oct. 1, 2014
What are “big ideas?”
Rosalind Franklin University Oct. 1, 2014
What are “big ideas?” (1)
By definition, big ideas are important and
enduring. Big ideas are transferable beyond the
scope of a particular unit . . . Big ideas are the
building material of understanding. They can be
thought of as the meaningful patterns that enable
one to connect the dots of otherwise fragmented
knowledge.
Wiggins and McTighe, Understanding by design,
expanded 2nd edition, Arlington, VA:
Association for Supervision and Curriculum Design., 2005, pp. 338-339
Rosalind Franklin University Oct. 1, 2014
What are “big ideas?” (2)
“Each [BIG IDEA] is well tested, validated, and
absolutely central to the discipline. Each
integrates many different findings and has
exceptionally broad explanatory scope. Each is
the source of coherence for many key concepts,
principles and even other theories in the
discipline.”
Duschl, R. A., Schweingruber, H. A. and Shouse, A. W. (Editors).
(2007). Taking science to school: Learning and teaching science
in grades K-8. Washington, DC: National Academies Press.
Rosalind Franklin University Oct. 1, 2014
Conceptual Assessment in Biology
(sponsored by NSF)
• Three meetings over a three year period
• First meeting brought together 24 biologist
from a WIDE variety of disciplines (A-Z almost)
• We immediately agreed that before talking
about how to do assessment we had to know
what we wanted assessed!
• So we decided to take as much time as
needed to generate a list of core concepts in
biology.
Rosalind Franklin University Oct. 1, 2014
Conceptual Assessment in Biology
(sponsored by NSF)
• To the surprise and amazement of all of us it
took us not more than 1 hour to reach
agreement on a list of 8 core concepts!
• We did acknowledge that different disciplines
emphasize different core concepts to a varying
extent.
– Physiology emphasizes a slightly different set of
concepts than does biochemistry or anatomy.
Rosalind Franklin University Oct. 1, 2014
“Big ideas” in biology (edited)
1. Living organisms are causal mechanisms . . .
2. The cell is the basic unit of life.
3. Information flow . . . .
4. Transformations of matter and energy
5. Homeostasis (and “stability”)
6. . . . structure and function . . .
7. Evolution . . . .
8. All life exists [in] an ecosystem . . . .
From 2nd CAB workshop (Michael and McFarland, 2008)
Rosalind Franklin University Oct. 1, 2014
“Big ideas” in biology (edited)
1. Living organisms are causal mechanisms . . .
2. The cell is the basic unit of life.
3. Information flow . . . .
4. Transformations of matter and energyBIOCHEM
5. Homeostasis (and “stability”) PHYSIOLOGY
6. . . . structure and function . . . ANATOMY
7. Evolution . . . .
8. All life exists [in] an ecosystem . . . .
From 2nd CAB workshop (Michael and McFarland, 2008)
Rosalind Franklin University Oct. 1, 2014
What are “core concepts?”
Rosalind Franklin University Oct. 1, 2014
What are “core concepts?”
Operationally,
Core Concepts = Big Ideas
Rosalind Franklin University Oct. 1, 2014
What are “core concepts?”
Operationally,
Core Concepts = Big Ideas
For me a core concept is what I want students
to remember in five years, even if they have
forgotten some or all of the details.
Rosalind Franklin University Oct. 1, 2014
How do you determine the core
concepts of your discipline?
Rosalind Franklin University Oct. 1, 2014
CAB
mtgs
Compiled a list
of 15 core
concepts
CAP
mtgs
Asked colleagues to rank
order items on our list
AND designate the top 3
Asked colleagues
via a survey
Selected set for the CAP
group to further explore
Rosalind Franklin University Oct. 1, 2014
The core concepts of physiology
identified by the CAP group
Causality
Cell-cell communications
Cell membrane
Cell theory
Energy
Evolution
Flow down gradients
Genes to proteins
Homeostasis
Interdependence
Levels of organization
Mass balance
Physics/chemistry
Scientific reasoning
Structure/function
Rosalind Franklin University Oct. 1, 2014
Michael and McFarland, 2011)
The core concepts of physiology
ranking by diverse faculty
Causality (14)
Cell-cell communications (3)
Cell membrane (1)
Cell theory (9)
Energy (6)
Evolution (15)
Flow down gradients (5)
Genes to proteins (11)
Homeostasis (1)
Interdependence (4)
Levels of organization (12)
Mass balance (13)
Physics/chemistry (10)
Scientific reasoning (8)
Structure/function (7)
Rosalind Franklin University Oct. 1, 2014
Michael and McFarland, 2011)
The core concepts of physiology
ranking by diverse faculty
Causality (14)
Cell-cell communications (3)
Cell membrane (1)
Cell theory (9)
Energy (6)
Evolution (15)
Flow down gradients (5)
Genes to proteins (11)
Homeostasis (1)
Interdependence (4)
Levels of organization (12)
Mass balance (13)
Physics/chemistry (10)
Scientific reasoning (8)
Structure/function (7)
Rosalind Franklin University Oct. 1, 2014
Michael and McFarland, 2011)
The core concepts of physiology
to be expanded by CAP
Causality (14)
Homeostasis (1)
Cell-cell communications (3)
Interdependence (4)
Cell membrane
Levels of organization (12)
(1)
Cell theory (9)
Energy (6)
Mass balance (13)
Physics/chemistry (10)
Evolution (15)
Scientific reasoning (8)
Flow down gradients (5)
Structure/function (7)
Genes to proteins (11)
Michael and McFarland, 2011)
Rosalind Franklin University Oct. 1, 2014
Unpacking core concepts
Homeostasis, cell-cell communications, and flow
down gradients are all, literally, BIG IDEAS,
comprised of many smaller ideas.
“Unpacking” a core concept is the process of
determining in a systematic way what those
subsidiary ideas are and how they are organized.
Rosalind Franklin University Oct. 1, 2014
Unpacking the core concept of
homeostasis
• It is too large for a PowerPoint slide, so I’ll ask
you to look at the handout I have provided as I
proceed.
Rosalind Franklin University Oct. 1, 2014
Top level unpacking of the concept of
homeostasis
H1. The organism maintains a more or less stable
internal environment.
H2. A substantial change to a regulated variable will
result in a physiological response to restore it
toward to its normal range.
H3. Homeostatic processes require a sensor (“what
can’t be measured can’t be regulated”).
H4. Homeostatic processes require a control center.
H5. Homeostatic processes require effectors.
Rosalind Franklin University Oct. 1, 2014
Unpacking of component concept of
homeostasis
H3.
Homeostatic processes require a sensor (“what
can’t be measured can’t be regulated”).
H3.1.
Sensors detect the regulated variable and respond by transducing
that stimulus into a different signal.
H3.2.
H3.3.
Sensors respond within a limited range of stimulus values.
H3.4.
An organ system may employ a variety of types of sensors (e.g.
chemoreceptors, baroreceptors, mechanoreceptors, etc) to
regulate variables associated with that organ system
Sensors are constantly active (not just active when the regulated
variable is not at the set point value or outside of a ‘normal’
range).
Rosalind Franklin University Oct. 1, 2014
We are working on unpacking core concepts of:
Cell-cell communications
Flow down gradients
Rosalind Franklin University Oct. 1, 2014
PLEASE NOTE . . .
The set of unpacked core concepts DOES NOT
constitute the content of a course. Nor is it a
physiology curriculum.
It is a framework around which one can
construct a course or a curriculum.
I’m sure this true in every discipline.
Rosalind Franklin University Oct. 1, 2014
Developing a conceptual assessment
instrument on homeostasis
Rosalind Franklin University Oct. 1, 2014
Developing a conceptual assessment
instrument on homeostasis (1)
• One cannot directly assess student
understanding of homeostasis
• Must assess student understanding of the
unpacked, smaller ideas that make up the core
concept
• We made a decision to build a concept
inventory targeting undergraduate students
taking introductory physiology or A&P courses
Rosalind Franklin University Oct. 1, 2014
Developing a conceptual assessment
instrument on homeostasis (2)
• We developed questions reflecting our own
students’ difficulties with the concept of
homeostasis
• We also solicited observed misconceptions
from colleagues at national meetings
• We took great care to keep the language
accessible to our assumed audience
• The result was a 20 MCQ inventory
Rosalind Franklin University Oct. 1, 2014
Validating the inventory
• We had a cohort of physiology teachers tells
us about the importance and degree of
difficulty of the questions and modified them
accordingly
• We have had 250 students at 7 institutions
take the inventory and have concluded a
preliminary analysis of their responses
Rosalind Franklin University Oct. 1, 2014
Application
General model
Question types and results
Rosalind Franklin University Oct. 1, 2014
Student responses to HCI
• Students did better on general model
questions (no physiological system identified)
than they did on application questions (which
reference a specific system).
• At one institution students in a 400 level
course did significant better than students in
the required 300 level course AND the 400
level students did not exhibit the difference
described above
McFarland et al., 2014
Rosalind Franklin University Oct. 1, 2014
Conclusions (1)
• What are the core concepts you want your
students to understand?
– You will have to unpack these core concepts into
their important pieces.
• This is NOT to suggest that there are not
important “facts” they need to know.
• Your list of core concepts is not a course
description or curriculum!
Rosalind Franklin University Oct. 1, 2014
Conclusions (2)
• If you want to know if your student
understand these “big ideas” you have to
write questions that test “big ideas” (their
component pieces) and not just facts.
– This is at least as difficult, if not more so, than
writing your usual exam questions!
Rosalind Franklin University Oct. 1, 2014
References
Michael, J. (2007). Conceptual assessment in the biological
sciences: A National Science Foundation-sponsored workshop.
Advances in Physiology Education, 31, 389-391.
Michael, J., McFarland, J., and Wright, A. (2008). The second
Conceptual Assessment in the Biological Sciences workshop.
Advances in Physiology Education, 32, 248-251.
Michael, J., Modell, H., McFarland, J., and Cliff, W. (2009).
The “core principles” of physiology: What should students
understand? Advances in Physiology Education, 33, 10-16.
Michael, J. and McFarland, J. (2011). The core principles (“big
ideas”) of physiology: results of faculty surveys. Advances in
Physiology Education, 35, 336-341.
Rosalind Franklin University Oct. 1, 2014
Michael, J., McFarland, J., Cliff, W., Modell, H., Wenderoth, M. P., and
Wright, A. (2013). Homeostasis in undergraduate physiology
textbooks. Experimental Biology 2013, Boston, MA.
Wright, A., McFarland, J., Cliff, W., Michael, J., Modell, H., and
Wenderoth, M. P. (2013). Preliminary results on the prevalence of
physiology students’ homeostatic misconceptions. Experimental
Biology 2013, Boston, MA.
McFarland, J., Michael, J., Wenderoth, M. P., Modell, H., Wright, A.,
and Cliff, W. (2014). Conceptual assessment of physiology (CAP):
Development of a concept inventory for homeostasis. Experimental
Biology 2014, an Diego, CA.
Wright, A., Michael, J., McFarland, J., Modell, H., Cliff, W., and
Wenderoth, M. P. (2014). What visual representations of homeostasis
do faculty use? Experimental Biology 2014, San Diego, CA.
Rosalind Franklin University Oct. 1, 2014
Wright, A., McFarland, J., Cliff, W., Michael, J., Modell, H.,
and Wenderoth, M. P. (2013). Preliminary results on the
prevalence of physiology students’ homeostatic
misconceptions. Experimental Biology 2013, Boston, MA.
McFarland, J., Michael, J., Wenderoth, M. P., Modell, H.,
Wright, A., and Cliff, W. (2014). Conceptual assessment
of physiology (CAP): Development of a concept inventory
for homeostasis. Experimental Biology 2014, an Diego,
CA.
Wright, A., Michael, J., McFarland, J., Modell, H., Cliff, W.,
and Wenderoth, M. P. (2014). What visual
representations of homeostasis do faculty use?
Experimental Biology 2014, San Diego, CA.
Rosalind Franklin University Oct. 1, 2014
• If I can be of any help to you or your
colleagues in thinking about or implementing
a core concepts approach, don’t hesitate to
contact me (jmichael40@gmail.com).
Rosalind Franklin University Oct. 1, 2014
• If I can be of any help to you or your
colleagues in thinking about or implementing
a core concepts approach, don’t hesitate to
contact me (jmichael40@gmail.com).
• Thank you for your attention!
Rosalind Franklin University Oct. 1, 2014