Concept Inventories
101
William Cliff
Department of Biology
Niagara University
Biology Scholars Program
SoTL Institute
July, 2011
Questions to be Addressed
• What is a concept inventory?
• How have concept inventories been used
in assessment?
• What are some field-tested inventories
available in biology?
What is a concept inventory?
from Wordle.net
What is a concept inventory?
• Set of field-tested conceptual
diagnostic questions (CDQs)
• Validated
− Importance
− Authenticity
− Interpretation
• Reliable
What is a conceptual diagnostic question?
• Based on key concepts in a field (as
identified by instructors/experts)
• Formated as a selected response item
− Correct answer
− Distractors derived from research on
student misconceptions
• Written in plain, ordinary language
Sample Conceptual Diagnostic Question
As seen from your location, when is the Sun directly overhead at NOON
(so that no shadows are cast)?
A. Every day.
B. On the day of the summer solstice.
C. On the day of the winter solstice.
D. At both of the equinoxes (spring and fall).
E. Never from the latitude of your location.
Sample item from the Astronomy Diagnostic Test (ADT) version 1
(Zeilik et al., 1998). The correct response is “E”.
http://www.wcer.wisc.edu/archive/cl1/flag/cat/diagnostic/diagnostic1.htm
Sample Conceptual Diagnostic Question
As seen from your location, when is the Sun directly overhead at NOON
(so that no shadows are cast)?
A. Every day.
B. On the day of the summer solstice.
C. On the day of the winter solstice.
D. At both of the equinoxes (spring and fall).
E. Never from the latitude of your location.
Sample item from the Astronomy Diagnostic Test (ADT) version 1
(Zeilik et al., 1998). The correct response is “E”.
BRIEFLY EXPLAIN/DEFEND YOUR RESPONSE
http://www.wcer.wisc.edu/archive/cl1/flag/cat/diagnostic/diagnostic1.htm
Sample Conceptual Diagnostic Question
As seen from your location, when is the Sun directly overhead at NOON
(so that no shadows are cast)?
A. Every day.
B. On the day of the summer solstice.
C. On the day of the winter solstice.
D. At both of the equinoxes (spring and fall).
E. Never from the latitude of your location.
Sample item from the Astronomy Diagnostic Test (ADT) version 1
(Zeilik et al., 1998). The correct response is “E”.
http://www.wcer.wisc.edu/archive/cl1/flag/cat/diagnostic/diagnostic1.htm
Conceptual Diagnostic Questions
CDQ: Jared, the Subway® man, lost a lot of weight eating a low-calorie diet.
Where did all the fat/mass go?
A. The mass was released as CO2 and H2O.
B. The mass was converted to energy and used up.
C. The mass was converted to ATP molecules.
D. The mass was broken down to amino acids and eliminated from the body.
D’Avanzo, C. Bioscience 58:1079-1085, 2008.
CDQ: When a person loses weight, what happens to the fat in the person’s body?
Circle True (T) or False (F) for each option.
T F
Some the fat is broken down and leaves the person’s body as water and gas.
T F
Some the fat is converted into energy.
T F
Some of the fat is used up and disappears.
T F
Some of the fat is broken down and leaves the person’s body as feces and urine.
http://www.biodqc.org/node/515
Conceptual Diagnostic Questions
CDQ: The selection of antibiotic-resistant, transformed bacteria is based upon a change in the:
A. phenotype of the bacteria
B. genotype of the bacteria
C. phenotype and genotype of the bacteria
D. genotype and physiology of the bacteria
E. genotype and morphology of the bacteria
Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009.
CDQ: Consider a short polar region and a short non-polar region in a long polypeptide chain.
When dissolved in water, the polypeptide will most likely fold to form a protein in which:
A. the non-polar region is exposed on its surface and the polar region is interior.
B. the polar region is exposed on its surface and the non-polar region is interior
C. both the non-polar and the polar region are exposed on its surface.
D. both the non-polar and the polar region are interior.
Shi, J. et al. CBE-Life Sciences Education 9:453-461, 2010.
Host Pathogen Interaction Concept Inventory
Question 1
TABLE 3A
Question and Answers
Selection of an antibiotic resistant
organism is based upon a change in the:
A. phenotype
B. genotype
C. both
D. neither
e. either
General
Microbiology
(n=68)
1
38
25
0
4
Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009.
Bacterial
Genetics
(n=25)
7
4
13
1
0
Host Pathogen Interaction Concept Inventory
Question 1
“Defend Your Response”
Ans. “b”. genotype “When an organism becomes
resistant to antibiotics (when it acquires an antibioticresistant gene that has been inserted as a marker), the
organism’s genotype has been changed.”
Ans. “b”. genotype “This must be a change in the
genotype because having antibiotic resistance will not
necessarily change the look of an organism (phenotype).
It will merely allow it to survive in situations where the
antibiotic is present”
Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009.
Host Pathogen Interaction Concept Inventory
Question 1(revised)
1. The selection of antibiotic-resistant, transformed bacteria is based upon a change
in the:
A. phenotype of the bacteria
B. genotype of the bacteria
C. phenotype and genotype of the bacteria
D. genotype and physiology of the bacteria
E. genotype and morphology of the bacteria
TABLE 5
Percentages of correct answers for question 1 on pre- and post-course surveys
Pre or
Post
Pre
Post
General
Micro
2006
25
30
Pathogen
Micro
32
24
General
Micro
2007
12
26
Bacterial
Genetics
Immunol
Epidemiol
35
44
56
67
36
42
Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009.
How have concept inventories
been used in assessment?
• To reveal prior knowledge and misconceptions that
students bring to the class room
• To measure conceptual gains associated with
different methods of instruction
• To enhance professional development of faculty
and curricular reform
Suggestions for Use
• Adopt already-developed, field-tested instruments
− must follow the guidelines for its use for the results to be
valid and reliable.
− give the assessment as a pre- and post-test.
− give all items in the order presented on the instrument.
• Adopt already-developed test items (CDQs)
− cannot directly compare results to those normed from the
complete instrument.
− may better match course goals.
• Develop your own test items (CDQs)
− match questions closely to course goals.
− investigate the research literature before beginning.
− build up a bank of well-constructed/researched items.
Zeilik, M. Conceptual Diagnostic Tests. Field-Tested Learning Assessment Guide.
www.wcer.wisc.edu/archive/cl1/flag/cat/cat.html
Limitations
• Limited number of biology concept inventories
are currently available.
• Inventories may not match assessment goals.
• Vocabulary – e.g. use of jargon
• Format – close ended, multiple choice
Some field-tested inventories
available in biology
• Introductory Biology
(Klymkowsky and Garvin-Doxas, 2008; Klymkowsky
et al. 2010)
• Natural Selection
(Anderson et al. 2002)
• Genetics
− Non Majors (Bowling et al. 2008)
− Majors (Smith et al. 2008)
• Host-Pathogen Interactions
(Marbach-Ad et al. 2009)
• Molecular and Cellular Biology
(Shi et al. 2010)
My Work with Conceptual Diagnostic Testing
• Case Based Learning-mediated Misconception Repair in
Respiratory Physiology
• Chemistry Misconceptions Associated with Body
Calcium Homeostasis
• Teleological Thinking in Physiology
• Misconceptions in Renal Physiology
• Core Principles of Undergraduate Physiology
− Developing a Physiology Concept Inventory
Physiology of Oxygen Transport by the Blood
Conceptual Diagnostic Question A
Sa / PO2 Misconception
1st TIER
2nd TIER
Michael JA, et al.. Am. J. Physiol. 277 (Adv. Physiol. Educ. 22): S127-S135, 1999.
Conceptual Diagnostic Question B
Sa / PO2 Misconception
CBL Remediates Student Misconceptions
Prevalence of Respiratory Misconceptions
% Population with Misconception
100
PRETEST
POSTTEST
* P<0.05
80
60
*
40
20
0
Sa/PO2
O2/CO2
VT/f
Misconception
Cliff, W.H. Adv. Physiol. Educ. 30: 215-223, 2006
Met/Vent
CBL Improves Conceptual Understanding
Cliff, W.H. Adv. Physiol. Educ. 30: 215-223, 2006.
Misconception Repair By CBL
Is Progressive
Cliff, W.H. Adv. Physiol. Educ. 30: 215-223, 2006
Calcium Balance in the Body
A beaker contains calcium - Ca+2, phosphate - PO4 –3and calcium phosphate Ca3(PO4)2. These substances are at equilibrium in a fixed volume of water.
Ca+2 + PO4 –3 ⇋ Ca3(PO4)2 (solid)
Conceptual
Diagnostic
Test about the
Chemical
Equilibrium of
Calcium Salts
1. Suppose you add CaCl2 to the beaker. All the CaCl2 dissolves in the water
and forms Ca2+ and Cl―. After adding the CaCl2, you would predict that the
concentration of phosphate will:
A. increase
B. decrease
C. not change
Explain your answer:
2. Suppose you were to add more Ca3(PO4)2 to the beaker. The concentration of
calcium will:
A. increase
B. decrease
C. not change
Cliff W. Adv. Physiol. Educ.
33:323-328, 2009
Explain your answer:
Results
CDQ1: Suppose you add CaCl2 to the beaker.
All the CaCl2
dissolves in the water and forms Ca2+ and Cl―. After adding the
CaCl2, you would predict that the concentration of phosphate will:
A. increase
B. decrease (* CORRECT)
C. not change
TABLE 1. Multiple Choice Answers to CDQ1
Ans
All
A
%
6%
B*
#
%
4 52%
C
#
%
34 42%
Cliff W. Adv. Physiol. Educ. 33:323-328, 2009
#
28
total
####
66
Conclusions
CDQ 1
Answer C
Students failed to apply the common ion effect to the solubility equilibrium
established between calcium, phosphate and calcium phosphate. As a
result, they did not use Le Châtelier’s principle to correctly predict the
change in phosphate concentration that will occur as a result of the
addition of calcium to the solution.
Implications for Teaching:
Students may not be able to correctly apply the common ion effect to
determine changes in the equilibrium concentrations involved in
physiochemical reactions.
Relevant Misconceptions in Physiology:
Hypercalcemia would not alter plasma phosphate concentrations.
Addition of sodium bicarbonate to the body fluids would not change the
concentration of hydrogen ions.
Results
CDQ2: Suppose you were to add more Ca3(PO4)2 to the
beaker. The concentration of calcium will:
A. increase
B. decrease
C. not change (* CORRECT)
TABLE 2. Multiple Choice Answers to CDQ2
Ans
All
A
%
86%
B
#
%
57 3%
C*
#
%
2 11%
Cliff W. Adv. Physiol. Educ. 33:323-328, 2009
#
7
total
####
66
Conclusions
CDQ2
Answer A
Students failed to use the principle that the concentration of a pure solid
is constant for a heterogeneous equilibrium. As a result, some students
mistakenly applied Le Châtelier’s principle to the situation and predicted
that the addition of solid calcium phosphate would cause the reaction to
shift to the left and the concentration of calcium to increase. Other
students based their answer on notions of chemical dissociation or
addition without explicitly considering the chemical equilibrium.
Implications for Teaching:
Students may mistakenly apply Le Châtelier’s principle to physiological
circumstances where it is not appropriate - e.g. the contribution of bone
to calcium homeostasis.
Relevant Misconception in Physiology:
Increased bone mass should cause an increase in the concentration of
calcium ions in body fluids.
Teleological Thinking in Physiology
Table 1. Frequency Distribution of Student Responses on 10 Item Conceptual Diagnostic Test
Figure 1. % Teleological Answers
compared to Course Grade (n=72)
Cliff, W.H. FASEB J. 23: 632.12, 2009.
Results and Conclusions
1. Overall, 39 % of the answers chosen on the ten-item test were teleological.
Percentages ranged from 6% to 76% for individual items (Table 1).
Considerable variance exists in student preference for teleological
explanations for the cause of different physiological phenomena.
2. There was no correlation between the total number of teleological answers
chosen by individual students and final course grade (Figure 1).
The preference for teleological answers on the conceptual diagnostic test
did not appear to interfere with student ability to think or reason
mechanistically in other contexts.
3. Structured interviews indicated that some students internally translate the
teleological wording of test answers into mechanistic processes.
Internal restructuring indicates that students may accept teleological
formulations as answers without necessarily embracing teleological
reasoning as a means of explanation.
Instructors need not hesitate to use teleological formulation in the class room
for its heuristic value, as long as students are able to finish with satisfactory
mechanistic explanations for physiological phenomena.
Glomerulus
Qualitative
Conceptual
Diagnostic
Question
on
Glomerular
Filtration
Bowman’s Space
[Glu]
[Glu]
[Glu]
ml/min
RDQ4: The plasma concentration of glucose increases
above normal and the rate (ml/min) of glomerular filtration
remains constant. The concentration of glucose found in the
filtrate within Bowman's space will:
A. increase
B. decrease
C. remain constant
Glomerulus
[Glu]
Bowman’s Space
ml/min
Cliff, W., M.P. Wenderoth, and J.
Michael. FASEB J. 17:A816, 2003.
Please explain your answer:
[?]
Qualitative Conceptual Diagnostic
Question on Glomerular Filtration
RDQ4: The plasma concentration of glucose increases above
normal and the rate (ml/min) of glomerular filtration remains
constant. The concentration of glucose found in the filtrate
within Bowman's space will:
A. increase (* CORRECT)
B. decrease
C. remain constant
TABLE 1. Multiple Choice Answers to
RDQ4
U – undefined on survey
Ans
All
A*
%
65.6
Responses 68%
B
#
%
82 7.2
7%
#
9
C
%
24
#
30
U
%
3.2
#
4
total
####
125
25%
Cliff, W., M.P. Wenderoth, and J. Michael. FASEB J. 17:A816, 2003.
Summary



68% of respondents predicted filtrate glucose concentration would
increase following increased plasma concentration and no change in GFR.
25% predicted no change.
7% predicted decrease.
 Many of the students who predicted no change indicated that they
believed that the GFR determined the glucose concentration in the filtrate.
 Some students who answered the prediction correctly indicated that
they thought both plasma concentration and GFR influenced the filtrate
glucose concentration. For example:
“Greater filtration means greater [ ] in filtrate.”
“more glucose will be filtered so its concentration will increase”
“one increases above normal and the other remains constant, so we should see an increase”
“if there is a higher glucose concentration per mL, and the mL filtered is the same, there will
be more glucose coming in per mL anyway.”
These answers suggest that a substantial number of students have the
misconception that the concentration of glucose in the filtrate is
equivalent to the filtered load.
Quantitative Conceptual Diagnostic
Question on Glomerular Filtration
RDQ4: After ingesting a meal, the concentration of glucose
in the blood increases from 1 mg/ml to 2 mg/ml. The
glomerular filtration rate (ml/min) of the kidneys does not
change. Under these circumstances, the concentration of
glucose found in the filtrate within the Bowman’s space will
be:
A. 1 mg/ml
B. between 1 and 2 mg/ml (greater than 1 mg/ml but less
than 2 mg/ml)
C. 2 mg/ml (* CORRECT)
D. greater than 2 mg/ml
TABLE 2 Multiple Choice Answers to RDQ4
Ans
All
A
%
17.1
B
#
%
25 31.5
C*
#
%
46 41.8
D
total
#
%
# ####
61 9.59 14 146
Cliff, W., M.P. Wenderoth, and J. Michael. FASEB J. 17:A816, 2003.
Summary

42 % correctly predicted the filtrate concentration will equal the final plasma
concentration.
31% predicted an intermediate concentration between the initial and final
plasma concentrations.
17 % predicted no change in concentration.
10% predicted a higher concentration.



 Some of the students who predicted an intermediate glucose concentration
reasoned that a combination of increased plasma concentration with no
change in the GFR leads to an intermediate filtrate concentration.
Some students who predicted correctly indicated that both plasma
concentration and GFR influenced the filtrate glucose concentration.
“The GFR does not change and so blood concentration won't change as it flows out of the
capillary into Bowman's capsule.”
“The concentrations will be the same if the rates are the same”
As with the qualitative survey, these students express the misconception that
the filtrate concentration is equivalent to the filtered load.
.
Summary cont.

.
 A substantial fraction of students who answered incorrectly (41%) would
have given a correct answer if asked to predict the qualitative change in
glucose concentration. Only in response to a quantitative query did they gave
evidence they held a misconception.
CONCLUSIONS

After instruction in renal physiology, a sizeable number of students
demonstrate they hold the misconception that the concentration of
glucose in the filtrate is equivalent to its filtered load.

This observation reinforces the conclusion from investigations of
general models in physiology that student misconceptions about
“concentration” and “amount” can have a significant impact on their
understanding of renal physiology.
IMPLICATIONS FOR
TEACHING AND LEARNING

It is important to help students recognize the distinction between
“concentration” and “amount” in order for them to avoid developing
misconceptions about solute and water transport in the kidney and
other areas of physiology.

A combination of qualitative and quantitative diagnostic probes can
be useful to uncover different facets of student understanding of
physiological concepts.
Defining and Assessing the Core Principles
for Undergraduate Physiology
NSF TUES Grant (2011-2014)
PI: Jenny McFarland (Edmonds Community College)
CO-PIs: William Cliff (Niagara University), Joel Michael (Rush Medical
College), Harold Modell (Physiology Education Research Consortium), Ann
Wright (Canisius College)
RECRUITING FACULTY PARTICIPANTS to HELP
DEVELOP PHYSIOLOGY CONCEPT INVENTORY!
More Information
• D’Avanzo, C. Biology Concept Inventories: Overview, Status, and Next Steps.
Bioscience 58:1079-1085, 2008.
• Knight, J. Biology Concept Assessment Tools: Design and Use. Microbiol.
Australia March, 5-8, 2010.
• Libarkin, J. Concept Inventories in Higher Education Science. National
Research Council Promising Practices in Undergraduate STEM Education
Workshop 2, Washington, DC, 13-14 October 2008.
http://www7.nationalacademies.org/bose/Libarkin_CommissionedPaper.pdf
• Smith, J and Tanner, K. The Problem of Revealing How Students Think:
Concept Inventories and Beyond. CBE-Life Sciences Education 9:1-5, 2010.
• Zeilik, M. Conceptual Diagnostic Tests. Field-Tested Learning Assessment
Guide. www.wcer.wisc.edu/archive/cl1/flag/cat/cat.html