Charting and Measuring Changes in
students’ Knowledge of a Curriculum Topic
Ross J Todd
Center for International Scholarship in School Libraries
Rutgers, The State University of New Jersey
New Brunswick NJ USA
rtodd@scils.rutgers.edu
www.cissl.scils.rutgers.edu
Acknowledgements
“Impact of School Libraries on Student Learning”
Funded by the Institute for Museum and Library
Services (IMLS) USA 2003-2005
Dr Carol Kuhlthau, Co-Principal Investigator
Dr Jannica Heinström, Research Associate CISSL
Nora Bird, Doctoral Student, Project Manager
Project Goals
Conceptual: To understand how students build on
their existing knowledge of a topic and how their
knowledge of a topic changes;
The transformation and integration of found
information into existing knowledge, and the
creation of new knowledge.
Methodological: To test approaches to the
elicitation and representation of new knowledge in
order to document changes to knowledge.
Professional: To develop school-based tools for
measuring and charting learning: SLIM Toolkit
Information Use
Physical and mental acts employed by humans to incorporate found
information into their knowledge base or knowledge structure (Wilson,
1981, 2000).
Information use research is the least developed in the information
behavior research Vakkari (1997); Spink & Cole (2005).
Given that school students typically spend a lot of time in the library
researching curriculum assignments:

do they actually learn anything new when they seek and use
information sources?

And if they do, what does this learning look like, in terms of new
knowledge about a curriculum topic?

How does their existing knowledge of a topic change through the
task?

How can this change of knowledge of a curriculum topic be
measured?
Knowledge Representation: Common
Assumptions
Subjective, private knowledge is structured, and can be adequately elicited
and represented (Rumelhart & Norman, 1985, Markman, 2002);
Knowledge may be depicted as the overall structure of concepts linked
together relationally (Ausubel, 1963; Ausubel, Novak & Hanesian, 1978;
Brookes, 1975);
Past experience and prior knowledge form the basis for constructing new
knowledge (Bartlett, 1932; Dewey, 1938; Vygotsky, 1962; Kelly, 1963);
Previous knowledge forms a filter in the selection and rejection of new
information, and as a result of encountering new information, knowledge
structures may or may not change (Brookes, 1980; Chinn & Brewer, 1998);
Conceptual change tends to develop gradually by modifying, refining and
transforming previous knowledge structures (Kintsch & Vab Dijk, 1978;
Dole & Sinatra, 1998; Todd, 1999b).
Gaps in our Understanding
Mapping the actual knowledge output as students
progress through the stages of information
searching has been given limited attention.
Need to develop more accurate techniques for
representing and measuring conceptual structures
and how they change (Pennanen & Vakkari, 2003)
Knowledge Representation & Measurement
in School Setting
Use of verbal and written protocols such as essays,
projects, exams, and presentations.
Amount and nature of knowledge of a topic is
determined primarily in classroom settings by
subject experts (teachers) who match the nature of
ideas to some expected target or expert conception
(typically curriculum content requirements), and
within a prescribed word / page count
Little measurement of change of knowledge in
terms of initial existing knowledge
Measuring Knowledge through Relations
and Structure
Studies by Chi (1992), Chi & Koeske (1983), and
Gobbo & Chi (1986)
Measure changes in knowledge structures in terms of
the number of links, the nature of clusters of
statements and the characteristics of coherence and
structural centrality.
AMOUNT OF KNOWLEDGE
KNOWLEDGE
STRUCTURE OF
Measuring Students’ Change in Knowledge:
Research Questions
Substance of Knowledge: What changes in content of relational
statements are evident in students’ knowledge descriptions as they
undertake guided inquiry project?
Structure of Knowledge: What changes in structure of relational
statements are evident in students’ knowledge descriptions as they
undertake guided inquiry project?
Amount of Knowledge: What changes in amount of relational statements
are evident in students’ knowledge descriptions as they undertake guided
inquiry project?
Extent of Knowledge: What changes in perceptions of their knowledge are
evident in students’ knowledge descriptions as they undertake guided
inquiry project?
Title of Knowledge: How do the students’ titles given to their topics change
as they undertake guided inquiry project?
Schools Context & Sample
10 New Jersey public schools
Experienced and expert school librarians
Diverse public schools
10 school librarians working on curriculum projects
with 17 classroom teachers
574 students in Grades 6 – 12; range of disciplines
Inquiry Training Institute Feb 24, 2004: overview and
critique of units, use of data collection instruments,
procedures and ethical guidelines
Data Collection Instruments
Five data collection instruments were used to collect the
data from the students:
1.
2.
3.
4.
5.
Writing Task 1 (at initiation of inquiry unit)
Writing Task 2 (at midpoint of inquiry unit)
Writing Task 3 (at conclusion of inquiry unit)
Search Journal Log
My Research Style
The instruments consisted of a combination of qualitative
and quantitative questions.
Writing Tasks
Writing task 1 and 2 consisted of the following questions
1. Write the title that best describes your research project at this
time.
2. Take some time to think about your research topic. Now write
down what you know about this topic.
3. What interests you about this topic?
4. How much do you know about this topic? Check () one box
that best matches how much you know. Nothing, Not much,
Some, Quite a bit and A great deal
5. Write down what you think is EASY about researching your
topic.
6. Write down what you think is DIFFICULT about researching your
topic.
7. Write down how you are FEELING now about your project.
Check () only the boxes that apply to you. Confident,
Disappointed, Relieved, Frustrated, Confused, Optimistic,
Uncertain, Satisfied, Anxious or Other.
Additional Questions at Writing Task 3
1. What did you learn in doing this research
project? (This might be about your topic,
or new things you can do, or learn about
yourself)
2. How did the SCHOOL LIBRARIAN help
you?
3. How did the TEACHER help you?
Substance of knowledge
Classification of Statements: based on nature of relationships
between concepts
Graesser & Clark (1985) Structures and procedures of implicit
knowledge. Norwood, N.J.: Ablex.
Properties:
Manner:
Reason:
Outcome:
Causality:
leads
Set Membership:
Implication:
Value judgment:
statements describing characteristics
statements describing processes, styles,
actions
statements of explanations of how and why
statements providing end result
statements showing some event causally
to another
statements about class inclusion
statements showing predictive relations,
inference, implied meaning
statements presenting personal position or
viewpoint
Classification of Statements
Properties:
Manner:
Reason:
Outcome:
Causality:
Set Membership:
Implication:
Value judgment:
The color of Valentine’s day is red
People drive aggressively in USA
The wall was constructed to block invaders
(People eat too much) As a result, people
got very sick
Too much alcohol can lead to liver failure
Michelangelo created works such as statue
of David, Cistine Chapel and the famous
Pieta
He was suspected of poisoning him
That’s not right
FACTS: property, manner, set membership
EXPLANATION AND RESULTS: Reason, outcome, causality
SYNTHESIS: conclusions, positions, viewpoints
Total number of statements by type recorded at
each stage of the data collection process
Statement Type
Facts
WT1 N
WT2 N
WT3 N
Number of
statements
Number of
statements
Number of
statements
Properties
1194
1820
2160
Manner
589
1190
1514
Set membership
143
218
343
Number of
statements
Number of
statements
Number of
statements
Reason
152
163
193
Outcome
35
58
89
Causality/consequence
48
105
116
Synthesis
29
70
82
Explanation and result
Patterns
SUBSTANCE: Students represented their topic predominantly by
factual property and manner statements, and that they used increasingly
more factual statements to represent their knowledge throughout the
three stages.
NUMBER OF STATEMENTS: There were significant differences
between the number of property statements that the students used
between WT1 and WT2 (t (220) =6.29, p = .0000; between WT2 and WT
3 (t(222) =4.26, p <=.00)
Overall the number of reason and outcome statements were lower than
property and manner statements, and did not substantially increase as
students progressed through the stages.
Students appeared to be oriented to gathering facts and knowing a set
of facts throughout the search process.
Two predominant patterns: Additive and Integrative
ADDITIVE APPROACH
Knowledge development characterized by
progressive addition of property facts
As the students built knowledge, they continued to
add property and manner statements, and to a lesser
extent, set membership statements
Stockpile of facts, even though facts were sorted,
organized and grouped to some extent into thematic
units by conclusion.
Remained on a descriptive level throughout
Horizontal
WT1: He is very famous for his plays …….(100, 285001)
WT2: He married Anne Hathaway. They had 3 children.
……He wrote 37 plays and 152 sonets. …… (100,
285001)
WT3: He was born on April 23, 1564 in Warichshire,
Stratford-upon-Avon, Britain………Married at age
18. Had three children; Judith, Hammet and Susana.
He was the first boy in the family, had 3 sister and 1
brother, Joan, Margaret, Gilbert sibling.(100,
285001)
INTEGRATIVE APPROACH
Initial: superficial sets of properties
Moved beyond gathering facts:
- building explanations
- address discrepancies
- organizing facts in more coherent ways
Interpret found information to establish personal
conclusions and reflect on these.
Some students subsumed sets of facts into fewer but
more abstract statements at the end
Immune reactions
WT1: It probably has most to do with how the body reacts to certain
problems in the body. Like how a body reacts to a sneeze or a cough.
WT2: The immune system is what protects you, anybody, from various
outside bacterias, viruses, and germs. …….The immune has I-cells and
other types of cells that help fight the……When you cut yourself you
can see the immune system at work because you can see the cells that
are rebuilding the tissues that were cut………(long reply)
WT3: (very long reply) The immune system was a big topic. I found out that
there is actually two types of immune systems in the body. ………I
learned that if a bacteria enter your body, it could enter a cell, replicate
in a manner of minutes and if there are…..So you could have millions of
bacteria in the body after an hour…….the body works against such
organ degration, mineral deficiency, mechanical damage and other.
What I really enjoy to learn was that the minute you are born….and
when you die your immune system shuts down letting in all the bad
stuff, so now the body is an open door….
Structure of Knowledge
Ideas are discrete and unrelated.
Some limited structure evident –meaning
more than one instantiation- some ideas are
joined or linked (grouped) while others are
discrete or unrelated.
Contiguous ideas are associated; set of
ideas may be somewhat continuous.
Overall, ideas are interrelated and
continuous.
Ideas are integrated and unified; there is
structural centrality, and overall unification.
Changes in Structure of Knowledge
Generally there was a change towards more structured replies from WT1 to
WT3, but it was not consistent.
Change from disparate, unstructured and random listing of facts to
endpoint representations that showed some organization of facts into
thematic groupings
For students showing an additive model of knowledge structure, WT1
showed more structure than WT2. This is consistent with the pervasive
viewpoint that people’s existing knowledge is structured.
However, as students built up background knowledge, the gathering of
facts rather than the sequencing and organization of these facts seemed to
take over, as evident in WT2 representations being less structured and
interlinked.
By WT3, some organization of these facts into conceptually coherent
groups and linked together had taken place, particularly for students
showing integrative model
Labeling (Title) of Knowledge
Little investigation done on the analysis of titles
given to a representation of knowledge
Given that the library profession is built on
classification schemes that reflect knowledge of the
domain being classified, we assumed a relationship
between title and substance of knowledge
We assumed that as content changed, this would
be reflected in a change of title.
Three Kinds of Titles
1. General title: A title that describes the project on
a general, overall level. For example: Ancient
Rome; pollution; The West
2. Specific title: The title brings forward a specific
aspect of the project. For example: Ancient
Roman Government; Rulers of Ancient Rome; air
pollution; westward expansion
3. The title is expressed in a creative, or artistic
way: For example: The best and most famous
poet who ever lived-William Shakespeare; PCP,
not just a drug or PCP, deadly, yet not gone; Is
your hair really getting clean?
Patterns in Changes of Title
(60%) The title remains the same and typically general throughout.
There was no change in how the student stated the title, for example:
Ancient Greece  Ancient Greece  Ancient Greece
(20%) Hourglass phenomenon, where a title is general in WT1 and
WT3, but specific in WT2. For example: Food ancient food  food;
Health in Ancient Greece  medicine in ancient Greece  Health in
Ancient Greece;
Increasing specificity: For example Ancient Greece  Fashion in
Ancient Greece  Daily clothing of Ancient Greece; Julius Caesar 
the life of Julius Caesar  Caesar: his death
General to specific to artistic/creative: For example: Entertainment
ancient Roman entertainment what the Romans did for fun;
perfume and cologne  chemistry behind the scent  chemistry stinks,
the chemistry behind perfumes and other scents
Associative changes: (small numbers) Roman military  Roman
fashion  Food in Rome
General to general to artistic: (small numbers) The culture of the
English people  The culture of the English people  The mother of
America but still so different.
Change in Titles
The title the students give their projects may not accurately
reflect their level of knowledge
Use of general titles throughout seems to indicate that they
simply replied the question by stating their “official” project
title instead of realizing the possibility of expressing their
own personal conceptualization of it.
This seems to be an outcome of instructions for the
completion of the questionnaires, rather than the students’
own misunderstandings of the question.
Some students whose titles seemed to develop to a more
specific expression (or an hourglass expression) over time
had a synthesized reply in WT3.
Estimate of Knowledge
We asked the students to estimate how much they thought
they knew about their topic as they progressed through the
task. A five-point scale was used: nothing (0), not much (1),
some (2), quite a bit (3), and a great deal (4).
On average the students own estimated knowledge
developed from “not much” (mean = 1.26, SD =.91) at WT1,
to “some” or “quite a bit” (mean = 2.6, SD =.83) at WT2 At
WT3, most students found they knew at least “quite a bit”
(mean = 3.2, SD =.76) about their topics.
There was a significant difference between the students’
estimates of their own knowledge between the stages
Did the students’ knowledge change?
Substance of Knowledge
Structure of Knowledge
Amount of Knowledge
Extent of Knowledge
Title of Knowledge
Knowledge outcomes of instruction: depth
vs width
Instructional interventions
Zones of intervention