ASSESSMENT OF COMPREHENSION MODEL
Using Literal, Inferential and Higher Order Thinking Skills
Prepared by
Judith Taylor-Dunn
Prepared for
Dr. A. D’Onofrio
Ed 726 Developmental Psycholinguistics
December 6, 2004
Assessment, Fieldwork and Analytic Report
Background
The process of comprehending a sentence is a complex task that takes into
account a number of variables. Those variables include whether the sentence being
examined was received visually through reading or was it received orally through
listening. Was the sentence presented in isolation or did some type of semantic priming
occur that aided the comprehender’s search for meaning. Another important variable is
the ability of the reader or listener to access lexical information from memory. Lexical
information includes the following:
 The use of high frequency open-class words in the sentence. When the reader
or listener is familiar with all the words used in the sentence, comprehension
is enhanced.
 Phonological considerations especially in oral communications as seen by
listeners confusing the use of words that sound similar, but have different
meanings.
 Syntactic use of lexicons impacts the comprehender’s ability to draw meaning
from a sentence. When the speaker or writer doesn’t use the grammar of the
intended audience, comprehension is more difficult.
 Lexical ambiguity makes comprehension more difficult especially if the
reader or listener is not familiar with the intended meaning of a particular
word being used in a sentence.
Given that information, the following three-step comprehension model was created. The
purpose of this study is to test the model with two subjects to see if, in fact, it is accurate.
Step 1 – The sentence is perceived, by entering working memory, either through
hearing or reading.
Step 2 – Information stored in our semantic memory is activated and parsing
occurs.
Step 3 – If all the lexical information makes sense, comprehension is quickly
achieved. If it doesn’t, episodic memory is activated and the reader or listener
draws on past experiences to determine how to make the sentence make sense.
Research Question
Does the three-step comprehension model apply to questions that require literal,
inferential, or higher order thinking skills during an oral comprehension assessment?
1
Methods
A selection from Sophie’s World by Jostein Gaarder was selected by the researcher. The
book was chosen after confirmation from the participants that neither had read the book
nor have they taken a philosophy class. An informal oral comprehension assessment was
created that consisted of five questions. Each question required a different
comprehension skill in order to answer the question correctly. The comprehension skills
are: literal, inferential, understanding words in context, analysis and evaluation,
application. The particular right/wrong answer was not an important aspect of this
investigation. The strategies that each participant used to arrive at the answers are
actually being examined.
A verbal analysis was conducted using two subjects who verbally thought out loud when
attempting to answer each type of comprehension question. Their verbal responses were
taped using a tape recorder. The information on the tape recorder was transcribed and
analyzed.
Materials
Reading selection, informal oral comprehension assessment, and a tape recorder.
Subjects
Subject A:
Adult, African-American male age 44
Educational level: BS in Accounting
Employment: Government Business Development and Sales (f/t)
Subject B:
Adult, African-American male age 18
Educational level: Current College Freshman (f/t)
Employment: Grocery Store Clerk (p/t)
Procedures
Subjects were met individually and privately. They were asked to read the selection
silently. The verbal report procedure was then explained to each subject. They were
asked to think out loud. They were told that the first question, which was a literal
question, is used for practicing thinking out loud. They could ask questions of the
process during the think-aloud of the first question. Once the second question had been
asked, they were instructed that they could no longer ask additional questions.
2
Data
Discourse from Think Aloud
Respondent 1
Interviewer: “Who was Plato’s teacher?”
Respondent 1: “Socrates”
Interviewer: “How did you know that?”
Respondent: “I remembered what I read? I need you to describe what you are thinking as
you think it.”
Interviewer: “Who is the narrator in the story and who is Sophie?”
Respondent: “The narrator is Plato. Sophie, I guess, is somebody he is talking to. Down
at the end it starts talking about Sophists. Okay…it was unclear to me who
Sophie was. I don’t know why it is called Sophie’s World…but she seems
to be a friend of Socrates based on the conversation. It appears that he is
talking to her. He’s talking about Plato then he’s talking about Socrates.
Why would you talk to someone that is supposed to be a theory? It just
seems very odd to me.”
Interviewer: “Define hemlock and immutable.”
Respondent: “Hemlock is a poison. I just know. Okay. Immutable…hmm … I’m not
sure. The word. I’ve heard it before. I ‘m assuming…If I see it in context…
(He re-reads the sentence.) Immutable means something that is
uncontestable from the way it is written.”
Interviewer: “The text states this about Plato, ‘…a mind that was to have a profound
influence on all subsequent European Philosophy.’ Why do you think the
narrator makes this statement? Give examples to support your answers.”
Respondent: “Because Plato wrote down, from what I read… Plato wrote most of ah,
Socrates information and he he published and started what is considered
academics of the academies that are spread across basically all academic
knowledge, okay, or the the gathering of people to accumulate academic
knowledge or learning things. That’s why his process or his whole… um…
what he did was he he made it a process that is more than people just sitting
around and talking. He made it that we all meet and discuss specifics and
learn about things.”
3
Interviewer: “The narrator states that philosophers are often unpopular. Think of a
modern day philosopher and describe how society views this person.”
Respondent: “A modern day philosopher… ah…could be…” (rubbing his head)
Interviewer: “What are you thinking about to get the answer?”
Respondent: “I’m thinking of people that may have different opinions that may not be
popular. I think of who that person is…I’m thinking of someone
like…ah…maybe...Cornell West. Based on what he believes as being a
truth. It may not be popular, but it is based on what he has seen. Based on
the academics that he has accumulated so his philosophies…though they
may not be popular.”
Interviewer: “How do you know that?”
Respondent: “I know because of my understanding of what is popular by …what I know
from reading the paper…okay…listening to the news, reading different
articles in magazines that tell the popular opinion. Usually what is popular
is what is presented in a positive light or in a more prevalent light. What is
unpopular is, especially in a society where you have a preponderance of
media and and that’s very influential the unpopular types of topics are those
and and the people who have unpopular topics are not well publicized.
They are kind of kept on the outskirts of the limelight. So, that’s why I
would think that Cornell West would be a a a unpopular type philosopher
because he has philosophy that may not be mainstream, may not be the the
the most ah ah ah most thought of positions. But, still and all, he has some
strong ah… beliefs.”
Interviewer: “How do you know? What are you thinking?”
Respondent: “I think about what people have said...like… Tavis Smiley and Jessie
Jackson. I have read in Harvard Review some of the writings of Cornell
West.”
4
Respondent 2
Interviewer: “Who was Plato’s teacher?”
Respondent: “Socrates”
Interviewer: “How do you know that? What were you thinking?”
Respondent: “It just came into my head because I read it (pointing to the text).”
Interviewer: “Who is the narrator in the story and who is Sophie?”
Respondent: “Umm…the author…Jostein Gaarder is the narrator. I’m looking for a
sentence that says who Sophie is. (He continues to look through the text.) I
can’t find it.”
Interviewer: “Define hemlock and immutable”.
Respondent: “I’m looking for the answer because I remember reading it. I can’t find it.
I’m looking for the paragraph with “hemlock.” I want to use context clues.
I can’t find it… I’m still looking. I don’t know what they mean. Usually, if
you have a word in a sentence you can tell what it means by the surrounding
words. I learned it in school.”
Interviewer: “The text states this about Plato, “…a mind that was to have a profound
influence on all subsequent European Philosophy.” Why do you think the
narrator makes this statement? Give examples to support your answers.”
Respondent: “Cause he was a philosopher that didn’t write anything down. He must have
an extraordinary mind to remember everything. It says in one of the
paragraphs that he didn’t write anything down.”
Interviewer: “The narrator states that philosophers are often unpopular. Think of a
modern day philosopher and describe how society views this person.”
Respondent: “I’m trying to think about who is a philosopher. I’m thinking about famous
people…people that wrote books. I’m just trying to think who would be
considered a philosopher. Anybody can think of anything and it would be a
philosophy. Like who...like maybe the civil rights leaders. They could be
philosophers. Maybe like Martin Luther King or somebody like that.”
5
Analysis
Step 1 occurred in all instances.
Question 1
Literal
Question 2
Inferential
Question 3
Understanding
Words in Context
Respondent 1
Respondent 2
“I remembered what I read.”
“It just came to my head…”
Only relied on information in text.
Only relied on information in text.
“Okay…it was unclear to me who
Sophie was.”
It appears that the text didn’t make sense
semantically and the respondent had
difficulty parsing.
Continues to rely on information in the
text and doesn’t apply any strategies
that are needed for making an inference.
In defining “hemlock” his semantic
memory was activated and
comprehension occurred instantly.
Went through all the steps in the model
related to accessing a reading
strategy that can be used for
understanding words in context.
In defining “immutable” he relied
on activating a reading strategy to
determine meaning.
Question 4
Analysis and
Evaluation
Only relied on information in the text
and did not activate long-term memory.
Only relied on information in the text
and did not activate long-term memory.
Question 5
Application
Activated schema on people with
unpopular opinions.
Activated schema of famous people.
Assessed information obtained from
personal experience reading and
hearing experts on the subject.
Literal: I think step 2 was skipped and step 3 occurred after the information was
perceived. From the statements made by the respondents, it is unclear if any thinking
occurred that required long term memory.
Inferential: Respondent 1 became stuck in step 3 of the model therefore, comprehension
did not occur. He was unable to make an “internal representation of the linguistic
relationship within the sentence.” (Carroll, p127)
“Down at the end it starts talking about Sophists. Okay…it was unclear to
me who Sophie was. I don’t know why it is called Sophie’s World…but she
seems to be a friend of Socrates based on the conversation. It appears that
he is talking to her. He’s talking about Plato then he’s talking about
Socrates. Why would you talk to someone that is supposed to be a theory?
It just seems very odd to me.”
Respondent 2 doesn’t seem to attempt to think beyond the information in the text. I don’t
think he goes beyond step 1 of the model.
6
Understanding Words in Context:
Respondent 1 tried to activate his long-term memory of the word “immutable” but was
unsuccessful. He then used his knowledge of semantics to analyze the sentence and
determine meaning.
“I’ve heard it before. I ‘m assuming…If I see it in context… (He re-reads
the sentence.) Immutable means something that is uncontestable from the
way it is written.”
Respondent 2 activated his memory of a teacher explaining how to determine the
definition of a word in context. He did not appear to have semantic memory related to
the meaning of either word, so he activated his episodic memory and applied what he was
taught to do when he was presented with a word he did not understand.
“Usually, if you have a word in a sentence you can tell what it means by
the surrounding words. I learned it in school.”
Analysis and Evaluation:
For this question, I have no evidence that either respondent attempted to activate longterm memory.
Application:
Based on the comment below, it appears that respondent 1 was activating his schema on
the subject.
“I’m thinking of people that may have different opinions that may not be
popular. I think of who that person is…I’m thinking of someone
like…ah…maybe...Cornell West.”
I would argue that this information rested in the respondent’s episodic memory and was
used to create the schema activated to answer the question.
“I think about what people have said...like… Tavis Smiley and Jessie
Jackson. I have read in Harvard Review some of the writings of Cornell
West.”
The second responded also activated his schema to answer the question.
“I’m thinking about famous people…people that wrote books.”
7
Conclusions
Based on this brief qualitative study, I would adjust the model of comprehension I
created earlier in the semester.
Step 1 – The sentence is perceived, by entering working memory, either through
hearing or reading.
Step 2 – If the answer is present in the information perceived through working
memory, comprehension occurs. If not, the person chooses to activate semantic
memory.
Step 3 - Information stored in our semantic memory or schema is activated and
parsing occurs. If all the lexical information makes sense, then comprehension is
quickly achieved. If not, the person chooses to activate episodic memory.
Step 4 – Episodic memory is activated and the reader or listener draws on past
experiences to determine the answer.
As was shown in this study, the older person (44 yrs.) has more life experience on which
to draw from when attempting to answer a question that required high order thinking
skills. The younger person relied heavily on the text and the information learned in
school.
This study brings up many more questions about reading comprehension, such as:
o Does a person naturally move through these steps or does it have to be taught?
o What results would I get if I repeated the study using a large sample?
o Would the study results have differed if I initially activated background
knowledge before they read the selection?
o What impact does age, experience and educational background have on the
results?
Finally, it is important to recognize the limitations of this approach. I cannot be 100%
sure that the respondents articulated all their thoughts. If the study were repeated, I
would recommend that the participants be taught how to voice their thoughts. Both felt
that it was difficult to articulate thoughts and that they kept focusing on getting the right
answer, rather than their though process.
8
Sophie’s World
A Novel About the History of Philosophy
By: Jostein Gaarder
(Excerpt pp. 81-83)
Plato’s Academy
Thank you for the pleasant time we spent together, Sophie. In Athens, I mean. So now I
have at least introduced myself. And since I have also introduced Plato, we might as well
begin without further ado.
Plato (428-347 B.C.) was twenty-nine years old when Socrates drank the hemlock.
He had been a pupil of Socrates for some time and had followed his trial very closely.
The fact that Athens could condemn its noblest citizen to death did more than make a
profound impression on him. It was to shape the course of his philosophic endeavor.
To Plato, the death of Socrates was a striking example of the conflict that can
exist between society as it really is and the true or ideal society. Plato’s first deed as a
philosopher was to publish Socrates’ Apology, an account of his plea to the large jury.
As you will no doubt recall, Socrates never wrote anything down, although many
of the pre-Socratics did. The problem is that hardly any of their written material remains.
But in the case of Plato, we believe that all his principal works have been preserved. (In
addition to Socrates’ Apology, Plato wrote a collection of Epistles and about twenty-five
philosophical Dialogues.) That we have these works today is due not least to the fact that
Plato set up his own school of philosophy in a grove not far from Athens, named after the
legendary Greek hero Academus. The school was therefore known as the Academy.
(Since then, many thousands of “academies” have been established all over the world.
We still speak of “academics” and “academic subjects.”)
The subjects taught at Plato’s Academy were philosophy, mathematics, and
gymnastics – although perhaps “taught” is hardly the right word. Lively discourse was
considered most important at Plato’s Academy. So it was not purely by chance that
Plato’s writings took the form of dialogues.
The Eternally True, Eternally Beautiful, and Eternally Good
In the introduction to this course I mentioned that it could often be a good idea to
ask what a particular philosopher’s project was. So now I ask: what were the problems
Plato was concerned with?
Briefly, we can establish that Plato was concerned with the relationship between
what is eternal and immutable, on the one hand, and what “flows” on the other. (Just like
the pre-Socratics, in fact.) We’ve seen how the Sophists and Socrates turned their
attention from questions of natural philosophy to problems related to man and society.
9
And yet in one sense, even Socrates and the Sophists were preoccupied with the
relationship between the eternal and immutable, and the “flowing.” They were interested
in the problem as it related to human morals and society’s ideals and virtues. Very
briefly, the Sophists thought that perceptions of what was right or wrong varied from one
city-state to another, from one generation to the next. So right and wrong was something
that “flowed.” This was totally unacceptable to Socrates. He believed in the existence of
eternal and absolute rules for what was right or wrong. By using our common sense we
can all arrive at these immutable norms, since human reason is in fact eternal and
immutable.
Do you follow, Sophie? Then along comes Plato. He is concerned with both
what is eternal and immutable in nature and what is eternal and immutable as regards
morals and society. To Plato, these two problems were one in the same. He tried to
grasp a “reality” that was eternal and immutable.
And to be quite frank, that is exactly what we need philosophers for. We do not
need them to choose a beauty queen or the day’s bargain in tomatoes. (This is why they
are often unpopular!) Philosophers will try to ignore highly topical affairs and instead try
to draw people’s attention to what is eternally “true,” eternally “beautiful,” and eternally
“good.”
We can thus begin to glimpse at least the outline of Plato’s philosophical project.
But let’s take one thing at a time. We are attempting to understand an extraordinary
mind, a mind that was to have a profound influence on all subsequent European
philosophy.
10
Oral Comprehension Assessment
Pre-reading Instructions: Please read the excerpt from Sophie’s World by Jostein
Gaarder. Once you have finished reading I am going to ask you 5 questions about
the selection. Please let me know when you are done.
Post-reading Instructions: I am going to read a question to you. Once I am finished, I
want you to tell me orally what you are thinking in order to figure out an answer. I am
going to tape your response so that I don’t miss any aspect of your thought process. The
first question will be used as a practice question in order to clarify any procedural
questions. You should feel free to ask me questions in this phase if you are unsure about
anything. I will not be able to answer any questions for you once I ask the second
question. During all 5 questions, you are free to go back to the text to find the answer.
Comprehension Questions – Sophie’s World
1. Who was Plato’s teacher? (Practice Question)
2. Who is the narrator in the story and who is Sophie?
3. Define “hemlock” and “immutable.”
4. “[Plato] …a mind that was to have a profound influence on all subsequent
European philosophy.” Why do you think the narrator makes this statement?
Give examples to support your answer.
5. The narrator states that philosophers are often unpopular. Think of a modern day
philosopher and describe how society views this person.
11
Bibliography
Bartlett, F.C., Remembering - A Study in Experimental and Social Psychology,
Cambridge University Press, 1932/1995
Carroll, D.W., Psychology of Language, Thompson Learning Inc., Belmont, CA, 2004.
Conrad, Frederick, Johnny Blair and Elena Tracey, “Verbal Reports Are Data! A
Theoretical Approach to Cognitive Interviews,”
<www.fcsm.gov/99papers/conrad1.pdf> (October 23, 2004).
Ericson, K.A., “Protocol Analysis and Verbal Reports,” 2002,
<http://www.psy.fsu.edu/ericsson/ericsson.proto.thnk.html> (October 23, 2004).
Newell A., H.A. Simon, “Cognitive Architecture,”
http://www.rci.rutgers.edu/~cfs/472_html/CogArch/Protocol.html (October 23,
2004).
12