Evaluating Textbooks
Mini Workshop
Mary Barrett
Mayo High School
1420 SE 11th Avenue
Rochester, MN 55904
mabarrett@rochester.k12.mn.us
Evaluating Textbooks
2
Assessing Text Difficulty – Beyond Readability Scores
Criteria Chart
Sentence
Structure
Short sentences;
familiar patterns
(subject, verb, object);
active voice
2 – Moderately
Difficult
Mostly familiar
words, multiple
meanings; concrete
concepts;
connotation; strong
contextual support;
some unfamiliar
words
Combination of
simple, compound,
and complex
sentences
Text
Organization
& Structure
Narrative; well
organized and
developed, with topic
and supporting
sentences; clear
structure or pattern of
organization
Expository;
organized around
common patterns of
nonfiction; topic
sentences with some
development of
details to support MI
Few ideas; concrete;
logically developed;
contains examples and
incidents to illustrate
the premise
Multiple ideas
around a central
concept; fairly
concrete; context and
prior knowledge
partially supplied
Some prior
knowledge needed
but some provided;
somewhat interesting
topic; age appropriate
Multiple ideas; no
central unifying idea;
unfamiliar concepts;
allusions;
decontextualized
Multiple and abstract
ideas; unfamiliar
concepts; contains
allusions & metaphors;
decontextualized
Reader must have
prior knowledge; topic
may be remote or
inaccessible; requires
cognitive maturity
Requires extensive prior
knowledge; relies on
abstract
conceptualization; topic
may be esoteric and
academic; requires
cognitive maturity
Comprehension is
aided by graphics;
bold headings,
illustrations, italics,
etc.
Informative; straightforward and without
subtlety
Text may contain
graphics, headings,
illustrations, etc. that
aid comprehension
Text is largely print with
little if any support from
text features
Narrative
Narrative/Expository:
Informative
Text may contain
detailed and complex
illustration of
concepts; may be in
addition to text
Informative or
persuasive, much
subtlety in idea
development; some
manipulation of ideas
through rhetorical
devices
Expository:
Informative/Persuasive
Criteria 
Vocabulary
Idea Density
Suitability
Text Features
Author's
Purpose
Genre
1 – Accessible
Familiar words with
concrete meanings;
strong contextual
support
No prior knowledge
needed or prior
knowledge is
provided; high
interest; not
challenging culturally;
age appropriate
Informative or
persuasive, but
straight-forward for
the most part; logical
development
3 - Difficult
Unfamiliar words;
multiple meanings;
abstract; some words
with connotation;
some contextual
support
4 – Extremely Difficult
Many unfamiliar words,
multiple abstract
concepts; very little
context; multiple
meanings; connotative
meanings
Longer sentences with
layers of subordination; phrases,
clauses, appositives;
passive voice
Expository; well
organized; many ideas
with few or no
supporting details or
elaboration;
Longer sentences with
layers of subordination;
passive voice;
unfamiliar, unusual, or
archaic structure
Expository; intrinsically
organized; explicit and
implicit premises and
conclusions; some
development with
supporting details
Informative or
persuasive; a great deal
of subtlety and
manipulation of ideas;
fallacies of logic;
rhetorical devices
Expository:
Informative/Persuasive
Evaluating Textbooks
3
Readability: Qualitative Assessment of Text Difficulty
Jeanne S. Chall
1. Choice of materials should include decisions about
 content in relation to the experiences and interests of the students,
 repetition of language patterns,
 vocabulary that is familiar, appropriate, or in context,
 illustrations that support text, and
 narrative or expository style and how well organized the ideas are.
2. Narrative passages are considerably easier to read than expository passages.
3. Difficult text is the result of
 vocabulary that is less common, less frequent, less familiar, more
abstract, technical, and/or literary;
 sentences that are longer, more complex, less direct, with greater
embedding of ideas;
 concepts that are less familiar and more abstract; and
 a high demand for cognitive processing, such as thought, inferences,
reasoning, analysis, and critical awareness.
Generally speaking, the readability scale used to determine text difficulty does not
matter; all scales give about the same results. Here are the names of some
common readability scales:
Fry
 visit http://school.discovery.com/schrock.guide/fry/fry_ages.gif
DRP (Degrees of Reading Power)
Flesch-Kincaid (Grade Equivalency or Scale of Reading Ease from 6 to 70)
Lexiles comes with the Stanford)
 http://www.lexile.com/DesktopDefault.aspx?view=ed8&tabindex=5&
tabid=18#3
SMOG
For an overview of several readability formulas and choosing appropriate texts,
visit:
 http://csep.psyc.Memphis.edu.cohmetrix/readability.research.htm
 http://www.thinkport.org/classroom/oftinteractive/readingtips/matchin
gtenets
 http://sarasota.k12.fl.us/sarasota/choosngapprtext.htm
Evaluating Textbooks
4
Qualitative Assessment of Text
SCALE =
1 (ACCESSIBLE)
2 (MODERATELY DIFFICULT)
3 (DIFFICULT)
4 (EXTREMELY DIFFICULT)
1.
COGNITIVE DIFFICULTY
The student's cognitive development affects the level and ease of comprehension. The more
challenging text might require prior knowledge, cultural knowledge, and an understanding of
allusions, etc. that are not explicitly stated in text. The concepts might require students to
perceive multiple layers of meaning, great subtlety in word choice, and the quality of the
development of ideas. The reader may have to analyze the logic of reasoning, analyze ideas,
make inferences, and have a critical awareness of text. The amount of interpretation that is
demanded of the reader helps to determine text difficulty.
2.
TEXT ORGANIZATION
The text will range from fairly straightforward to relatively subtle. The text that is easier to read
would have clearly stated premises followed by well-developed and plentiful details. While the
text would be rich with literal content, it would also provide for interpretive and evaluative
questions. The text that is more difficult might have a premise that must be inferred, fewer
details and less development. Additionally, the text might include allusions and make
assumptions about the reader's broad prior knowledge and wide experience.
Text may be organized in a predictable pattern. Informative patterns would include cause/effect,
compare/contrast, topical, and chronological. Persuasive patterns would include
problem/solution, opinion/reason, and thesis/proof. Organizational patterns help with
comprehension in two ways. The reader can use the pattern to help him organize the ideas into
manageable chunks of understanding. The reader can also make predictions and ask questions
based on the organization.
3.
CONCEPTUAL DIFFICULTY
Text can have differing levels of density. Some text contains only a single idea followed by
elaboration that explains and develops that idea. More difficult text has a great deal of idea
density, very often without context or background knowledge.
Concepts may be concrete or abstract. Text that contains concrete ideas tends to be easier to
comprehend than text that is more abstract.
4.
SOPHISTICATION
SENTENCE STRUCTURE
The text could be simply written, with many simple and compound sentences with a common
structure of Subject-Verb-Object. More difficult text would have longer, more complex
sentences with multiple layers of subordination, including clauses, phrases, appositives, and other
structures that contribute to complex structure. More complex text would also contain a greater
number of passive sentences.
Evaluating Textbooks
5
TEXT STRUCTURE
More accessible text is organized with clearly stated topic sentences followed by supporting
details and summarized in a concluding sentence. Appropriate transitions link the flow of ideas
and the flow of paragraphs. The author provides any context necessary for full understanding.
More difficult text requires the reader to make more inferences and may be less fully developed
or rely more on allusions and prior knowledge. The organizing idea might occur in the middle or
at the end of a section, or the reader might have to infer the premise and conclusion.
VOCABULARY/WORD CHOICE
The words selected would reflect a more difficult vocabulary with words that have very specific
meanings as opposed to a more general meaning (for example "conflagration" as opposed to
"fire"). The vocabulary in the text might appear in the text with less frequency. Words that occur
frequently are easier and more familiar; words that are rarer tend to be more difficult. The type of
words in the text will contribute to its level of difficulty. Words chosen might have multiple
meanings, and the author might play with those meanings through puns. The reader would have
to be aware of both the connotation and denotation of words in order to fully comprehend the
author's meaning and intent.
5.
AUTHOR'S PURPOSE
The author's intent could be to inform or to persuade. In persuasive articles, the reader would
need to distinguish attempts to manipulate the reader's opinion. The reader would need to
recognize bias, logic of reasoning, and other persuasive and rhetorical techniques. The presence
of rhetorical manipulation makes the text exponentially more difficult.
6.
GENRE
Text that is narrative is easier to understand since much of our lives revolve around stories.
Stories are how we make sense of the world. Expository text tends to be more abstract and to rely
more on prior knowledge, specialized vocabulary, and good organization.
7.
TEXT FEATURES
Text may contain special features to aid comprehension. Examples of the special features would
include bold-faced headings, charts or other illustrations, margin glosses, specific introductions
and conclusions. Text that is devoid of organizing features is more difficult to process and
comprehend.
Evaluating Textbooks
6
Readability and MS Word
To turn on readability in MS Word:
1.
Open up a page in MS Word
2.
Click on the TOOLS menu
3.
Pull the highlight bar down to OPTIONS
4.
Click on the file folder tab titled SPELLING AND GRAMMAR
5.
Check the bottom two boxes:
√ Check grammar with spelling
√ Show readability statistics
6.
Click OK. You are now set.
7.
Each time you click SPELL CHECK, the last screen you see will tell
you the readability of the text.
8.
Look at the Flesch-Kincaid grade level to find readability. If it says
“7.4” that means difficulty is suitable for a student in the 4 th month of
7th grade. You can put this grade equivalency up to the DRP chart to
see how the scores relate. A Flesch-Kincaid score of 12.0 will be
about a 68 or 69 for DRP. The Scale of Reading Ease rates text from
6 to 70, with 6 being the easiest.
Accurate readability
1.
The text must be at least 100 words long. Since readability is based
on sentence length, try to finish a sentence even if that means having
98 words or 107 words.
2.
If you want the readability of something of the length of a short story,
type in three passages of 100 words each. Choose 100 words from the
beginning, 100 words from the middle, and 100 words from the end of
the passage.
3.
If you want the readability of an entire textbook, follow the steps in #2
for at least 3 chapters in the book since a different author probably
wrote each chapter. Average the readability from the three chapters.
Don’t retype
If you have access to an article on the Internet (for example at the web site for the
Pioneer Press or the Washington Post or Time magazine), you can highlight the
article, copy from the EDIT menu, go to a Word document and PASTE the article.
Once you have the text in Word, you simply need to run Spell Check to get the
readability. You will also get a word count, which is useful if you are working
with improving rate.
331 Words; 8.1 GE
Evaluating Textbooks
7
Checklist for Reviewing Reading/Literature Materials
2003-2004
Name:
Program Title:
JOB TITLE:
SCORING
I.
II.
Low Adequate High
1
2 3 4
5
QUALITY OF READING MATERIAL
A. The literature is appropriate developmentally.
B. The vocabulary is appropriately controlled at K-2.
C. The literature can be flexibly integrated with other disciplines.
D. Multiculturalism and gender equity are evident in pictures, stories, authors, and illustrators.
E. The program offers a balance of fiction, nonfiction, and technical reading in authentic reading selections.
F. The program matches the district goals for reading education.
G. Strategies and instructional methods reflect current best practice and research.
H. The materials are visually appealing.
INTEGRATION OF LANGUAGE ARTS
A. The program includes authentic examples and explicit instruction in comprehension of the following areas in
an integrated manner:
1. Narrative text
2. Expository text
3. Descriptive text
4. Technical text
5. Writing
6. Listening skills
7. Speaking
8. Spelling
9. Grammar and Conventions
SCORING
B. The program offers instruction in the writing process, including brainstorming, graphic organizers for
1
2
3
4
5
Evaluating Textbooks
8
planning and organizing writing, six-traits, and scoring rubrics.
C. Daily Oral Language is a component of the program
D. the program includes vocabulary acquisition strategies such as context clues and word analysis.
E. The series includes instructional materials on comprehension strategies for use in the content area.
III.
ASSESSMENT
A. A variety of assessment methods is available, including performance assessment.
B. A management system for documenting student progress is included.
C. Assessment tools provide diagnostic information.
D. A skills inventory is included.
E. assessment materials are organized and easy to use.
F. Assessment results can be used for graduation standards assessment and reporting.
IV.
TEACHER’S MANUAL
A. The teacher’s manual is clearly labeled for efficient lesson planning and organizing.
B. Management tips help the teacher plan for both large group and flexible small-group instruction.
C. The skills being tested are clearly identified.
V.
SUPPLEMENTAL MATERIALS
A. All recommended resource materials are complete and packaged for ease of use.
B. The program has software available and other software suggestions are included.
C. Multimedia resources are available in the program.
D. Workbooks and activity sheets are available and have black line master alternatives.
E. The program provides support.
F. Technology is integrated into lessons.
SCORING
VI.
PREPARATION TIME
A. The amount of time necessary to become familiar with materials is reasonable.
B. The vendor provides on-site training in use of the program.
VII.
PLEASE ADD YOUR OWN COMMENTS
1
2
3
4
5
Evaluating Textbooks
9
Evaluating Textbooks
10
Checklist for Qualitative Assessment of Text
Yes/No
Reviewer:
Date:
Target Audience:
Grade Level:
Course:
Course Number:
Textbook Title:
Authors/Editors:
Publisher:
Copyright/Publication Date:
ISBN:
Number of Pages:
Number of Chapters:
TEXT FEATURES
1.
Table of Contents
2.
Author Information
3.
Purpose Statement
4.
Chapter Introductions
5.
Key Vocabulary Introduced
6.
Graphics/Illustrations/Maps
Quality
Up to date
9.
Comprehension Aids
Margin Glosses
Timelines
Italics
Graphic Organizer
10.
Chapter Questions
11.
Questions fostering critical thinking.
12.
Chapter Summaries
13.
Glossary
14.
Index
15.
Other special features (please list)
Overall evaluation of text’s functionality:
Intermediate
Bold-faced vocabulary
Bold/Color Headings
Final
Evaluating Textbooks
11
QUALITY OF WRITING IN TEXT
1.
Writing is well organized and unified
2.
Text contains anecdotes to help students make connections.
3.
Writing is cohesive and contains transitions to show relationships in ideas.
4.
Author provides background knowledge when students would not have prior knowledge.
5.
Text density is offset by anecdotes, primary source quotes, illustrations, activities, etc.
6.
Abstract ideas are demystified with anecdotes, metaphors, analogies, etc.
Overall evaluation of the quality of the writing:
Rate Text Density:
0
Very Dense
1
2
3
4
Moderately Dense
5
6
7
Accessible
8
9
10
Easily Accessible
TEACHER/STUDENT SUPPORT MATERIALS
What materials are available for teachers to use with this text? Code as F for free or as C for costs money
to purchase. If the item is not available, write N.
TEACHER
STUDENT
Test bank
Practice quizzes
Lesson plans or designs
Study Guides
PowerPoint presentations
Online Resources
Supplemental readings
Online HW Help
Student workbooks
CD ROM
Blackline masters for duplicating
Simulations
Lab experiences/applications
Other (Please list)
Color OHs and other graphic presentations of data
Online access to lessons, tests, course outlines, etc.
Taped/Braille version for DHH
CD Rom version for Visually Impaired
Other (Please list)
Overall evaluation of the supplemental materials: