Instructional Materials (Print-based): Criteria to Consider
Many of you are will be working on studies that call for assessment of print-based materials (content analyses, perhaps). Following are elements that evaluators commonly consider when tasked with such an assignment. Two exceptionally good resources you’ll want to consult include:
 Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational

objectives. New York: Longman.
Marzano, R. J. (2001). Designing a new taxonomy of educational objectives.
Thousand Oaks, CA: Corwin Press, Inc.
1.
Objectives
Does the program have them?
Are they well-written?
Are they worthwhile?
Comments:
2.
Content
Does the content relate directly to the objectives?
If not, how does the extra/ancillary material add to or detract from the program?
Comments:
3.
Presentational style (flow)
Are learners given the information they need to attain each objective?
How is the information written (e.g., is it clear and to the point)?
Are examples provided?
Do the examples provide a range of possibilities? Do they reflect a spectrum of instances?
Comments:
4.
Practice
Is practice or review provided for each objective?
What form does it take (e.g., written or oral)?
© MJBober, 1998 (rev. 2000, 2004, 2008) 1

Does it seem to be sufficient (quality, scope, etc.)?
Is practice/review matched to the objective(s)?
Are the practice items and directions to learners/teachers well written?
Is constructive feedback (relative to skill, understanding, or progress) provided
(or made available)?
Comments:
5.
Assessment
How is learner performance assessed?
Are the objectives, the content, and the assessment aligned?
What is the quality of the writing (of the test items)?
How are the objectives weighted?
Are the directions for administering assessment clear?
How is the assessment scored? How difficult a task is scoring?
Is scoring subjective (norm-referenced) or objective (criterion-referenced)?
Comments:
6.
Appeal
Is there variety in the types of activities?
How are learners motivated to complete the tasks before them? What hooks them?
In what ways, if any, is the product appealing (e.g., representative of good design principles)?
Is the format easy to follow? Is it consistent?
Do graphics/icons/symbols enhance or detract from comprehensibility?
Does the format seem to fit the content?
Comments:
7.
Efficiency
© MJBober, 1998 (rev. 2000, 2004, 2008) 2

Do the objectives justify the length (amount of time) of the program
(unit/lesson)?
Are there opportunities for enrichment or remediation?
Comments:
8.
Ease of use
Instructor/facilitator
[If not self-paced] can instructors/facilitators learn to “use” the program easily?
[If the program (unit/lesson) is not self-contained] are the set-up notes clearly delineated?
Learner
[Whether or not self-paced] are directions/instructions for how to proceed clear and unambiguous?
Are outcomes/performance criteria (for activities) explained?
*Are instructor and learner materials well-aligned? Do they complement one another?
Comments:
9.
What are three to five positives about the product?
10.
What are three to five negatives about the product?
© MJBober, 1998 (rev. 2000, 2004, 2008) 3






Other issues you might want to consider:
 Appropriate for multiple ways learners might be grouped (individually, teams, whole class, etc.)?
Mix of instructional strategies?
Promoting higher-ordering thinking, complex skills, problem-solving?
Underlying instructional philosophy?
Adaptable for different learning styles and/or ability levels?
Scalable for learners with differing prerequisite skills/competencies?
© MJBober, 1998 (rev. 2000, 2004, 2008) 4

Instructional Materials (Print-based): Criteria to Consider
By now you’ve realized that the criteria list above has limited utility. First, its qualitative nature could translate into hours of laborious analysis. As important, its structure doesn’t well attend to potential rater bias/subjectivity.
Following, then, are some ways to transform it into a scaled tool easily used (after brief training) by multiple raters.
1.
Objectives
Evident?
Well-written?
Worthwhile?



Yes
Yes
Yes



No
No
No
Overall rating
Deficient

Adequate

Good

Excellent

Comments:
Deficient

Adequate

Good

Excellent

2. Content
Extent to which content relates directly to the objectives?
Comments:
© MJBober, 1998 (rev. 2000, 2004, 2008) 5

3. Presentational style/flow
Learners given the information they need to attain each objective?
Information clear and to the point?
Quality of the examples provided?
Examples provide a range of possibilities OR reflect a spectrum of instances?
Comments:
Deficient

Deficient


Deficient

Deficient
4. Practice
Practice or review provided for each objective?
Form of the practice


Yes
Oral
Adequate

Adequate

Adequate

Adequate



No
Written
Good

Good

Good

Good

Excellent

Excellent

Excellent

Excellent

© MJBober, 1998 (rev. 2000, 2004, 2008) 6

Sufficiency (quality, scope)?
Extent to which practice/review matches objectives?
Quality of the directions?
Deficient

Deficient

Deficient

Comments:
Etc ...
Adequate

Adequate

Adequate

Good

Good

Good

Excellent

Excellent

Excellent

© MJBober, 1998 (rev. 2000, 2004, 2008) 7