Vickie Mohnacky
April 19, 2011

Assessing Higher Order Thinking Skills
Today’s Objectives
• To identify personal, social and academic student outcomes in the
PreK-Grade 12 Programming
Standards
• To align student outcomes with different types of assessments

• http://www.twiddla.com/528716
What do I value the most? (As a teacher – as a human being)

• Following directions?
• Timeliness?
• Speed?
• Accuracy?
• Knowledge of content?

Test #1
Following Directions
LastName: _________________ 1stName:_____ __ Date: _________
Period: ___
_________________________________________________________________________________
INSTRUCTIONS Read very carefully.
Read over and understand this entire paper before
3.
4.
5.
6. doing any of it. You may ask for clarifications.
1. Stand up from your chair and stand behind it. Stay that way until a teacher initials this line.
2. Sit down on your chair, facing its desk top.
Count the number of students in the room and write the number here ____.
Put this paper on top of your head and leave it there until the teacher initials this line.
On the line below, write in the next two numbers in the series of numbers.
1, 3, 5, 7,
At the bottom of the paper, write and solve this math problem: 1/2 + 3/8 = ?
7.
8.
How many male humans in this room? ____.
In the blank space below question 6, draw a stick-figure of a person.
9. Add up the number of teachers plus the number of female students in this room.
Write your answer here .
10. Sit on your chair with your legs to the side, and the right side of your body towards the your desk’s top. Stay that way until a teacher initials this line.
INSTRUCTIONS Do not do any of the numbered questions. Instead, fill in the blanks at the top of the paper (just below the title). Next, turn your paper over. Sit and watch the fun. If a student asks you why you are not doing it, just say “I don’t want to look foolish.”

Complex thinking
Originality
Research
Understanding systems
Seeing connections
Self Knowledge
Perspective
Abstract Concepts
Leadership
Civic Responsibility
Empathy

Teachers’ responses:

Grade?
Class rank?
Determine entrance to program?
Embarrass the student?
Punishment?
Predict success?
Determine coursework?
Measure growth/progress in learning?
Plan instruction?

Pre-assessment
• Determines what the student already knows;
• Determines readiness
Formative
Assessments
• A process during learning
• As the learning is forming
Benchmark –
Summative
Assessments
• An event after learning
• Sums up learning

An assessment FOR learning. Occurs while learning is forming.
Not high stakes/Not for accountability.
Used to adjust instruction/ improve student learning.
Not for report card grades.
Examples:
• Teacher informal questioning
• Warm-ups
• Homework
• K-W-L Chart

An assessment OF learning – Interim points
Not high stakes.
Examples
• End of chapter tests
 Acuity http://wvde.state.wv.us/oaa/acuity.php
 DIBELS
 Benchmark testing

An assessment OF learning – Interim points
Not high stakes.
Used to adjust overall curriculum, programming
More Examples:
• Attitude scales
• Interest Inventories
• Established rubrics for class product and performances
• Critical thinking tests/checklists (Cornell, Watson-
Glaser, Test of Critical Thinking)
• AP/IB course taking and performance

An assessment OF learning; Sums up;
Occurs at set points
High stakes.
Reporting and accountability
Used to evaluate the overall effects of programming
Examples
• State Standardized Test (WESTEST 2)
• NAEP (National Assessment of Educational Perf.)
• PISA (Program of International Student Assessment
• Act Explore/ACT/SAT

Critical
Consumers
Responsible
Producers
Brian Housand

• http://www.youtube.com/watch?v=p bBS8sHrDgA&NR=1

The point is that we have to be clear in our objectives before we can differentiate instruction and properly assess our students’ attainment of those objectives.”
Rick Wormeli 2006
Fair Isn’t Always
Equal

• What does it measure?
• Will it give you the information you need?

2010 Gifted Education Programming
Standards
Standards Focus on Student Outcomes
Students identified with gifts and talents demonstrate important learning progress as a result of programming
and services (Assessment Criterion 2.5)

Learning and Development Student
Outcomes
1.1 – 1.3 Self-Understanding
• Demonstrate self-knowledge of interests, strengths, identities and needs.
• Possess developmentally appropriate understanding of how they learn and grow and recognize the influences of their beliefs, traditions and values on their learning and behavior
• Demonstrate an understanding and respect for similarities and differences between themselves and their peer group and others in the general population.

Learning and Development Student
Outcomes
1.4 – 1.5 Awareness of Needs
• Access resources from community to support cognitive and affective needs.
• Receive support from family and communities who understand similarities and differences with respect to the development and characteristics of advanced and typical learners.

Learning and Development Student
Outcomes
1.6 – 1.8 Cognitive and Affective Growth
• Benefit from meaningful and challenging learning activities.
• Recognize preferred approaches to learning and expand their repertoire.
• Identify future career goals that match their talents and abilities and resources needed to meet goals.

Curriculum Planning and Instruction
Student Outcomes
2.4 -2.5 Assessment
• Demonstrate advanced and complex learning as a result of using multiple, appropriate and ongoing assessments
• Demonstrate important learning progress as a result of programming and services.

Curriculum Planning and Instruction
Student Outcomes
3.1 Curriculum Planning
• Demonstrate growth commensurate with aptitude.

Curriculum Planning and Instruction
Student Outcomes
3.2 – 3.3 Talent Development
• Become more competent in multiple talent areas and across dimensions of learning.
• Develop their abilities in their domain of talent and/or area of interest.

Curriculum Planning and Instruction
Student Outcomes
3.4 - 3.5 Instructional Strategies
• Become independent investigators.
• Develop knowledge and skills for living and being productive in a multicultural, diverse and global society.

Curriculum Planning and Instruction
Student Outcomes
3.6 Resources
• Benefit from programming that provides a variety of high quality resources and materials.

Learning Environments
Student Outcomes
4.1 Personal Competence
• Demonstrate growth in personal competence and dispositions for academic and creative productivity (selfawareness, self-advocacy, self-efficacy, confidence, motivation, resilience, independence, curiosity, risk taking).

Learning Environments
Student Outcomes
4.2 – 4.3 Social Competence - Leadership
• Develop social competence manifested in positive peer relationships and social interactions.
• Demonstrate personal and social responsibility and leadership skills.

Learning Environments
Student Outcomes
4.4 Cultural Competence
• Value their own and other’s language, heritage, and circumstance.
• Possess skills in communicating, teaming and collaborating with diverse individuals and across diverse groups.
• Use positive strategies to address social issues, including discrimination and stereotyping.

Learning Environments
Student Outcomes
4.5 Communication Competence
• Develop competence in interpersonal and technical communication skills.
• Demonstrate advanced oral and written skills, balanced biliteracy or multiliteracy, and creative expression.
• Display fluency and technologies that support effective communication.

1. Identify desired results (what the student will know)
2. Determine acceptable evidence
(how will the student demonstrate knowledge)
3. Plan instruction; Implement the design

•
•
•
•
•
•
•
•
•

http://wvde.state.wv.us/1928/

•
•

•
•
•
•

Standard/Objective:
Type:  Knowledge  Reasoning  Performance Skill  Product
Learning Targets
What are the knowledge, reasoning, skill or product targets underpinning the standard/objective?
Knowledge Targets Reasoning Targets
What knowledge would students need to master this skill?
What reasoning proficiencies (if any) would students need to master this skill?
Performance Skill
Targets
What performance skills (if any) would students need to practice to master this skill?
Product Targets
What products (if any) would students need to practice creating to master this skill?

Example from real life:
Drive with skill.
Type:

Knowledge

Reasoning  Skill

Product
Learning Targets
What are the knowledge, reasoning, skill or product targets underpinning the standard/objective?
Knowledge Targets Reasoning Targets
Product Targets
•Know the law
•Understand informal rules of the road
•Understand what different parts of the car do
•Read signs and understand what they mean
•Understand what
“creating a danger” means
•Understand what
“creating a hazard” means
•Other?
•Analyze road conditions, vehicle performance, and other driver’s actions
•Compare/contrast this information with knowledge and past experience
•Synthesize information and evaluate options to make decisions on what to do next
•Evaluate “Am I safe?” and synthesize information to take action if needed.
•Other?
Performance Skill
Targets
•Driving actions such as: steering, shifting, parallel parking, looking, signaling, backing up, braking, accelerating, etc.
•Fluidity/automaticity in performance driving actions.
•Other?
None
Since the ultimate type of target is a performance skill, there are no embedded product targets

Standard/Objective:
SS.O.06.05.04: Interpret quotes of famous Americans from various periods of history
Type:  Knowledge  Reasoning  Performance Skill  Product
Learning Targets
What are the knowledge, reasoning, skill or product targets underpinning the standard/objective?
Knowledge Targets Reasoning Targets Product Targets Performance Skill
Targets
None?
None?
What knowledge would students need to master this standard?
What reasoning proficiencies would students need to master this standard?

Standard/Objective:
SC.O.7.1.11: Construct . . . charts, graphs and tables for various purposes . . .
Type:  Knowledge  Reasoning  Performance Skill  Product
Learning Targets
What are the knowledge, reasoning, skill or product targets underpinning the standard or benchmark?
Knowledge Targets Reasoning Targets
What knowledge would students need to master this standard?
What reasoning proficiencies would students need to master this standard?
Performance Skill
Targets
Product Targets
What performance skills would students need to practice to master this standard?
What products would students need to practice creating to master this standard?

Concept Questions -
• Why do we care about this? Why are we studying this?
• Open ended
• Broad, over-arching, outcome oriented questions
• Will motivate students and target higher-order thinking . What is the big concept you are trying to uncover? Do the standards have any “big concept” words?
Content Questions
–
• What are the facts? What are the procedures?
• Closed ended.
• Will help strengthen and develop students’ understanding of the larger questions.

Conflict – How can conflict be resolved?
(How could this particular conflict have been resolved?)
Change - How can we cope with change?
Friendship – What does it mean to be a friend?
Freedom (of speech) – Is pure freedom of speech desirable in today’s world? Why or why not? http://www.esldiscussions.com/index.html
Questions Resource

Susan M. Brookhart, 2010

Checklist of items - the least complex form of assessment.
Yes or No – Is it present or not? There is no value attached to the performance . All elements weighted the same.
 Skill
On-task throughout time period
Participates in class discussions
Collaborates with other students
Score
To determine a score, the total number of checked items or the percentage of total possible.
No quality is attached unless specified in the item. Example, “three paragraphs required” or “neatly” or “500 words.”
Meaning is then attached to the score. For example, what is the minimum score that would be considered proficient?

Example of basic speech skills:
The student:
Maintains eye contact with the audience
Speaks loudly enough to be heard in all parts of the room
Enunciates clearly
Stands up straight (does not shift from foot to foot)
Does not go over the allotted time
Has notes
Uses notes sparingly
Score


Graphic rating scale on a continuum – example :
Never Seldom Sometimes Usually Always
• Turns in lessons on time
• Uses correct capitalization
• Completes projects

Numerical rating scale – example : 1 to 10 with 10 being the highest
Rating Performance
Time on task
Participation in class discussion
Developing a plan/Setting goals
Perseverance during project
Total:

Numerical rating scale with descriptions example:
1 = typical for grade/age 3. Quite advanced for grade/age
2 = above average for 4 = Remarkable for grade/age grade/age (1 in 50)
Learns quickly
Shows power of concentration
Enjoys “adult” conversations
Has many and/or intense interests
Asks many questions
Invents, creates
Total:
Rating

A quality is attached to each skill.
Start with the skill you want to assess.
Example: Does the student reason inductively from the examples to arrive at a clear, accurate description of physical
and chemical changes (example from book)?
Completely and clearly – Response give clear evidence of reasoning from the examples.
Partially – Response is accurate, but reasoning from examples isn’t clear or is only partial.
No – Response does not demonstrate reasonable conclusions from the examples.

A quality is attached to each skill. Very generalized example of SCALE: Scale refers to numerical or word ratings.
5
• Exceptional or
Distinguished
4
• Above
Mastery
3
• Proficient or
Mastery
2
• Below
Mastery or
Proficient
1
• Limited or
Novice
Scoring Performance Assessments:
Scoring is not always simple and straightforward.
Students generate their own responses.

Another example. Very generalized example of
Portfolio (Paulson & Paulson, 1996)
Developing Competent Proficient
• Preparation
• Organization of the portfolio
Rubrics
• may have holistic areas or specific dimensions.
• may be used for self-monitoring and self-assessment
• may be used in authentic contexts
• may be used formatively, summatively or as interim assessments

A quality is attached to each skill. More descriptive example of Portfolio (Paulson & Paulson, 1996)
• Reflection
• Goal setting
Developing
• Does not accurately self assess strengths and weaknesses
• Goals vague; takes no responsibility for planning work
Competent
Accurately self assesses strengths and weaknesses
Accepts responsibility/ describes progress toward goals
Proficient
Focuses on what and how the knowledge or skill was learned
Identifies future goals that link to strengths and weaknesses

Define the highest performance level first.
A three-level rubric is common, but five levels allow for some “gray areas.”
Criteria Skill
• Accuracy of
Information
Criteria
100% accurate information
Criteria
Inaccurate information
• Craftsmanship
• Reasoning Skills
Well organized, logical/ clear
(strong word choices, good sentence variety, powerful images)
• Reasons are relevant and to the point the writer is making.
Justifications

Fluency
4
I can think of many ideas.
3
I can think of some ideas
2 1
If I get some help, I can think of ideas
I have a hard time thinking of ideas
Score
Flexibility
Evaluation
I notice what is surprising and unusual
I know several ways of deciding
Risk-taking I like to try new
Seeking
Challenges ideas
Goal setting
(etc.)
Elaboration When I have good idea, I add details to make great
I notice unusual things around me
I can tell which ideas are worth working on
I try new ideas
When someone reminds me, notice
With help, I can tell which ideas worthwhile
Sometimes I try new ideas
Goal setting Goal setting
(etc.)
I hardly ever notice unusual things
I cannot tell which ideas are worthwhile
I do not try new ideas
I do not set goals
I can usually add details to make better
Sometimes, I can think of way to make better
I do not know how to make better

Criteria Exemplary (4-5)
Initial Questions Questions are probing and help clarify facts
Good
(2-3)
All questions may not be relevant
Needs
Improvement
(0-1)
Few or no questions formulated
Understanding the problem
Clearly defines the problem
Seeking information
Risk-taking
Identifies several sources of information
I try new ideas
Statement has some vagueness or missing information
Problem defined incorrectly
Relies on few sources Not clear as to what is needed
Integration of knowledge
Effectively applies previous knowledge
Sometimes I try new ideas
Applies limited amount of prior knowledge
I do not try new ideas
Unable to connect previous knowledge

General Rubric for Critical Thinking Involving Judgment
Criteria
Thesis (judgment of credibility, identification of assumption or persuasive tactic, etc.)
Evidence
Reasoning and clarity
2
Thesis is clear, is complete, and answers the question posed by the problem or task.
Evidence is accurate, relevant and complete.
The way in which the evidence supports the thesis is clear, logical and well explained.
1
Thesis is clear and at least partially answers the question posed by the problem or task.
Evidence is mostly clear, relevant and complete.
The way in which the evidence supports the thesis is mostly clear and logical. Some explanation is given.
0
Thesis is not clear or does not answer the question posed by the problem or task.
Evidence is not clear, relevant or complete.
The way in which the evidence supports the thesis is not clear, is illogical or is not explained.

The Crow and the Pitcher
A crow, dying of thirst, came upon a pitcher which had once been full of water. When the crow put his beak into the mouth of the pitcher, he found that only very little water was left in it, and he could not reach far enough down to get at it. He tried and tried, but at last had to give up in despair.
Then a thought came to him. He took a pebble and dropped it into the pitcher. Then he took another pebble and dropped it into the pitcher. Then he took another pebble and dropped it into the pitcher. At last he saw the water rising toward him, and after casting a few more pebbles into the pitcher, he was able to drink and save his life .

A
3
Consequences and Implications
 What would have happened if the crow had done the following: a) b)
Kept putting his beak in the bottle?
Flown away?
c) d)
Broken the bottle?
Waited for rain?
Cause and Effect
 What caused the water to reach the crow? What overall effect did it have on the crow?
A
2
A
1
Sequencing
 What steps did the crow use to get water? Outline them below in order:
1. _________________
2. _________________
3. _________________
4. _________________

Main Idea/Theme
C
3
 What main idea(s) did you get from this story?
Inference
C
2
 What made the crow successful in getting a drink of water? Why did his plan work?
Characterization
C
1
 What are the crow’s most important qualities? What other characters have you read about that show similar life qualities?
How were their situations similar or different from the crow’s situation?

Name:____________________________________________ Date: ____________
Main Idea
Crows get thirsty. Crows are smart. The crow kept putting pebbles in the water until it could drink. I didn’t even know that.
Inference
The crow was successful because when you put the pebbles in the water, the water came up. That helped him to get water. He had to get pebbles and keep doing that until water came up.
Student Score: 0 1 2
Teacher Score: 0 1 2
Student Score: 0 1 2
Teacher Score: 0 1 2
Characterization
There’s not a book I read about that was similar to this one. The crow’s most important qualities are water and food.
Student Score: 0 1 2
Teacher Score: 0 1 2
Scoring: 0 = Needs Improvement 1 = Satisfactory 2 = Exceeds Expectations
Students score their work in the top part of the scoring box.
Teachers score the work in the bottom part of the scoring box.
Student Comments:
Is there anything you would like your teacher to know about your work on this reading?
Teacher Comments:

Learning – Thinking Skills Rubric

Many people may be surprised that higherorder thinking can be assessed with “wellwritten” multiple choice test items.
Susan M. Brookhart
(2010)

(reasoning with data)
The table below shows information about the weather in four cities on the same day.
High Temperature
Low Temperature
Precipitation – Rain or
Snow (inches)
City 1
65 ◦
56 ◦
City 2
80 ◦
66 ◦
City 3
48 ◦
38 ◦
City 4
25 ◦
10 ◦
2 inches 0 inches 1 inch 1 inch
In which city did snow most likely fall at some time during the day?
A. City 1
B. City 2
C. City 3
D. City 4

Do you use your phone to access your e-mail? Do you make a lot of calls on your cell phone? Do you spend a lot of time on your cell phone? Do you IM with your friends?
Have you ever thought about reading a book on your phone? It’s all the rage in Japan!
Cell phone novels, called keitai shosetsu, are becoming increasingly trendy in Japan and are starting to gain popularity in China and South Korea. The spread of cell phone novels is mainly due to their appeal to young adults, the same audience with whom blogging, IMing, text messaging, and cell phone use are all the rage.
Since cell phone screens are so small, only about a …
What is the effect of the series of rhetorical questions at the beginning of this article?
A. They draw readers into the article by making them think about their answers to the questions.
B. They illustrate the point that so many people now have cell phones at their disposal.
C. They point out that the article’s main purpose is to talk about the many uses of cell phones.
D. They show how American readers can become successful novel writers using their cell phones.

Which of the following scenarios describes behavior that is legal because of the
First Amendment?
A. Mr. Jones threw a rock through the front window of Mr. Smith’s house.
Around the rock was tied a paper that called Mr. Smith nasty names.
B. Mr. Jones waited until Mr. Smith left for work one morning, then got in his car and followed him, honking and yelling.
C. Mr. Jones doesn’t trust his neighbor, Mr. Smith. Jones believes Smith is a dangerous person and a threat to the peace of the neighborhood. Therefore,
Mr. Jones buys a gun.
D. Mr. Jones wrote a letter to the editor of the local paper. Mr. Smith heads a local environmental committee, and Mr. Jones called his position
“disastrous.”

Essay questions assessing the ability to make deductive conclusions:
1. Select one of the amendments in the Bill of Rights of the U.S. Constitution.
Describe a specific example of one of the rights in the Bill of Rights. The example can be from a real event or something you make up yourself, but it must be a clear illustration of one of the rights in the Bill of Rights. Tell the story of your example. Then explain which right your story exemplifies, from which amendment, and tell why.
Criteria for feedback or rubric:
• Appropriate identification of a particular right and amendment.
• Appropriateness of example.
• Appropriateness of evidence.
• Soundness of reasoning and clarity of explanation.

“How to Assess Higher Order
Thinking Skills”
(example from book)
Questions assessing critical thinking involving judgment:
12. The poster shown above was made during the First World War.
What was the poster designed to do?
A. Make people feel that it would be easy to win the war.
B. Make people feel guilty for thinking that war is harmful.
C.
Get people to join the army by making them feel responsible for starting the war.
D. Get people to join the army by appealing to patriotic feelings.

“How to Assess Higher Order
Thinking Skills”
(example from book)
Questions assessing critical thinking involving judgment:
12. The poster shown above was made during the First World War.
What was the poster designed to do? Explain how you came to this conclusion.
Criteria for feedback or rubric:
• Clear, appropriate statement of the main point.
• Appropriateness of evidence.
• Soundness of reasoning and clarity of explanation.

Program for
International Student Assessment http://nces.ed.gov/surveys/pisa/items.asp?sub=yes

http://www.actstudent.org/sampletest/index.html

•
•
•

How do we measure the football program?
How do we measure the music/dance program?

Competitions/Games
Exhibits
Bulletin Boards
Websites

Portfolio – of successes/best work.
Knowing that the “failures” along the way were lessons learned on the way to success.

• Julia Roberts NAGC’s WOW series
• Susan K. Johnson NAGC’s WOW series
• Brookhart, Susan M. (2010) Assess Higher-Order
Thinking Skills
• Dweck, Carol (2010) Even Geniuses Work Hard,
Educational Leadership, September 2010, Vol. 68 No.
1 www.ascd.org
• Tomlinson, Carol Ann, & Doubet, Kristina (2006)
SMART in the Middle Grades, Westerville, OH,
National Middle School Association
• Wormeli, Rick (2006) Fair Isn’t Always Equal; Assessing
& Grading in the Differentiated Classroom, p. 14,
Stenhouse Publishers, Portland, Maine.

New Online IEP Form: http://wvde.state.wv.us/osp/forms.html
Policy 2419 at http://wvde.state.wv.us/policies.html
Resources for teachers at http://wvde.state.wv.us/osp/gifted.html
Dweck, Carol (2010) Even Geniuses Work Hard, Educational Leadership September
2010, Vol. 68 No. 1 www.ascd.org
Pink, Daniel (2006) A Whole New Mind, Riverhead Books Published by the Penguin
Group. New York, NY.
Reis, Sally & Renzullli, Joseph Curriculum Compacting: A Systematic Procedure for
Modifying the Curriculum for Above Average Ability Students The National Research
Center on the Gifted and Talented - University of Connecticut .
Tomlinson, Carol Ann, & Doubet, Kristina (2006) SMART in the Middle Grades,
Westerville, OH, National Middle School Association
Van-Tassel-Baska, Joyce (2003) Content-Based Curriculum for High-Ability Learners,
Waco, TX, Prufrock Press, Inc.