On Scoring Guides
everything you were afraid to ask
PART TWO
Critical thinking as
“distributed compentency”?
Generalized thinking (e.g., Bloom)
explained to others in the institution?
Generalized thinking explained to laypersons
outside the institution?
Integrative thinking?
Thinking in the major or discipline?
Thinking in a specific community?
Some combination?
Other
St. Olaf E-Portfolio Thinking
Reflective Thinking
Integrative Thinking
Thinking in Context
Thinking in Community
Specific and Flexible
WPA Outcomes Statement for
First-Year Composition
•Rhetorical Knowledge
•Critical Thinking, Reading, and
Writing
•Processes
•Knowledge of Conventions
Rhetorical Knowledge
By the end of first year composition, students should
--Focus on a purpose
--Respond to the needs of different audiences
--Respond appropriately to different kinds of
rhetorical situations
--Use conventions of format and structure
appropriate to the rhetorical situation
--Adopt appropriate voice, tone, and level of
formality
--Understand how genres shape reading and
writing
--Write in several genres
Rhetorical Knowledge
for ASU Composition
Our writing courses will focus on helping students develop and use
a rhetorical framework to analyze writing situations, in a number of
ways. Students will learn how to
--use heuristics to analyze places, histories, and cultures
--be aware of the components of argument and create their own
arguments in conversation with other members of their
discourse communities
--synthesize and analyze multiple points of view
--use a variety of argumentative strategies to write for a variety
of audiences
--express a working knowledge of key rhetorical features, such
as audience, situation, and the use of appropriate
argument strategies
--adopt appropriate voice, tone, and level of formality
--use conventions of format, structure, and language
appropriate to the purpose of the written texts
--be able to focus on a specific rhetorical purpose
So . . .
PLAN
COLLECT DATA
USE RESULTS (aka, feedback loop)
UNC Charlotte English Education
*Portfolio
*Interview
to assess readiness to student teach
*What is your favorite reading?
*What African American literature would
you like to teach? What Native American
literature would you like to teach?
Issues . . .
*why were students preferring shorter texts?
*why could they not identify more than a
single text in a given field? Was this a
problem?
*how did they perform on national tests?
*what feedback did we get from schools?
CHANGES
•
•
•
•
Added a course in Native American Lit
Added a course in African American Lit
Required a course in Ethnic Literature
Encouraged all faculty to use “real” texts
RESULTS?
new issues!!
PORTLAND STATE CRITICAL THINKING:
program design & METHOD
~~interdisciplinary, team-taught first year seminar~~
A random sampling of these portfolios that was
stratified for each class formed the analysis set.
Portfolios were selected using a random number
generator and a numbered list of students from each
Freshman Inquiry class. Five student names and one
alternate (and instructions for choosing a second
alternate, if necessary) were given to each instructor.
Alternate names were used if one or more of the
original five students chosen appeared on the class
list, but did not complete the course.
PORTLAND STATE AND CRITICAL THINKING
The scoring guides (rubrics) used in the
Summer Portfolio Review were internally
developed. A previous attempt to use an
externally developed rubric for critical thinking
was not successful because the rubric was
not contextually relevant to the PSU
student work.
The new rubric for critical thinking was not
completely developed by the time of
this summer’s review, and the review itself
stood as a development process for this
rubric.
WHAT HAPPENED?
Following the June 2000 review of Freshman
Inquiry portfolios, each Freshman Inquiry
team was required to review information from
the portfolio review and, if available, from
the end-of- year course evaluations. Each team
reported to the Freshman Inquiry Coordinator,
specific, planned course revisions based on the
asessment information.
GET SPECIFIC! WHAT DOES THIS MEAN?
In the summer 2000 meeting, the
“Metamorphosis” team decided to focus on the
critical thinking goal. To this end, they instituted a
major re-working of the fall term course,
revising the texts and focusing more clearly on
central concepts. They also added more
science content in the first and second quarter
courses. This team raised questions about
the critical thinking rubric, which is scheduled for
revision this year.
Did you say technology?
What some of this means for guides . . .
• Scoring Guides Can Show Development, Achievement, or
Both
--What is the guide intended to document, and why?
• Text and Context: Scoring Guides Don’t Act Alone.
--How will you introduce them to students? And for what
purpose/s?
• Scoring Guides Signal the Philosophy of the Institution.
--Is the model of learning showcased in the guide a deficit model
of learning (think error avoidance or removal) or an asset
inventory model (think building on strengths)?
**Scoring Guides Inherently Enact a Visual Rhetoric
--Most scoring guides list a set of elements organized into
an analytical framework or a holistic framework, suggesting that each
item is weighted equally. When the guide is put into an institutional
context and/or is used to rate student work, often some items are
more valuable than others. Which items are the more valuable, and
why, and how can that be signaled within the guide itself?
**Scoring Guides, as Genre, Change
--The scoring guide we would have created for writing twenty
years ago would exclude much of what we do in writing today; critical
thinking is likewise defined more ambitiously today than in earlier
times. The scoring guide we create today should do the best job it can
do today. It will need to be changed tomorrow.
How to begin?
*Gather some student work.
(What counts as work?)
*Gather some model scoring guides.
*Get some food.
**Talk!
**Identify what you like—that’s
vocabulary. Consume? Produce?
Achievement? Development?
**Identify a structure.
**Give it a go, and call it a pilot.
You’re never
really ready . . .
It’s Better to Start Small,
but to Start . . .
Call it what you will—a pilot, a field test—start! Start
small. Get a taste of success, and build on that.