Sarah Murnen, Kenyon College
Barbara Andereck, Ohio Wesleyan University
(Nancy Grace, College of Wooster)
FACULTY USE OF PRIMARY TRAIT
ANALYSIS TO ASSESS CREATIVE
THINKING
FUNDED BY THE TEAGLE FOUNDATION
Grant to the “Ohio-5” Colleges administered
by the College of Wooster 2006-2009
 “Creativity and Critical Thinking: Assessing
the Foundations of a Liberal Arts Education”
 Organized by Lori Bettison-Varga, former
Associate Provost at the College of Wooster,
and then Nancy Grace, Professor of English
at the College of Wooster
 http://www.wooster.edu/teagle

How Do You Define
Creativity?
STARTING OUR PROJECT

Year 1:
Assembled groups of faculty across four colleges in
the Ohio-5 consortium of colleges to talk about
evaluating creative and critical thinking among
students
 Developed survey for students and faculty, asked
about traits defining critical and creative thinking

Creative Thinking
Critical Thinking
Openness to Novelty
Idea Generation
Curiosity
Elaboration
Synthesis,
integration &
combination
Invention
Imagination
Complexity
Reasoning by
metaphor & analogy
Divergent
thinking
Playfulness
Abstraction &
simplification
Awareness of
Environment
Evaluation &
assessment
Categorization &
classification
Reasoning through
logic
Identification
Interpretation
Analysis
Description
How Can We Best Assess
Creative Thinking in
Students?
PRIMARY TRAIT ANALYSIS
•
•
•
Identify the “primary traits” – essential or central
components of the discipline (both knowledge and
skills) to be learned by the student
Build a scale for scoring the student’s performance on
the trait
Evaluate the student’s performance against those
criteria
PRIMARY TRAIT ANALYSIS
Resources:
Presentation by Dr. Douglas Eder on “Primary Trait
Analysis”
Presentation by Barbara Walvoord on the use of rubrics for
effective grading
Walvoord, B. E., P& Anderson, J. A. (1998). Effective
Grading: A tool for learning and assessment. Jossey Bass:
San Francisco, CA.
STAGES IN CONSTRUCTING A RUBRIC
From: Stevens & Levi (2005). Introduction
to rubrics: An assessment tool to save
grading time, convey effective feedback,
and promote student learning. Stylus
Publishing: Sterling, Virginia.
Reflecting –
•
•
•
•
What do we want for our students and from
our students? (Learning objectives)
Why did we create the assignment?
What happened the last time we used the
assignment?
What are our expectations for the
assignment?
STAGES IN CONSTRUCTING A RUBRIC
Listing – Focus on particular details of the
assignment and what specific learning
objectives we hope to see completed (Sometimes
helps to imagine the best and the worst
performance on the assignment)
Grouping and Labeling the goals together Organize the results of our reflections in first
two stages, grouping similar expectations
together in what will probably become the
dimensions of the rubric
Application – Transfer groupings to a rubric
grid; define assessment categories
GROUP DISCUSSION RUBRIC
HTTP://WWW.MASHELL.COM/~PARR5/TECHNO/GROUP.HTML
Measures Emerging (1), Competent (2), and Exemplary (3) Performance on Each Trait
LISTENING
1. Recognizes and responds to others speaking.
2. Uses and practices listening processes regularly.
3. Habitually uses listening processes.
NON-VERBAL COMMUNICATION Eye contact, gestures, posture, facial expression, voice.
1. Comprehends some information from non-verbal cues.
2. Draws accurate conclusions from body language and facial expressions.
3. Able to recognize and use subtle non-verbal communication cues.
CO-OPERATION
1. Sometimes shows ability to wait to give appropriate verbal / non-verbal responses.
2. Usually shows ability to wait to give appropriate verbal / non-verbal responses.
3. Habitually shows ability to wait with openness and awareness to give appropriate verbal / non-verbal
responses.
PARTICIPATION Tells thoughts, feelings, ideas so others understand.
1. Rarely talks during the discussion or talk is off the subject. Offers few ideas to the discussion.
2. Shares freely and explains with details. Makes connections to what others say.
3. Talk inspires others. Supports and leads others in discussion.
DEVELOPING RUBRICS THROUGH PRIMARY
TRAIT ANALYSIS
 Year

Faculty developed rubrics to measure creative
and critical thinking for their courses or
departments
 Year


1:
2:
Taught other faculty to develop and use
rubrics, and examined the rubrics they created
along with student performance data from the
rubrics
Developed a “generic” rubric using PTA
designed to measure creative and critical
thinking
“GENERIC RUBRIC”
CRITICAL AND CREATIVE THINKING TRAITS
1. Elements of Argumentation
a.
b.
c.
d.
e.
Explanation
Analysis
Evaluation
Interpretation
Logic
2. Domain and Disciplinary Knowledge
3. Synthesis and Connections
4. Abstract Thinking
5. Complexity of Thought
6. Ideas Generated
a. Fluency
b. Flexibility
“GENERIC RUBRIC”
CRITICAL AND CREATIVE THINKING TRAITS
7. Completeness/Coherence
8. Elegance
9. Divergent Thinking
10. Novelty or Uniqueness
a. Germinal
b. Original
c. Transformational
11. Engagement
12. Risk Taking
“GENERIC RUBRIC”
CRITICAL AND CREATIVE THINKING
RATING SCALE DESCRIPTIONS
6 = Trait is evident to an exceptionally high
degree
5 = Trait is evident to a high degree
4 = Trait is evident to an intermediate degree
3 = Trait is somewhat evident
2 = Trait is barely evident
1 = Trait is not evident
What Assignments/
Pedagogies Promote
Creative Thinking?
FACULTY USE GENERIC RUBRIC
 Year

3:
Faculty across many disciplines measured
student performance using the generic rubric
in 31 classes (N = 444 students) across three of
the participating colleges (College of Wooster,
COW; Kenyon College, KC; and Ohio Wesleyan
University, OWU)
YEAR 3: KEY FINDINGS
Creative and critical thinking can be assessed in
students, and we found improvement of
performance on many creative and critical
thinking traits among the students we studied (N
= 444 students across 31 courses).
 The traits measured most frequently across all
disciplines were complete and original measured
in 90% of students, followed by knowledge
measured in 85% of students. The traits least
likely to be measured included abstract, logic,
and novel. There were some differences across
disciplines in the traits that were emphasized; for
example, faculty teaching in interdisciplinary
courses were more likely to examine complexity,
and they shared with fine arts faculty a greater
concern for engagement and risk.

YEAR 3: KEY FINDINGS



In the longitudinal data measuring change in
student performance across a semester, the traits
where the most change was evident included
complete, elegant, knowledge, and engaged; and the
most change occurred among students in
interdisciplinary courses.
In the cross-sectional data comparing less expert
students to more expert, the most reliable
differences were on the traits of analyze, logic,
knowledge, and explain. There was no significant
difference between the groups’ performance on
abstract, fluent, flexible, elegant, divergent, novel,
and risk.
Participating faculty who filled out an on-line
survey indicated that they had discussed the use of
the rubric with colleagues, and that the rubric
helped them learn more about creative and critical
thinking among their students
TRAITS MEASURED IN CLASSES BY
DISCIPLINE – ALL DATA
ALL
FA
HUM
NSCI
SOSC
INT
Explain
0.63
0.64
0.89
0.87
1.00
0.12
Analyze
0.54
0.44
0.89
0.87
0.72
0.12
Evaluate 0.44
0.00
0.54
0.73
1.00
0.12
Integrate 0.43
0.75
0.74
0.56
0.28
0.12
Logic
0.40
0.44
0.12
0.71
0.72
0.12
Knowledge 0.85
1.00
0.60
0.84
0.72
0.99
Synthesis 0.53
0.79
0.48
0.58
1.00
0.12
Abstract
0.38
0.00
0.12
0.71
1.00
0.12
Complex
0.62
0.00
0.63
0.58
0.55
1.00
Fluent
0.63
0.45
0.74
0.58
0.72
0.65
Flexible
0.45
0.21
0.89
0.28
0.45
0.47
Complete 0.90
0.89
0.63
0.86
1.00
1.00
Elegant
0.58
0.89
0.12
0.28
0.56
0.81
Divergent 0.52
0.21
0.48
0.14
0.72
0.81
Novel
0.40
0.25
0.74
0.69
0.45
0.12
Original
0.90
0.79
0.89
0.73
1.00
1.00
Transform 0.74
0.68
0.88
0.56
0.72
0.81
Engaged
0.72
0.89
0.12
0.56
0.72
1.00
Risk
0.59
0.75
0.48
0.14
0.28
1.00
_____________________________________________________________________
N Students 444
73
65
85
82
139
N Classes
31
5
6
6
8
6
_____________________________________________________________________
AVERAGE PERFORMANCE ON TRAITS
ACROSS TIME – LONGITUDINAL DATA
Trait
M time 1 M time 2 tvalue
Explain
3.85
4.52
7.26***
Analyze
3.76
4.73
7.84***
*** indicates t value is significant at p<.001
Evaluate
3.28
4.27
8.86***
revealing change in mean across time
Integrate
3.67
4.51
5.86***
Logic
3.76
4.78
6.85***
Note: High responses are noted in bold,
Knowledge
3.40
4.35
10.63***
low in italics.
Synthesis
3.77
4.63
8.50***
Abstract
3.53
4.58
9.76***
Complex
3.27
4.14
9.60***
Fluent
3.56
4.39
9.16***
Flexible
2.85
3.82
8.56***
Complete
3.56
4.41
10.98***
Elegant
3.03
4.05
10.83***
Divergent
2.53
3.61
9.00***
Novel
3.19
4.22
5.78***
Original
3.19
4.01
9.40***
Transform
2.23
3.31
9.22***
Engaged
3.56
4.48
10.58***
Risk
2.80
3.55
7.60***
_____________________________________________________________
CROSS-SECTIONAL DATA
Scores on the traits could range from 1 to 6, with a higher score indicating better performance.
Differences between groups were analyzed with independent sample t-tests.
Trait
Explain
Analyze
Evaluate
Integrate
Logic
Knowledge
Synthesis
Abstract
Complex
Fluent
Flexible
Complete
Elegant
Divergent
Novel
Original
Transformation
Engaged
Risk
M Less Exp M More Exp
3.99
4.71
3.64
4.65
4.18
4.70
3.47
4.19
3.87
4.80
3.37
4.01
3.67
4.61
4.26
4.56
3.15
3.71
4.45
3.93
3.92
3.59
3.60
4.16
3.04
3.23
2.87
3.21
3.38
3.67
2.91
3.58
2.30
3.02
3.81
4.29
2.34
2.53
Difference?
4.35***
5.86***
2.51*
3.72***
5.28***
4.39***
4.32***
ns
3.04**
ns
ns
3.52**
ns
ns
ns
3.30***
3.34***
2.59**
ns
Note:
ns indicates the t value is not
significant;
indicates significant at p<.05,
**p<.01,
***p<.001
YEAR 3 INTERVIEWS PARTICIPATING
FACULTY
Twenty-two faculty who employed the “generic”
rubric to measure creative and critical thinking
among the students were interviewed about their
experience
 Interviewers asked questions about

the specific assignment used and whether it
promoted creative thinking,
 other assignments and pedagogies that might
promote (or inhibit) creative thinking,
 aspects of the discipline that might promote (or
inhibit) creative thinking,
 aspects of the college environment that might be
important

YEAR 3 INTERVIEWS PARTICIPATING
FACULTY
Most of the faculty who participated chose an
assignment they had used previously, and did not
develop a new assignment for the grant.
 Many of these assignments seemed designed to
stimulate critical thinking more than creative
thinking, but using a rubric that emphasized
creative thinking increased the focus on these
traits and likely increased the amount of
creativity students showed.
 In general, most people thought their
assignments did promote creative thinking in
students (and this was validated by the
quantitative data gathered from the rubrics).
However, the assignments people used for their
data collection were not necessarily the most
creative assignments the faculty had used or
could imagine using.

FACULTY INTERVIEWS YEAR 3:
VALUE OF “ACTIVE LEARNING TECHNIQUES”

Take devil’s advocate position

Use problem solving

Students debate ideas

Use small group discussion

“Hands-on” science activities
FACULTY INTERVIEWS YEAR 3:
VALUE OF “ACTIVE LEARNING TECHNIQUES”
One natural scientist spoke about student experience
with creating their own experiments versus using
“canned” labs. She said, “I didn’t know when I
started doing this in my upper-level comparative class
whether they would like it.... The experiments don’t
usually work, because experiments usually don’t work,
period. And, so, I would always say next year I am
going to write labs. By the end of the class students
come to my office to tell me emphatically, they write it
on their evaluations, do not change the lab. This is
what we want to do. This is what scientists do. I am
always surprised by that… they always like it that
way. Even if they have to put in more time. Evenings,
weekends to feed their animals, take measurements….
the students like it better, it is better for them…
thinking of something that has not been done but is
doable is hard and you do have to be creative to do it.”
FACULTY INTERVIEWS YEAR 3: VALUE
OF “TAKING THE PERSPECTIVE OF OTHERS”

Similarly, another historian said, “I ask them a
question that is surprising. It might be something
like, ‘What does it smell like in 14th century Florence?’
A question we have not talked about at all, but they
have done a great deal of reading from primary
sources about markets in Florence, or about
demography, or about domestic spaces and ask them
to think about that evidence in some new sort of way
and to extrapolate about the lived realities of 14th
century people about things they know from records.”
FACULTY INTERVIEWS YEAR 3: VALUE
OF “TAKING THE PERSPECTIVE OF OTHERS”

A couple of other professors also mentioned types of
“role playing.” One humanities professor said, “In my
journalism class they do mock press conferences. They
are given a topic – political topic - that is not a
scandal. They have to draw on real-world knowledge,
using realistic people who would appear at a press
conference. Have to generate the story, present the
information, and the rest of the class asks questions as
reporters and they write the news story as journalists.
Then the student presenting has to field the questions.
They think outside, but within the box as well. They
have to be active. I go back to the definition of
[creativity as] ‘new but useful.’ “
FACULTY INTERVIEWS YEAR 3:
VALUE OF “STUDENT CHOICE”

One professor of an interdisciplinary course talked
about the creation of a student-led senior colloquium.
She said, “...second semester senior year giving seniors
a term paper is just a death sentence. So I designed
the senior colloquium as a way to get them to be more
creative and to flow a little bit more with the material,
rather than a research paper. The students design
some of the colloquium itself, they get to pick the topic,
they get to pick the readings and they get to determine
the project that they will do collectively…. So for
instance this year they did a magazine and I think
they liked the opportunity to explore the genre of this
sort of women’s magazine at the same time trying to
make some sorts of statements about women and
feminist issues that are important to them.”
FACULTY INTERVIEWS YEAR 3: VALUE
OF “PEER EVALUATION, INFLUENCE”

Many faculty talked about the importance of students
engaging with their work, of “owning” their work.
Faculty thought that to the extent that their peers
would see their work, students would care more about
it. The professor who had her students use “Moodle”
to discuss their ideas publicly found that there was
healthy competition among the students to present
their unique voice which led to very creative
expression.
FACULTY INTERVIEWS YEAR 3: VALUE
OF STUDENTS MAKING CONNECTIONS
OUTSIDE THE CLASSROOM

A social science professor who has students do work outside
the classroom said, “If they are out in the community, if they
are working with a group of people, I think it gives them a way
to think about an issue in a much bigger way… I mean you
can look at the school violence issue, you can look at a bunch of
material about school violence and have good book knowledge
of it and the way that boys and girls interact in high school,
but when you go to a football game and you watch it happen,
you see these little kids pushing each other around and you see
the bullies… it makes them come back with different questions,
different ways to think about what they read.”
FACULTY INTEVIEWS, YEAR 3:
VALUE OF “INTERDISCIPLINARITY”

At Ohio Wesleyan the “National Colloquium” is a way
to promote interdisciplinary thinking and discussion
of issues outside of the classroom. One of the
professors brought this up and said,

“There’s a different topic every year, a major national or
international topic, and they bring in speakers and we try
to bring tie in other events on campus and stage a play that
focuses on that topic. Students will receive partial credit for
attending the lectures or the performances or participating
in activities which have something to do with the topic of
the National Colloquium and one year it was about the
arts. One of my colleagues raised the topic of having the
National Colloquium for the arts and students could get
other class credits for attending art openings or jazz band
performance…. Making it part of the curriculum kind of
forces the students to take advantage of things that are
available to them… but that kind of thing is what I think
educational institutions could do.”
Where does Creativity
Happen on Campus?
SUMMARY YEAR 2 SURVEY DATA
Surveys were administered to 260 first-year
students, 375 senior students, and 147 faculty
across the four colleges during the 2007-2008
academic year to assess perceptions and
experiences. Out of a wealth of information
yielded by this research, a few interesting items
are described below.
1. There seem to be more opportunities for critical than
creative thinking in classes.
2. Class projects are evaluated by students as a means to
engage in both critical and creative thinking.
3. Our campus environments were rated favorably with
respect to the potential for creative thinking, although
faculty rated “idea time” and “risk taking” as somewhat
limited in their experience.
SUMMARY YEAR 2 SURVEY DATA
4. Women students reported more experience with both
creative and critical thinking and more positive attitudes
about creativity compared to men students.
5. When asked to write about their most creative experiences,
61% of faculty mentioned their own research. Teaching
and collaboration with students were each mentioned by
32% of the faculty.
6. When asked about barriers to creative thinking on campus,
the most common response of senior students was that
class pedagogy was a barrier (40% of students mentioned
this); while faculty most often wrote about time as a barrier
(18% of faculty).
7. A diverse campus environment (in terms of class, race, sex,
sexual orientation, and national origin) is perceived to
facilitate creative thinking.
YEAR 2 SURVEY DATA: FACULTY AND
STUDENT PERCEPTIONS OF CREATIVE
ENVIRONMENT CHARACTERISTICS
Participants rated the extent to which each characteristic existed on their campus: 1 = not at all, 7 =
extensively
Faculty
5.02
5.08
4.69
4.45
4.65
4.50
4.72
3.83
3.73
4.14
4.67
4.86
4.58
4.88
1st Yr
5.50
5.50
5.30
5.28
5.11
5.28
5.23
4.79
4.80
4.28
5.40
5.00
5.14
5.22
Seniors
5.31*
4.94*
4.90*
4.88*
4.89*
5.05*
5.32*
4.36*
4.45*
4.56*
4.99*
5.04
4.98*
5.10
Item
Challenge
Freedom
*indicates significant difference
Idea Support
between groups
Trust/Openness
Dynamism/Liveliness
Playfulness/Humor
Debate
Risk Taking
Idea Time
Conflict
Supportive Environment
Working in groups
Active models of creative thinking
Assignments encourage independent
problem- solving & risk-taking
YEAR 2 SURVEY: WHERE DOES CREATIVE
THINKING HAPPEN ON CAMPUS, ACCORDING
TO STUDENTS?
Students rated how much each event involved creative thinking,
rating 1 = low to 5 = high
CREATIVE THINKING
First Year Seniors
Engaging in a hobby
Completing projects for class
Writing papers for classes
Extra-curricular or co-curricular activity
Interacting w/ students outside of class
Cultural events on campus
Interacting w/ faculty outside class
Interacting w/ students, faculty in class
Listening to speakers on campus
Communicating w/ peers internet
Reading material for class
“Surfing” the internet
4.18
3.74
3.74
3.45
3.51
3.19
3.02
3.02
2.78
3.03
2.83
2.63
4.13
3.78
3.68
3.55
3.52
3.18
3.14
3.04
2.74
2.73*
2.61*
2.59
*indicates significant
difference between groups
STUDENT IDEAS ON THE ENCOURAGEMENT OF
CREATIVE THINKING, FOCUS GROUPS YEAR 2
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Professors allow students to give input (open-minded professors);
discussion-based classes
Bringing in outside artists, bands, speakers, poets to “stir the pot”
Open-ended assignments in classes
Opportunities to connect work in different courses
Classes with real-world applications
Need more time
Student-themed housing
Classes where professors challenge students “to the edge”
Comfortable spaces where students can come in and “do their own thing”
Lunch tables with faculty
Diversification and diversity requirements encourage students to learn
something new
Classes that require internships, service learning (first-year seminars
with these components were praised)
Study abroad
Senior research project (projects in general);
Fewer required courses would mean more freedom for students; Any way
for students to propose their own classes?
FACULTY IDEAS ON THE ENCOURAGEMENT OF
CREATIVE THINKING, FOCUS GROUPS YEAR 2
• General Comments.
•
•
•
•
•
•
•
•
•
•
•
•
Spaces for faculty to get together and discuss pedagogy
Fifty minute time periods are not long enough
Student-controlled spaces, events, organizations
Need to find time to live “the life of the mind”
Service learning that involves many members of the
community
Senior seminars; Independent research project, summer
science programs, senior projects;
Teaching abroad with students from your institution
Encourage diversity, appreciation for internationalism
Taking students to conferences, on field trips, etc.
Provide faculty with opportunities to do things
differently – team-teach, time to do projects
Make interdisciplinary work easier, e.g., break down
departmental autonomy
Allow faculty to remain “learners” as much as possible
FACULTY IDEAS ON THE ENCOURAGEMENT OF
CREATIVE THINKING, FOCUS GROUPS YEAR 2
•
In the classroom.
•
•
•
•
•
•
Guest speakers
Use of Moodle and other forms of technology (that
encourage engagement)
A final project with several options to the traditional
paper (assignments encouraging active engagement)
Provide opportunities for students to interact with the
“real world”
Setting up projects, assignments where outcome is
unclear
Assignments that require “vulnerability” – students
have to show something about themselves
FACULTY INTERVIEWS, YEAR 3:
VALUE OF ASSESSMENT
“Doing the rubrics – I didn’t know how that would
work, I had my doubts two years ago. I actually
started using a rubric and I find it easier to grade
consistently …. I really like it and use it now for
all my labs, it helps my grading. If students
would look at them they could realize exactly
what they aren’t doing well. I give this back to
students with written comments, and the ones
who look at it could improve their labs which is
helpful…. I found it helpful and I was totally
shocked. Someone said you can’t do it with lab
reports – I think you can – I showed that you
could.”
FACULTY INTERVIEW YEAR 3:
QUOTE
“The challenge is to encourage faculty members to be
experimental in the ways they imagine student work,
without abandoning key commitments to core
educational goals having to do with having students
express themselves clearly in argument to also say that
the ways in which you might go about cultivating a
sense of the discipline should go beyond, at least
experiment with going beyond, writing 5 or 25 page
papers. How you encourage faculty members to do
that I am not sure. I suppose one way of doing that is
giving them inspirational models. Make sure people
within their departments and across disciplines talk
to each other about whatever works, whatever seems to
gain traction in this area.”
FACULTY INTERVIEWS YEAR 3:
Changing
Classroom
“Expectations of what a classroom is supposed to
be like are changing. Using the classroom as a
place to do presentations, as a place to do small
group work, as a place to do debates… use the
classroom as a theatre some days, making a place
where we do lecture some days… making it a
place where when they are getting ready to go to
my class they are wondering what are we going to
do today? Mixing it up, building some excitement.
It can ‘t be like that everyday, but…. “
AAC&U VALUE AND OHIO 5 TEAGLE
VALUE
TEAGLE
15 general rubrics
1 general rubric
Developmental scale
Evaluative scale
Wide range of institutions
Liberal arts colleges
Adaptable
Adaptable
Useful for assessment
Useful for assessment
www.aacu.org/value/rubrics
www.wooster.edu/teagle