Developing Assessment
Methods at Classroom,
Unit, and Campus-Wide Levels
Presented at
The Robert Gordon University
Aberdeen, Scotland
March 19, 2004
Presented by
Trudy W. Banta
Vice Chancellor
Planning and Institutional Improvement
Indiana University-Purdue University Indianapolis
355 N. Lansing St., AO 140
Indianapolis, Indiana 46202-2896
tbanta@ iupui.edu
http://www.planning.iupui.edu
Outcomes Assessment
The process of providing credible
evidence of the outcomes of
higher education undertaken for
the purpose of improving
programs and services within the
institution.
Banta, T. W.
Plan
Implement
Improve
Assess
ASSESSMENT . . .
“a rich conversation
about student learning
informed by data.”
-- Ted Marchese -AAHE
Assessment of Individual
Student Development
Assessment of basic skills for use
in advising
• Placement
• Counseling
Periodic review of performance
with detailed feedback
End-of-program certification of
competence
• Licensing exams
• External examiners
Key Results of Individual
Assessment
 Faculty
can assign grades
 Students
learn their own
strengths and weaknesses
 Students
become selfassessors
A Second Look
Across
students
Across
sections
Across
courses
 Where
 What
is learning satisfactory?
needs to be retaught?
 Which
approaches produce the
most learning for which
students?
Group Assessment
Activities
•
•
•
•
•
•
•
Classroom assignments, tests,
projects
Questionnaires for students,
graduates, employers
Interviews, focus groups
Program completion and placement
Awards/recognition for graduates
Monitoring of success in graduate
school
Monitoring of success on the job
Use of Results of Group
Assessment
•
Program improvement
•
Institutional and / or state
peer review
•
Regional and / or national
accreditation
Some Purposes of
Assessment
1.
2.
3.
4.
5.
Students learn content
Students assess own strengths
Faculty improve instruction
Institutions improve programs/services
Institutions demonstrate accountability
Outcomes Assessment
Requires Collaboration





In setting expected program
outcomes
In developing sequence of learning
experiences (curriculum)
In choosing measures
In interpreting assessment findings
In making responsive improvements
Barriers to Collaboration
in the Academy
1.
2.
3.
4.
Graduate schools prepare
specialists
Departments hire specialists
Much of our scholarship is
conducted alone
Promotion and tenure favor
individual achievements -interdisciplinary work is harder to
evaluate
Campus Interest in
Assessment
WHAT WORKS in….




increasing student retention?
general education?
use of technology in instruction?
curriculum in the major?
Good assessment is
good research . . .





An important question
An approach to answer the
question
Data collection
Analysis
Report
-Gary R. Pike (2000)
Most Faculty Are Not Trained as
Teachers
FACULTY DEVELOPMENT
Can Help Instructors:

Write clear objectives for student learning in
courses and curricula
 Individualize instruction using a variety of
methods and materials
 Ask questions that make students active
learners
 Develop assessment tools that test higher
order intellectual skills
Organizing for Assessment
Goal
Course Measure Findings Uses
Write
Portfolio
Speak
Speech
Think
Test
Find
Information
Project
Taxonomy of Educational
Objectives
(Bloom and Others, 1956)
Cognitive domain
categories
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Sample verbs for outcomes
Identifies, defines, describes
Explains, summarizes, classifies
Demonstrates, computes, solves
Differentiates, diagrams, estimates
Creates, formulates, revises
Criticizes, compares, concludes
Learning Outcomes for English
Majors
Graduates can:
1
2
3
4
5
Demonstrate how language influences
intellectual and emotional responses
Apply knowledge of rhetorical context by
writing effectively and appropriately in
context
Assess critically spoken, written, and
visual representations
Apply research strategies appropriate to
area of study
Synthesize diverse issues and responses
raised in collaborative discussions of texts
Learning Outcomes in
Science
1.
2.
3.
4.
5.
6.
Define and explain basic principles, concepts,
theories of science
Identify characteristics that distinguish math
and science from each other and from other
ways of obtaining knowledge
Illustrate how developments in science can
raise ethical issues
Solve theoretical or experimental problems in
science
Evaluate the validity and limitations of
theories and scientific claims in interpreting
experimental results
Evaluate scientific arguments at a level
encountered by informed citizens
In each course
Statement
Of
Competence
Critical
Thinking
Teaching/
Learning
Strategy
Group
Project
Assessment
Methods
Written
Paper
Planning for Learning and
Assessment
1. What
2. How
3. How will
general
would you
you help
outcome
know it
students
are you
(the
learn it?
seeking?
outcome)
(in class
if you saw
or out of
it? (What
class)
will the
student
know or
be able to
do?)
4. How could
5. What are 6. What
you
the
improvemeasure
assessments
each of the
ment
might be
desired
findings?
based on
behaviors
assesslisted in #2?
ment
findings?
Organizational Levels for Assessment
National
Regional
State
Campus
College
Discipline
Classroom
Student
Direct Measures of Learning
Assignments, exams, projects, papers
Indirect Measures
Questionnaires, inventories, interviews
- Did the course cover these objectives?
- How much did your knowledge increase?
- Did the teaching method(s) help you
learn?
- Did the assignments help you learn?
GOOD ASSESSMENT INCLUDES BOTH
Fast Feedback
(at end of every class)




Most important thing learned
Muddiest point
Helpfulness of advance reading
assignments for day’s work in class
Suggestions for improving class /
assignments
Bateman & Roberts
Graduate School of Business
University of Chicago
Student Suggestions for
Improvement





Install a portable microphone
Increase type size on
transparencies
Leave lights on when using
projector
Don’t cover assigned reading in
detail
Provide more examples in class
Wingspread Seven Principles for
Good Practice in Undergraduate
Education
(Chickering & Gamson, 1987)
1.
2.
3.
4.
5.
6.
7.
Student-faculty contact
Cooperation among students
Active learning
Prompt feedback
Time on task
High expectations
Respect for diverse talents and ways of learning
Student Learning Oriented
Course Evaluation
1.
2.
3.
4.
Learners held high expectations for one
another
Learners interacted frequently with
others
Learners participated in learning teams
Learners respected diverse talents and
ways of learning
-Cournoyer
Advances in Social Work – Fall 2001
Primary Trait Scoring
Assigns scores to attributes (traits) of a task
STEPS

Identify traits necessary for success in
assignment
 Compose scale or rubric giving clear
definition to each point
 Grade using the rubric
Can Develop a Research
Paper
Outstanding
Accept- Unaccept
able
-able
1.
Narrows and defines
topic
2.
Produces bibliography
3.
Develops outline
4.
Produces first draft


5.
Produces final draft


6.
Presents oral defense








Bibliography
Outstanding – References current,
appropriately cited, representative,
relevant
Acceptable – References mostly current,
few citation errors, coverage adequate,
mostly relevant
Unacceptable – No references or
containing many errors in citation
format, inadequate coverage or
irrelevant
Mapping Course Outcomes
to Program Outcomes
Outcomes
Course 1
Course 2
Course 3
1





2
3



4
5

6

7


Sophomore Competence in Mathematics
(Multiple choice responses & supporting
work)
Score
3
2
1
0
Criterion
Clear conceptual understanding, consistent
notation, logical formulation, complete
solution
Adequate understanding, careless errors,
some logic missing, incomplete solution
Inadequate understanding, procedural errors,
logical steps missing, poor or no response
Problem not attempted or conceptual
understanding totally lacking
Ball State University
Assessment in Sociology and
Anthropology
Focus groups of graduating students
 Given
a scenario appropriate to the discipline,
a faculty facilitator asks questions related to
outcomes faculty have identified in 3 areas:
concepts, theory, methods.
 2 faculty observers use 0-3 scale to rate each
student on each question
 GROUP scores are discussed by all faculty

Murphy & Goreham
North Dakota State University
Assessment of Group
Interaction
The Student Participant:
 Listened to others
 Actively contributed to discussion
 Challenged others effectively
 Was willing to alter own opinion
 Effectively explained concepts/insights
 Summarized/proposed solutions
5=Consistently excellent
3=Generally satisfactory
1=Inconsistent and/or inappropriate
Capstone Course in Business
Comprehensive case study analysis
Graded using scoring rubric covering
 writing
skills
 knowledge of historic and current theories
and practice
 ability to integrate tools and techniques
from business specialties in analyzing the
case

Olney & Menger
St. Mary’s University
In a Comprehensive
Assessment Program...
INVOLVE
 Students
 Faculty
 Student Affairs
Professionals
 Administrators
 Graduates
 Employers
Student Advisory Council at
Montevallo
A way to provide continuous student
assessment
Student Recommendations
1
2
3
4
5
Develop a statement of expected
ethical behaviors for students
Add a second research course with
lab
Increase comparative psychology
Add terminals for statistics lab
Increase opportunities for research,
writing, and speaking
Collaboration in Evaluation
(Harvard and Samford)

Involve a student team in evaluating
a course or program
 Interaction
increases
 Students develop communication skills
teamwork competence
self-confidence
understanding of others’ perspectives
 Involvement
in learning increases
PRINCIPLES OF
UNDERGRADUATE LEARNING
(PULs)
1.
2.
3.
4.
5.
6.
Core communication and quantitative
skills
Critical thinking
Integration and application of
knowledge
Intellectual depth, breadth, and
adaptiveness
Understanding society and culture
Values and ethics
Approved by IUPUI Faculty Council
May 1998
Student Electronic Portfolio
 Students
take responsibility for
demonstrating core skills
 Unique individual skills and
achievements can be emphasized
 Multi-media opportunities extend
possibilities
 Metacognitive thinking is enhanced
through reflection on contents
- Sharon J. Hamilton
IUPUI
Virginia Commonwealth University
1. First-year English students wrote 1
response / week for 15 weeks
2. Faculty-student affairs teams read
essays
3. Sample of writers interviewed 18
months later
Responses to findings:
Central advising center, new advising
handbook, multicultural workshops
Involving Employers
Combination of survey and focus groups
for employers of business graduates




Identified skills, knowledge, personality attributes
sought by employers
Encouraged faculty to make curriculum changes
Motivated students to develop needed skills
Strengthened ties among faculty, students,
employers
- Kretovics & McCambridge
Colorado State University
Colorado State University
College of Business
Curriculum changes based on employer
suggestions:

1 credit added to Business Communications for team
training and more presentations
 Ethics & social responsibility now discussed in intro
courses
 New Intro to Business course emphasizing career
decision-making
 More teamwork, oral & written communication,
problem-solving in Management survey courses
- Kretovics & McCambridge
Program
Objective
Course
Objective
Measurement
Method
Findings
Action
Needed
Apply CT
skills in SW
practice
Use
statistical
analysis to
inform
practice
decisions
Lab exercises
___________
(80% of
students will
earn 3.0 or
better on lab
exercises
64% of
students
scored 3.0
or better
on lab
exercises
Students
ask for
more lab
time; 5 days
of lab
added
Columbia College
Authentic Assessment
at
Southern Illinois University - Edwardsville

Business - Case Study Analysis with Memo
 Education - Professional Portfolio
 Psychology - Poster on Research Project
 Engineering - Senior Design Project
 Nursing - Plan of Care for Patient
Responses to Assessment
at
Southern Illinois University-Edwardsville
•
•
•
•
•
Business – More case studies and research
Education – More practice in classroom
management
Psychology – Curriculum change in statistics
Engineering – More practice in writing and
speaking
Nursing – Simulation lab with computerized
patients
Assessment in Fine Arts
—
Portfolio review and/or audition
for every student every semester
by
Panel of faculty, students, community
representatives, staff or faculty outside fine
arts
Results - Students creativity,
conceptualization, technique have
improved
Alec Testa
— Eastern New Mexico University
—
Assessment Essentials
1. Plan for assessment
2. Involve all stakeholders
3. Develop goals and objectives
4. Select or design assessment methods
5. Report assessment findings
6. Promote the use of assessment findings
7. Evaluate the assessment program
- Palomba & Banta
Success Factors
Committed leadership
2 Collaboration between faculty and student
affairs leaders
3 Teamwork in planning and implementation
4 Supportive campus climate
Concern for students, continuous
improvement
5 Involvement in design of assessment
6 Results effectively communicated
7 Conscientious follow-up
8 Persistence
1
The Future

Need for evidence of accountability will
increase
 More faculty will recognize benefits of
assessment
 More electronic assessment methods will be
developed
 More sharing of assessment methods will
take place
 Faculty will learn more about learning and
student learning will improve