Building an Evidence-Based
Culture in Student Affairs
Presented at the
Ivy Tech Community College
Student Affairs Leadership Summit
July 1, 2009
By
Trudy W. Banta
Professor of Higher Education
and
Senior Advisor to the Chancellor for
Academic Planning and Evaluation
Indiana University-Purdue University Indianapolis
355 N. Lansing St., AO 140
Indianapolis, Indiana 46202-2896
tbanta@ iupui.edu
http://www.planning.iupui.edu
Outline
1.
Characteristics of an evidence-based
culture
2.
An excursion into national
accountability
3.
Assessment in student affairs
© TWBANTA-IUPUI
Why Assess Outcomes in
Student Affairs?
1.
2.
3.
4.
5.
Improve programs & services
Demonstrate accountability
Survive
Inform decision-making
Maintain accreditation
© TWBANTA-IUPUI
St. Louis Community College (1)
Developing a Culture of Inquiry
Faculty and staff collaborate to
- discover
- interpret
- act on
INFORMATION
© TWBANTA-IUPUI
St. Louis Community College (2)
College-wide assessment used to
Revise new-student orientation
Track student success at UM-SL and
improve counseling/advising
Monitor impact of late registration
Monitor effectiveness of new tuition
payment plan
© TWBANTA-IUPUI
St. Louis Community College (3)
Impact of Culture of Inquiry
More collaboration in pursuing
mission
More thoughtful interpretation and
use of data
Better follow-up on identified issues
More use of outside resources like
CCSSE and CAS Standards
© TWBANTA-IUPUI
Council for the Advancement of
Standards in Higher Education
www.cas.edu
© TWBANTA-IUPUI
Plan
Improve
Culture of
Evidence
Implement
Evaluate
© TWBANTA-IUPUI
Planning, Evaluation and Improvement at IUPUI
Planning & Budgeting
1. Mission, Vision, Goals developed
2. Unit goals aligned
3. Programs based on assessable goals,
with performance indicators
4. Annual reports on web
Assessable
Outcomes
 Web-based data
 Electronic portfolios
Culture
Of
Evidence
Instrumentation
external stakeholders
3. Proposing improvement
initiatives based on assessment
findings
4. Improving assessment methods
Application
of Findings
Improvement
1. Reporting to internal constituents
2. Demonstrating accountability to
Tracking
Data Collection
Analysis
Evaluation
1.
2.
Assessment of prior learning
Assessment of learning outcomes
 in major
 in general education (based on PULs)
3. Constituent surveys
4. Academic and administrative program
reviews
5. Campus performance indicators
6. Management information and analysis
7. Program cost analysis
8. Web-based evaluation tools
9. Course evaluations
10. Program evaluation/action research
11. Institutional accreditation
© TWBANTA-IUPUI
Implementation
(Everyone on campus
implements goals)
PLANNING
1.
Campus mission, goals
2.
Unit goals aligned
3.
Programs based on assessable goals
with PIs
4.
Annual reports on the Web
© TWBANTA-IUPUI
Outline for Annual Reports
IUPUI Theme
 Unit Goal
 Objective
Actions Taken
Actions Planned
 Evidence of Progress
© TWBANTA-IUPUI
PAII Evaluation Services
1.
2.
3.
4.
5.
6.
7.
8.
Assessment of learning
Surveys
Program reviews
Performance indicators
Program cost analysis
Web-based evaluation tools
Program evaluation/action research
Accreditation
© TWBANTA-IUPUI
IMPROVEMENT AGENDA
1. Reporting to internal constituents
2. Demonstrating accountability
3. Proposing improvement
-initiatives based on findings
4. Improving assessment methods
© TWBANTA-IUPUI
Surveys
1.
Enrolled Students
Our own
NSSE
2.
3.
4.
5.
6.
Graduates
Employers
Stop outs
Faculty
Staff
© TWBANTA-IUPUI
Information Gateway
http://reports.iupui.edu/gateway/
Information about
Students
Faculty
Staff
Alumni
Finances
© TWBANTA-IUPUI
Since 1993
Campus-wide surveys
have stimulated changes in
Curricula
 Advising
 Increased writing practice
 Increased attention to first-year
experiences
 Placement of graduates

© TWBANTA-IUPUI
What is ABC?
ABC is a costing methodology based upon
the fact that different activities and products
consume different proportions of resources
Product A
Resources
Product B
Activities
Product C
© TWBANTA-IUPUI
Activities performed to produce a
graduate
admit /
enroll
test &
orient
recruit
provide
financial aid
graduate
& place
register
instruct
(recycle)
© TWBANTA-IUPUI
Some Activities within Orientation
1.
2.
3.
4.
5.
6.
7.
8.
Planning the program
Inviting students and parents
Purchasing/duplicating materials
Obtaining spaces and equipment
Setting up spaces and equipment
Providing the program
Evaluating the program
Cleaning space and returning equipment
© TWBANTA-IUPUI
What Is ABC?
Traditional vs. ABC
Traditional Accounting Perspective
n Salary & wages
1,350,000
n Benefits
495,000
n Travel
45,000
n Facilities
220,000
n Supplies
90,000
Total
$2,200,000
Activity-Based Perspective
n Planning
940,000
n Materials
430,000
n Spaces & equipment
250,000
n Program
350,000
n Evaluation
230,000
________________________
Total
$2,200,000
© TWBANTA-IUPUI
Some Applications of
Economic Model
1. Estimate costs of administrative
services as compared to cost of
outsourcing
2. Determine fees for
various programs
3. Restructure processes
to expedite work flow
and minimize costs
© TWBANTA-IUPUI
Since 1992
Activity-based Costing
has stimulated changes in
Planning
n Budgeting
n Assessment
n
© TWBANTA-IUPUI
Elements of Program Review

Self Study

Review by Respected Peers

Recommendations

Follow-up
© TWBANTA-IUPUI
Goals of Program Review at
IUPUI





To improve student learning
To assess and improve program
quality
To increase cross-disciplinary
collaboration
To enhance community connections
To reinforce importance of aligning unit
and campus planning
© TWBANTA-IUPUI
Following the Review
1.
2.
3.
4.
5.
6.
Department reviews the
recommendations
Department prepares written
response
Department presents response in
administrative hearing
Responsible administrators agree
upon responsive action(s)
Responsive actions are
implemented
Re-review ensures action
© TWBANTA-IUPUI
Since 1995
Program Reviews
have stimulated changes in
Planning for the future
 Research emphases
 Faculty hiring priorities
 Advisory councils
 Cross-disciplinary collaboration

© TWBANTA-IUPUI
Program Review at IUPUI
www.planning.iupui.edu/assessment/
© TWBANTA-IUPUI
Goal and Objectives for
Student Learning
Enhance undergraduate student learning
and success
1. Strengthen generic skills
2. Provide honors programming
3. Offer learning communities
4. Strengthen advising
5. Provide tutoring and mentoring
© TWBANTA-IUPUI
Employ Multiple Methods
1) Direct
Projects, papers, tests, observations
2) Indirect
Questionnaires, interviews, focus groups
Unobtrusive measures
Syllabi, transcripts
© TWBANTA-IUPUI
In a Comprehensive
Assessment Program...
INVOLVE
t Students
t Faculty
t Student Affairs
Staff
t Administrators
t Graduates
t Employers
© TWBANTA-IUPUI
Since 1994
Assessment of Learning
has stimulated changes in
Student
support programs
Curriculum
Methods of instruction
Internships
Methods of assessment
© TWBANTA-IUPUI
Establishing a Culture of Evidence
takes

Strong leadership

Support

Time

Evidence that findings are used
to guide improvements
© TWBANTA-IUPUI
Open sharing of information and
evidence-based decision-making

Financial and satisfaction data for units

Annual planning/budgeting hearings

Performance indicators derived from unit
reports over time

Campus performance report for community
© TWBANTA-IUPUI
Program Review &
Assessment Committee
Ÿ
Ÿ
Ÿ
2 reps from each school
2 librarians
Other units
Student Life
Faculty Development
Internship coordinator
© TWBANTA-IUPUI
Characterizing the Culture
New initiatives require assessment
University College
student support programs
Distance learning
New academic programs
© TWBANTA-IUPUI
Characterizing the Culture
Appointment of Assessment
Specialists
University College
Student Life
Service Learning
Library
Enrollment Services
Appointment of Associate Deans for
Assessment
© TWBANTA-IUPUI
Group Assessment Has Failed to
Demonstrate Institutional Accountability
•
Focus on improvement at unit level
•
Rare aggregation of data centrally
•
Too few faculty involved
•
HE scholars focused on K-12 assessment
© TWBANTA-IUPUI
Now We Have
the
Press to Assess with a Test
© TWBANTA-IUPUI
2006
Commission on the Future of
Higher Education
 We
need a simple way to compare
institutions
 The
results of student learning
assessment, including value added
measurements (showing skill
improvement over time) should be . . .
reported in the aggregate publicly.
© TWBANTA-IUPUI
Voluntary System of Accountability
Report Scores in
critical thinking, written communication,
analytic reasoning
using
CAAP
MAPP
CLA
© TWBANTA-IUPUI
At the University of Tennessee
CAAP
Academic Profile (now MAPP)
COMP (like CLA and withdrawn
by 1990)
College BASE
© TWBANTA-IUPUI
In TN We Learned
1)
2)
3)
4)
No test measured 30% of gen ed skills
Tests of generic skills measure
primarily prior learning
Reliability of value added = .1
Test scores give few clues to guide
improvement actions
© TWBANTA-IUPUI
An Inconvenient Truth
.9 = the correlation between SAT
and CLA scores of institutions
thus
81% of the variance in institutions’
scores is due to prior learning
© TWBANTA-IUPUI
How Much of the Variance in Senior
Scores is Due to College Impact?
• Student motivation to attend that
institution (mission differences)
• Student mix based on
• age, gender
• socioeconomic status
• race/ethnicity
• transfer status
• college major
© TWBANTA-IUPUI
How Much of the Variance in Senior
Scores is Due to College Impact?
(continued)
•
Student motivation to do well
• Sampling error
• Measurement error
• Test anxiety
 College effects
______
19 %
© TWBANTA-IUPUI
Better Ways to Demonstrate
Accountability
1. Performance Indicators
Access, social mobility
Diversity
Workforce development
Economic development
Engaging student experience
© TWBANTA-IUPUI
Better Ways to Demonstrate
Accountability
2. Measures of Learning
Standardized tests in major fields
Internship performance
Senior projects
Electronic portfolios
External examiners
© TWBANTA-IUPUI
Student Electronic Portfolio
•
•
•
•
Students take responsibility for
demonstrating core skills
Unique individual skills and
achievements can be emphasized
Multi-media opportunities extend
possibilities
Metacognitive thinking is enhanced
through reflection on contents
- Sharon J. Hamilton
IUPUI
© TWBANTA-IUPUI
Student Development

Involve Student Affairs staff in planning for
assessment

Set expectations for student development
s
Recreation
s
Leadership development
s
Mental and physical health
s
Spiritual development

Monitor student progress

Take warranted improvement actions
© TWBANTA-IUPUI
Questions Guiding Assessment of
Student Services
1.
2.
3.
4.
5.
6.
7.
8.
9.
How many students know about the program?
How many students say they need the program?
How many students use the program?
Do students persist in the program?
What knowledge, skills, attitudes, values do
students develop in the program?
How satisfied are students with the program?
Do program outcomes justify its cost?
Can the program be improved?
Should the program be terminated?
© TWBANTA-IUPUI
Taxonomy of Educational
Objectives
(Bloom and Others, 1956)
Cognitive domain
categories
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Sample verbs for outcomes
Identifies, defines, describes
Explains, summarizes, classifies
Demonstrates, computes, solves
Differentiates, diagrams, estimates
Creates, formulates, revises
Criticizes, compares, concludes
© TWBANTA-IUPUI
Planning for Learning and
Assessment
1. What
2. How
3. How will
general
would you
you help
outcome
know it
students
are you
(the
learn it?
seeking?
outcome)
(in class
if you saw
or out of
it? (What
class)
will the
student
know or
be able to
do?)
4. How could
5. What are 6. What
you
the
improvemeasure
assessments
each of the
ment
might be
desired
findings?
based on
behaviors
assesslisted in #2?
ment
findings?
© TWBANTA-IUPUI
Student Services Activity
Records
•
# participating in activities over the years
• # changing majors
• # meals served in Union
• Average age of students in residence halls
• # voting in campus elections
• # employers coming to campus to interview
• # visits to health clinic
© TWBANTA-IUPUI
Student Affairs Learning Outcomes
•
•
•
•
•
•
•
Leadership skills
Self confidence
Communication skills
Time management skills
Ability to manage stress
Teamwork skills
Overall health
© TWBANTA-IUPUI
Measures of Student Learning





Questionnaires
Interviews
Focus groups
Observations
Transcript analysis
© TWBANTA-IUPUI
Primary Trait Scoring
Assigns scores to attributes (traits) of a task
STEPS

Identify traits necessary for success in
assignment
 Compose scale or rubric giving clear
definition to each point
 Grade using the rubric
© TWBANTA-IUPUI
Primary Trait Scoring
~ Example ~
Trait Desired: Self expression of a feeling
evoked by an event
Scale:
1 - No real expression presented
2 - Feeling expressed but inadequately described
3 - Expression generally competent
4 - Feeling expressed in great detail
5 - Detail plus intelligent response
to feeling evoked
© TWBANTA-IUPUI
Observation of Student Behavior
(including participant observers like RAs)
Check-list of behaviors for
• Actual setting
• Videotape of advising sessions
• Role-playing
• Simulation of incidents
© TWBANTA-IUPUI
Faculty and Staff Development





Focus faculty and student affairs
professionals on improving learning in and
outside class
Attend conferences together
Study literature on student learning
Provide workshops on teaching and learning
Provide resources (e.g., grants, summer
salary, release time)
© TWBANTA-IUPUI
Virginia Commonwealth
University
1. First-year English students wrote 1
response / week for 15 weeks
2. Faculty-student affairs teams read
essays
3. Sample of writers interviewed 18
months later
Responses to findings:
Central advising center, new advising
handbook, multicultural workshops
© TWBANTA-IUPUI
Chicago State University
Purpose: Determine reasons students
leave
TELEPHONE INTERVIEW
Finding: No personal contact for 78%
Response: Freshman seminar revised,
early warning system, referral to
mandatory learning strategies course
© TWBANTA-IUPUI
Santa Barbara City College

Problem: Need to improve retention and
satisfaction with goal achievement

COMMUNITY COLLEGE STUDENT
EXPERIENCES QUESTIONNAIRE

Findings: Strong relationships between
involvement and progress toward goals and
satisfaction with instruction

Response: More extra-curricular activities,
departmental clubs, space for group study
© TWBANTA-IUPUI
Designing Effective Assessment:
Principles & Profiles of Good Practice
Trudy W. Banta
Elizabeth A. Jones
Karen E. Black
Jossey-Bass (Wiley) 2009
© TWBANTA-IUPUI
Profiles
Invited over 1000
Received 146
Selected 49 for use in full
Categorized all 146 and published
Web sites
© TWBANTA-IUPUI
Outline for Profiles
Background and Purpose
Methods over ? Years
Resources Required
Findings
Use of Findings
Impact of Using Findings
Success Factors
Web sites
© TWBANTA-IUPUI
Plan
Improve
Culture of
Evidence
Implement
Evaluate
© TWBANTA-IUPUI
~ Organization ~
of
Principles & Profiles
Planning
Implementing
Improving & Sustaining
- Building a Scholarship of Assessment
Banta & Associates
Jossey-Bass 2002
© TWBANTA-IUPUI
Planning Principles
1.
2.
3.
4.
5.
Engaging stakeholders
Connecting assessment to valued
goals & processes
Creating a written plan
Timing assessment
Building a culture based on evidence
© TWBANTA-IUPUI
Implementation Principles
1.
2.
3.
4.
Providing leadership
Creating faculty/staff development
Assessing processes as well as
outcomes
Communicating and using findings
© TWBANTA-IUPUI
Improving/Sustaining Principles
1.
2.
3.
4.
Providing credible evidence of learning
to multiple stakeholders
Reviewing assessment reports
Ensuring use of results
Evaluating the assessment process
© TWBANTA-IUPUI
Cal State-Sacramento (1)
Sources of Motivation for Assessment
1.
2.
3.
4.
New VP for Student Affairs
Reaccreditation looming
Enrollment & budget challenges
Pledge to become more data-driven
and focused on student learning
© TWBANTA-IUPUI
Cal State-Sacramento (2)
1.
2.
Align department & division missions
Develop SMART goals, 1 for student
learning
Specific
Measurable
Aggressive, yet attainable
Results-oriented
Timely
© TWBANTA-IUPUI
Cal State-Sacramento (3)
Measures
Pre-post MC tests on policies, resources
Essays with rubrics (reinstatement)
Portfolios
Observation of skills
(Leadership, RA reports on scenarios,
role-playing)
© TWBANTA-IUPUI
Cal State-Sacramento (4)
Findings
1.
2.
3.
4.
5.
Some SLOs met
Some SLOs not met
Some measures not effective
Too few participants to assess
Too many participants to assess
effectively
© TWBANTA-IUPUI
Cal State-Sacramento (5)
Use of Findings
1.
2.
3.
4.
5.
6.
Better training for RAs in reporting
Better training for peer mentors in
orientation (emphasizing policies)
More time to discuss films
Better Powerpoint presentations
Increase participation in counseling
Redesign vague test items
© TWBANTA-IUPUI
IUPUI (1)
Themed Learning Communities (TLCs)
3 first-year courses for a cohort
Themes such as Crime in America,
Health & Wellness, African-American
perspectives
Planning group involving faculty,
student affairs, advisor, peer mentor
© TWBANTA-IUPUI
IUPUI (2)
TLCs & Control Group
~ MEASURES ~
1.
2.
3.
4.
GPA
Persistence for 1 year
Common assignments
Student survey responses
© TWBANTA-IUPUI
IUPUI (3)
TLC Benefits
1.
2.
3.
4.
5.
Higher GPAs
Higher persistence rates
See more connections across
disciplines
Encounter more diverse perspectives
Experience more teamwork
© TWBANTA-IUPUI
IUPUI TLCs (4)
Use of Assessment Findings
More detailed collaborative planning
Improved common assignments
More cocurricular activities
More faculty from more disciplines
More TLCs!
© TWBANTA-IUPUI
An Excellent Resource –
Assessment Practice in Student Affairs
by
John H. Schuh & M. Lee Upcraft
Jossey-Bass
2001
© TWBANTA-IUPUI
Assessment Institute
in
Indianapolis
October 25-27, 2009
Westin Indianapolis
www.planning.iupui.edu
© TWBANTA-IUPUI