Assessment at Selective
Liberal Arts Colleges
A Model for Motivating Inquiry
Christine Brooks Cote
Bowdoin College
Mark Freeman
Colby College
Erin Lowery-Corkran
Rhode Island Office of Higher Education / NEASC
NEASC Annual Meeting, Boston, 2004
1
“Motivating Inquiry Regarding Teaching
and Learning at Critical Transition Points
in Liberal Education”

Funded by the Andrew W. Mellon Foundation

Supported by NEASC, strong leadership from
Bob Froh

One-year Pilot Project

Participating colleges were Amherst, Bates,
Bowdoin, Colby, Smith, Trinity, and Wellesley

Many other colleges attended conferences and
gave support and direction to the project
2
“Motivating Inquiry Regarding Teaching
and Learning at Critical Transition Points
in Liberal Education”
Goal was to test out forms of collaboration
intended to...
...motivate faculty engagement and
...foster a culture of inquiry
3
Liberal Arts and
Assessment
We don’t like it
 Liberal arts education is different
 But it’s here to stay, so we should do it
right

--Dan Chambliss, Hamilton College
4
“Much of the discussion about
assessment, even though focused on
learning, sounds more like a managerial
movement than an academic endeavor.”
--Robert R. Newton, Boston College
5
Outcomes-Oriented
Approach to Assessment
1.
Identification of departmental/programmatic
learning outcomes supporting institutional
mission
2.
Collecting data to determine whether
outcomes have been met
3.
Utilize data to improve or change
departmental/programmatic operations
(curriculum/pedagogy/student life)
6
Inquiry-Oriented
Approach to Assessment
1.
Identify topics of interest for research and
examination within curriculum/pedagogy/student life;
identify research questions
2.
Collect data that will inform the issues; answer the
questions
3.
Provide opportunities for faculty to discuss the data,
what has been learned about the topic, the answers
to the questions; discuss educational goals and
expectations
4.
Use results for change and improvement
7
“Motivating Inquiry Regarding Teaching
and Learning at Critical Transition Points
in Liberal Education”

‘Student Reflective Reports’ – Exploring Student
Intentionality in the First Year

‘Mellon Fellows’ – Integrating Study Abroad with
Campus Experiences and Curriculum in the
Major

Course/Capstone Memos – Assessing the
Accomplishment of Learning Goals in the Core
Curriculum and the Major
8
Why it Worked
Structure:
Collaboration
Through
NEASC
Process:
Inquiry-based
Assessment
Outcomes
assessment
Outcome:
Reflection on
Learning
Outcomes
Changes in
policy /
practice
Further
discussion /
reflection
9
‘Student Reflective Reports’ –
Exploring Student
Intentionality in the First
Year
10
Exploring Student Intentionality
Interview Questions

Round One: After arriving on campus…
– What are your goals for your education? How did you decide upon
these goals? What courses are you taking? How do you think these
courses will help you achieve your educational goals? What
knowledge and competencies or skills do you hope to acquire or
develop during your first year in college?

Round Two: At the end of the first semester…
– What have you learned about yourself and your academic interests
during the first semester? Have your educational goals changed in
any way? How? Why? What courses do you plan to take in the
spring semester? How do these match with the courses you earlier
planned on taking in the spring? Have you done any thinking about a
possible major? What are your thoughts?
11
Exploring Student Intentionality
Data

Student responses differed – some were
more sophisticated, more connected, more
thoughtful, more eloquent than others

It appeared that some were more intentional
and others were less intentional in their
approach to college

Might some kind of developmental model
help us in understanding the data?

Might we come up with some kind of
“intentionality scale?”
12
Exploring Student Intentionality
Data Analysis

Read through students’ responses and
recognized nine dimensions of student
intentionality

Students were rated on each of the
dimensions

Checked for inter-rater reliability
13
Exploring Student Intentionality
Dimensions of Student Intentionality
1.
2.
3.
4.
5.
6.
7.
8.
9.
Developing Skills
Exploring the Curriculum
Pursuing Knowledge
Declaring a Major
Declaring a Career
Making Connections
Planning and Mapping the Future
Reflecting on Own Ideas and Actions
Owning the Self
14
Exploring Student Intentionality
Dimension One: Developing Skills

Low Rating
– Students spoke about gaining proficiency in a
“laundry list” of skills.

High Rating
– Students spoke about their interest and
purpose in developing proficiency in one or
more skills. They spoke of the long-term or
personal benefits of acquiring proficiency in
skills.
15
Exploring Student Intentionality
Dimension Two: Exploring the Curriculum

Low Rating
– Students spoke generally about getting a
“liberal arts education” or becoming “wellrounded”.

High Rating
– Students spoke about wanting to explore the
curriculum and explained why.
16
Exploring Student Intentionality
Dimension Three: Pursuing Knowledge

Low Rating
– Students spoke about the importance of
grades or fulfilling course requirements.

High Rating
– Students expressed a personal interest or
purpose in learning that went beyond
external factors.
17
Exploring Student Intentionality
Dimension Four: Declaring a Major

Low Rating
– Students were undecided or were unable
to explain the reasons for declaring a
certain major or wanting to pursue study
of a particular academic field.

High Rating
– Students spoke about narrowing their
choice of majors and why they wanted to
pursue studying a particular academic field.
18
Exploring Student Intentionality
Dimension Five: Declaring a Career

Low Rating
– Students were ambiguous about future
career plans.

High Rating
– Students spoke of narrowing their choice of
professions based on personal interest and
pursuing experience and knowledge that
would prepare them for their chosen
profession.
19
Exploring Student Intentionality
Dimension Six: Making Connections

Low Rating
– Students did not see a connection between
life and learning inside and outside of the
classroom.

High Rating
– Students were able to connect and integrate
their experiences outside the classroom with
the knowledge they gained in the classroom.
20
Exploring Student Intentionality
Dimension Seven: Planning and Mapping the Future

Low Rating
– Students spoke only of short-term educational
plans.

High Rating
– Students spoke of a long-term educational plan,
such as taking courses to fulfill a major that
would point them towards a desired
profession(s). Some students went so far as to
analyze and reflect on their plans for attaining
their educational and career goals.
21
Exploring Student Intentionality
Dimension Eight: Reflecting on Own Ideas and Actions

Low Rating
– Students spoke of only one or two things
they had learned about themselves, usually
pertaining to their interests or aptitude.

High Rating
– Students spoke about many things they
had learned about themselves that
changed how they were thinking about
their educational plans.
22
Exploring Student Intentionality
Dimension Nine: Owning the Self

Low Rating
– Students spoke about external factors that
shaped their goals.

High Rating
– Students demonstrated a commitment and
interest in pursuing their goals by realizing
that they were responsible for their
education and for making decisions about
how to live their lives.
23
Exploring Student Intentionality
The Intentional Learner
...is purposeful and self-directed. Purpose implies
clear goals, an understanding of process, and
appropriate action. Further, purpose implies
intention in one’s actions.
...possesses a self-awareness about the reason for
study, the learning process itself, and how
education is used.
...makes connections among seemingly disparate
information and draws on a wide range of
knowledge to make decisions or solve problems.
...connects study to personal life, formal educational to
work, and knowledge to social responsibility.
-- AAC&U, “Greater Expectations”
24
‘Mellon Fellows’ –
Integrating Study Abroad
with Campus Experiences
and Curriculum in the
Major
25
Study Abroad
Mellon Fellows
 Advising
4-6 students
 Evaluating senior capstone projects
 Consulting with campus offices and
NEASC to develop senior survey
26
Study Abroad
Advising Logs - Themes





Purpose of study abroad experience
Integration of study abroad experience into
core Liberal Arts curriculum and/or major
Language based versus non-language based
programs
Program policies
Conducting research while abroad
27
Study Abroad
Review of Senior Capstone Projects

To what extent was there direct evidence of the impact of
study abroad experience in the content of the capstone
project?

To what degree did capstone projects based upon study
abroad experiences align well with students’ overall
academic programs?

What was the design and content of capstone projects that
included referencing study abroad experiences?

To what degree do the criteria for the evaluation of
capstone projects address the impact of study abroad
experiences?
28
Study Abroad
Capstone Review
Process allowed faculty to:

Learn more about other types of capstone
projects

Develop methods of evaluation for reviewing
student work

Reflect and articulate standards/criteria used
to evaluate such projects
29
Study Abroad
Study Abroad Survey Components
Students’ motivations for off-campus
study choices
 Preparation and advising for study away
experience
 Student experiences while away
 Perceptions of the impact on academic
and personal growth
 Senior academic activities that built
upon the study abroad experience
 Several “free-response” questions
30

Study Abroad
Study Abroad Survey





Amherst, Bates, Bowdoin, Colby, Smith
Common content, multiple methods
Sharing .html code facilitated merging of
datasets
Large sample size (n
Response
Count
= 795) to test
School
Rate
Amherst
70%
102
relationships
Bates
66%
179
Compare schools in
Bowdoin
66%
131
terms of outcomes,
Colby
96%
281
processes that
Smith
40%
102
“work”
% of
Sample
13%
23%
16%
35%
13%
31
Study Abroad
Study Away Reasons - Cluster Analysis
Reasons for studying abroad items are
the best candidates for a priori or causal
variables.
 If reasons presuppose certain outcomes
then a model focusing on them has
diagnostic value in practice.
 Cluster analysis into types to simply
interpretation.

– A four cluster solution fit the five-school
data best
32
Study Abroad
Selected Cluster Solution
Cluster
characteristics
Traditional Academic Focus?
SA in " similar" country?
Language learning important?
Listen to advice in SA choice?
SA is escape from [Colby]?
Previous visit to SA country?
Likely to be language major?
More than avg non-white?
More than avg. female?
Ex
pe
rie
nti
ali
sts
Na
Ac rrow
ad
em
ics
We
Int ll-Ro
ern und
ati
e
on d
ali
sts
Clu
bM
ed
s
Cluster names are subjective
interpretations of item response patterns
No
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
No
No
Maybe
No
No
No
< Average
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
No
No
Yes
No
No
No
No
33
Study Abroad
Does Cluster Membership Predict OCS Experiences?
Well-Roundeds and Experientialists strong
on cultural immersion, language learning
experiences; weakest of the four clusters on
“adjusting to academic structure”
 Narrow Academics stand out on “keeping in
touch with professors back home” and “course
learning”, “part of intellectual community”;
weakest on cultural immersion experiences
 Club Meds weak pretty much everywhere,
especially traditional academics – but more
cultural immersion than Narrow Academics

34
Study Abroad
Does Cluster Membership Predict OCS Outcomes?
Well-Roundeds strongest on most outcomes,
virtually across the board – “academic integration”
as well as “personal growth” and “cross-cultural
understanding” outcomes
 Narrow Academics weakest on “personal
growth” and “cross-cultural understanding” –
almost as strong as Well-Roundeds in
“integration with post-OCS academics”
 Club Meds bring up the rear virtually across the
board, especially on post-OCS academic
integration. They do, however, report that that
the experience was “rewarding”
35

Study Abroad
Variation Across Schools
Large dataset, many variables, permitted powerful
analysis of school differences while controlling for
program differences
 Accounted for variation in outcomes:

– Non-English speaking country
– Gender (female)
– Natural Science majors weak on “cross-cultural”
learning outcomes
– Well-rounded Internationalists
– School did still account for some variation, after these
factors were controlled
36
Course/Capstone Memos –
Assessing the
Accomplishment of
Learning Goals in the Core
Curriculum and the Major
37
Capstone
Cross-Discipline Discussions

Discipline based conference calls to:
– Share information regarding the different
types and requirements of capstone
project/senior theses
– Develop questions to be answered
– Consider the format of the Capstone Memo
38
Capstone
Capstone Memo Guidelines
 Context
 Evaluation
of Student Work
 Evaluation
of the Course
39
Capstone
Differences Across Departments
 Chemistry
 English
 Social
Sciences
40
Capstone
Discussion Themes
Role of the capstone – culminating
experience or crowning achievement?
 Preparation of students prior to
completing the capstone
 Different opportunities for students to
demonstrate learning through a
culminating experience

41
Therefore, what?
42
What We Learned and Gained
Structures That Worked

Collaboration: Multiple school involvement enhanced
perceived value and importance of project.
– Economy of scale
– Added value, opportunity for peer comparisons

Inquiry-based: Clarifying learning goals within each of
the study areas was very much part of the project itself

Defined focus: Focus on specific “transitional points”:
– Increased faculty interest and involvement
– Encouraged faculty to “think outside the department”

NEASC sponsorship: Not a funding issue so much as a
“signaling” issue.
43
Link to Presentation Materials
http://academic.bowdoi
n.edu/ir/conferences/
neasc2004.shtml
44
Next Proposal
Inquiry-Based Collaborative
Assessment
of Student Learning
in Liberal Arts College
stay tuned!
45