STUDENT
LEARNING
ASSESSMENT
Options and Resources
Middle States Commission on Higher Education
STUDENT
LEARNING
ASSESSMENT
Options and Resources
SECOND EDITION
Published by the
Middle States Commission on Higher Education
3624 Market Street
Philadelphia, PA 19104
Telephone: 267-284-5000
Fax: 215-662-5501
www.msche.org
© 2007, Copyright by the Middle States Commission on Higher Education
Second Edition
All rights reserved.
First Edition, 2003: This publication replaces Framework for Outcomes Assessment (1996) and Outcomes
Assessment Plans: Guidelines for Developing Assessment Plans at Colleges and Universities (1998).
Second Edition, 2007: The publication Assessing Student Learning and Institutional Effectiveness: Understanding
Middle States Expectations (2005) replaces Chapter 4 in the First Edition.
Reprinted August 2008 with typographical error corrections.
Permission is granted to colleges and universities within the jurisdiction of the Middle States Commission on Higher
Education to photocopy these guidelines for the purpose of institutional self-study and continuing institutional
improvement. Bound copies may be purchased through the publications order form, which is available on the
Commission’s website at www.msche.org.
Printed in the United States of America
ii
Contents
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
v
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
How Accreditation Helps, Accreditation Standards; Purpose and Scope of this
Handbook; Guiding Principles; The Organization of this Handbook; and MSCHE
Website Resources
1. Motivating and Involving Campus Communities . . . . . . . . . . . . . . . . . .
5
Partners in Teaching and Learning; and Leading Assessment Initiatives
2. Learning Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
Benefits of Clearly Articulated Learning Goals; Relationship among Learning Goals
at All Levels; First Steps Towards Developing Learning Goals; Defining Learning
Goals before Selecting Assessment Methods; Ensuring the Quality and Relevance of
Learning Goal Statements; and Resources for Creating Student Learning Goal
Statements
3. Evaluating Student Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
Formative and Summative Assessment; Benchmarking; Direct and Indirect Methods
for Assessing Student Learning; Quantitative vs. Qualitative Evidence; Other
Methodological Considerations; Key Questions When Choosing Assessment
Instruments; Easy-to-Implement Tools and Techniques; and Other Assessment Tools
4. Understanding Middle States Expectations for
Assessing Student Learning and Institutional Effectiveness . . . . . . . . . . . .
54
What is the Assessment of Institutional Effectiveness (Standard 7)? What is the
Assessment of Student Learning (Standard 14)? Why does the Commission Expect
Student Learning and Institutional Effectiveness to be Assessed? What are the
Characteristics of Assessment Processes that Meet Middle States Expectations?
What Should Institutions Document Regarding Assessment? How Should This
Information be Organized and Formatted for Review by the Commission and
Its Representatives? How are the Documentation of Institutional Assessment and
Student Learning Assessment Related? What Might the Commission and Its
Representatives Look For in Assessment Documentation?
5. Using Results to Improve Teaching and Learning. . . . . . . . . . . . . . . . . .
Institutional Support Strategies Designed to Encourage the Use of Assessment
Results; Translating Assessment Results into Better Learning; and A Message to
Faculty: The Interconnectedness of Assessment, Teaching, and Learning.
iii
59
Appendices:
1. Assessment Standards in Characteristics of Excellence in Higher Education . . .
2. Enhancing the Campus Climate for Assessment: Questions for Academic Leaders
3. Assessment Practices Quiz . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4. Key to “Assessment Practices Quiz” . . . . . . . . . . . . . . . . . . . . . . . .
5. Department/Program Student Outcomes Survey . . . . . . . . . . . . . . . . . .
6. Learning Goals and Assessment Techniques . . . . . . . . . . . . . . . . . . . .
7. From Effect to Cause: A Brainstorming Exercise. . . . . . . . . . . . . . . . . .
8. Student Learning Styles: Frequently Asked Questions . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
75
84
86
87
88
91
92
93
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
97
List of Figures
1.
Relationship between Institutional and Program Goals . . . . . . . . . . . . . . .
13
2.
Relationship between Program and Course Goals. . . . . . . . . . . . . . . . . .
14
3.
Leading Questions for Developing Learning Goals . . . . . . . . . . . . . . . . .
22
4.
Teaching Goals Inventory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
5.
Examples of Direct and Indirect Measures of Student Learning
(Course, Program, and Institutional Levels) . . . . . . . . . . . . . . . . . . . . .
29
6.
Criteria-based Rating Scales (Rubrics) . . . . . . . . . . . . . . . . . . . . . . .
43
7.
Excerpt from a Simple Rating Scale . . . . . . . . . . . . . . . . . . . . . . . . .
44
8.
Example of a Detailed Rating Scale . . . . . . . . . . . . . . . . . . . . . . . .
46
9.
Example of a Holistic Scoring Guide (For Critical Thinking). . . . . . . . . . . .
48
10.
Student Self-reflection Questions for a Course or Program . . . . . . . . . . . . .
49
11.
Example of a Test Blueprint . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50
12.
Commonly-administered Measures of Critical Thinking . . . . . . . . . . . . . .
52
13.
Considerations when Deciding to Use Portfolios . . . . . . . . . . . . . . . . . .
53
14.
Using Assessment Results at the Institutional Level:
Maintaining Mission and Achieving Vision . . . . . . . . . . . . . . . . . . . . .
64
Using Assessment Results at the Program Level:
Preparing Students for Future Success . . . . . . . . . . . . . . . . . . . . . . .
66
16.
Using Assessment Results at the Course Level: Ensuring Learning . . . . . . . .
68
17.
Strategies to Improve Student Learning . . . . . . . . . . . . . . . . . . . . . . .
71
15.
iv
Acknowledgements
The Middle States Commission on Higher Education is grateful for the assistance of its 2001 Advisory
Panel on Student Learning and Assessment for developing the basic text for this publication, with
additional contributions by the Commission, its staff, and some colleagues in higher education.
The Advisory Panel on Student Learning and Assessment
[Positions as of Date Appointed: June 2001]
Dr. Peter J. Gray (Chair), Associate Director, Center for Support of Teaching and Learning, Syracuse University
Dr. Michael J. Dooris, Director, Planning Research & Assessment, The Pennsylvania State University
Dr. Thomas V. Fernandez, Professor of Physics, Nassau Community College
Dr. Bruce Keith, Assistant Dean for Academic Assessment, United States Military Academy
Dr. Armand S. LaPotin, Professor of History and Academic Program Coordinator, SUNY College at Oneonta
Dr. Elizabeth Larsen, Professor, Coordinator of General Education, West Chester University of Pennsylvania
Dr. Jodi H. Levine, Assistant Vice Provost for University Studies, Temple University
Dr. Rosalyn Lindner, Senior Advisor to Provost for Assessment, SUNY College at Buffalo
Dr. Peter J. Miller, Assistant Professor, Department of Physical Therapy, University of the Sciences in
Philadelphia
Dr. Paula E. Peinovich, Vice President for Academic Affairs, Excelsior College
Ms. Linda A. Suskie, Director of Assessment, Towson University; Former MSCHE Fellow
Dr. Barbara E. Walvoord, Director, Kaneb Center for Teaching and Learning; Concurrent Professor of English;
and Fellow, Institute for Educational Initiatives, University of Notre Dame
Prof. Katrina A. Zalatan, Assistant Professor of Management, Hartwick College
Additional Support Provided by:
Dr. MaryAnn Baenninger, First Edition Editor-in-Chief; MSCHE Executive Associate Director
Ms. Siobhan Underwood, MSCHE Graduate Fellow and 2001 Assistant to the Advisory Panel
Ms. Linda Suskie, Second Edition Editor-in-Chief; MSCHE Vice President
Ms. Jean Avnet Morse, Editor; MSCHE President
Mr. Oswald M. T. Ratteray, Editor; MSCHE Associate Director for Communication
v
Introduction
C
olleges and universities have long defined
and assessed student learning using
course-embedded assessments of student
learning, such as tests, papers, projects, as well as
standardized or “custom” qualitative and
quantitative measures. All of these are valid and
valuable assessment tools if used properly.
One of the means by which the public can
understand higher education is through information
about the assessment of student learning. As an
institutional accreditor, the Middle States
Commission on Higher Education, with the support
of its institutional members, acts on the judgments
of volunteer peer reviewers who certify that
institutions assess themselves in all areas, including
student learning.
In order to reach out more effectively to students
and to the public, the Middle States Commission on
Higher Education revised its accreditation
standards, Characteristics of Excellence in Higher Accreditation Standards
Education: Eligibility Requirements and Standards
for Accreditation, to refine the requirements and
Among the principles that guided the revision of
recommendations for establishing learning goals
the Commission’s standards is greater emphasis on
and assessing student achievement.
institutional assessment and the assessment of
student learning. By complying with the standards,
The members also concluded that the process of
defining and assessing student learning would assist accredited institutions assure the public that they
provide quality higher education. Specifically, the
faculty in their teaching, students in selecting
Commission’s process demonstrates that
institutions and in managing their own learning,
institutions identify student learning goals for
and institutions in planning and supporting
educational offerings that are appropriate to its
students.
higher education mission; that its offerings
This handbook serves as a resource for institutions display appropriate academic content, rigor, and
seeking a bridge between the Commission’s
coherence; that its curricula are designed so that
standards for accreditation and the practical daily
students demonstrate college-level proficiency in
challenges of assessment and continuous
general education and essential skills, including
improvement.
oral and written communication, scientific and
quantitative reasoning, critical analysis and
reasoning, technological competence, and
How Accreditation Helps
information literacy; and that assessment
demonstrates that students at graduation have
It is in the interests of currently enrolled and
achieved appropriate higher education goals.
prospective college students, faculty members,
parents, high school teachers and guidance
The accreditation standards relating to assessment
counselors, legislators, employers, and the general are intended to foster and cultivate the progress of
public to be informed consumers of higher
member institutions. They are not intended to be
education.
prescriptive. Each standard stresses the significance
of self-study and peer review as a developmental
activity. The ways in which individual institutions
carry out assessment activities and determine the
extent to which their goals for student learning have
been met is an institutional prerogative. Because of
the diversity of institutional types, missions, and
educational practices that characterize the members
in the Middle States region, Characteristics
One function of accreditation is to provide the
public with an explanation of the broad scope of
higher education and to assure the public that the
goals of higher education have been achieved by
evaluating each institution within the context of
its mission.
1
provides institutions with guidance on how
different types of institutions might fulfill each
standard.
Purpose and Scope of this Handbook
This handbook is intended to clarify principles and
methods for setting goals for student learning
within the context of institutional mission, for using
methods chosen by the institution for evaluating the
achievement of these goals, and for using the
information gathered to continue to improve
student learning. It is not an expansion of the
Standards for Accreditation described in
Characteristics; it is meant only as a resource.
Characteristics of Excellence in Higher Education
reflects this renewed and increased emphasis on
institutional assessment and the assessment of
student learning in several ways. Each of the
14 standards is accompanied by “fundamental
elements” that guide the institution in assessing
itself on the standard.
With the increasing use of alternative forms of
delivery, including distance learning and
asynchronous delivery, a focus on outcomes
becomes even more essential. Student learning
outcomes form a “common currency” with which
one can judge the equivalence and value of various
learning experiences.
Teams that evaluate institutions at the time of their
decennial self-studies and evaluators who review
institutions’ Periodic Review Reports, Follow-up
Reports, and Substantive Change proposals will use
the standards themselves, rather than this
handbook, to assess the institution.
The audience for this handbook includes all
stakeholders of a college or university, including
faculty, students, staff, administrators, and the
general public. It is intended to assist directly those
responsible for setting goals for student learning
and for evaluating the achievement of those goals.
The standards are organized into two subsections:
Institutional Context and Educational Effectiveness.
The concluding standards of each of these two
subsections require that an institution define,
evaluate, and continually refine its overall goals
(Standard 7), with special emphasis on goals for
student learning (Standard 14):
This handbook describes:
Standard 7: The institution has developed and
implemented an assessment plan and process that
evaluates its overall effectiveness in achieving its
mission and goals, its efficiency in the use of its
resources, and its effectiveness in assuring that its
students and graduates achieve the appropriate
learning and other outcomes.
q How faculty and staff members can define
clearly student learning and affective goals
appropriate for an institution’s mission;
Standard 14: Assessment of student learning
demonstrates that the institution’s students have
knowledge, skills, and competencies consistent
with institutional goals and that students at
graduation have achieved appropriate higher
education goals.
q How student learning can be improved by
relating outcomes to the institution’s
operations and resources; and
q Various direct and indirect methods of
evaluating student learning and the value
and appropriate use of each approach;
q How traditional methods of teaching and
learning can be enhanced to produce clear
and useful information about how and what
students are learning, both inside and
outside the classroom, so that faculty,
students, the institution, and the general
public can benefit from improvements.
These standards are mutually supportive, because
they recognize the centrality of student learning to
institutional effectiveness and stress that the
assessment of outcomes should be integrated into
the institutional planning process. See Appendix 1
for an expanded description of these standards.
The handbook presents various possible means of
meeting the Commission’s standards. It describes
various contexts and options for assessing student
learning, and it provides resources as examples of
how institutions might approach the assessment of
student learning on their campuses. It also discusses
2
some of the many considerations that should be
explored before intensive institutional effort is
directed at articulating learning goals, choosing
means of evaluating the accomplishment of those
goals, and crafting an institutional plan for
assessment.
Guiding Principle 1: Existing Culture
Begin by acknowledging the existence of
assessment throughout the institution in order to
ensure that the assessment plan is grounded in
the institutional culture.
Guiding Principle 2: Realistic Plan with
Appropriate Investment of Resources
Guiding Principles
Plans for assessment at the program, school, and
institutional levels should be realistic and
supported by the appropriate investment of
institutional resources.
This handbook serves as a starting point for
institutions beginning or enhancing their
self-assessment activities, particularly those
activities related to student learning.
It is written for faculty, staff, and administrators— Guiding Principle 3: Involvement of Faculty
those who will actually be leading and conducting and Students
assessment efforts on their campuses. Its purpose is
Academic leadership is necessary in order to gain
not limited to providing a route to achieving
the support and involvement of faculty members,
accreditation or reaffirmation of accreditation, but
staff, administrators, and students across the
rather it is intended as a resource and guidebook for
institution.
institutional self-reflection, improvement, and
achievement of the best possible outcomes for
Guiding Principle 4: Clear Goals
students.
Assessment activities should be focused by
It also can be a resource for those who wish to learn
clear statements of expected student learning
about assessment practice in general: what it is,
(knowledge, skills, and competencies).
why it is important, who benefits from it, how it is
accomplished, and how accreditation supports
Guiding Principle 5: Appropriate Methods
assessment.
Assessment should involve the systematic and
thorough collection of direct and indirect
evidence of student learning, at multiple points in
time and in various situations, using a variety of
qualitative and quantitative evaluation methods
that are embedded in courses, programs, and
overall institutional processes.
Six guiding principles serve as the framework for
this handbook, and they are relied on throughout
the handbook. These principles are adapted from
AAHE’s principles for good practice in assessing
student learning (Astin, 1991) and Assessment in
practice: Putting principles to work on college
campuses (Banta, Lund, Black, and Oblander,
1996).
Guiding Principle 6: Useful Data
While the recommendations, ideas, resources, and
perspectives in this handbook are offered as
examples and as flexible models and blueprints,
there is no “one size fits all” type of institutional
assessment or student outcomes assessment. Thus,
the principles presented here should serve as the
basis and guiding structure for assessment activities
and the resources described in the handbook should
serve as possible tools—among many—for
achieving institutional goals.
Data gained through assessment activities should
be meaningful. They should be used, first, to
enhance student learning at the institutional,
program, and course levels; second, in
institutional planning and resource allocation;
and third, to evaluate periodically the assessment
process itself for its comprehensiveness and
efficacy.
3
The guiding principles are intended to help
Each of the chapters in this handbook focuses on a
institutions answer the following general questions: different component of the assessment process and
describes considerations, options, and resources
related to that component. The chapters are meant
q What should our students learn? (Chapter 2)
to stand alone or in combination with each other,
q What are our institutional strengths and
and the handbook can be valuable to support
challenges for improvement? (Chapters
different approaches.
1 and 2)
q How are we currently organized for
evaluating learning? (Chapters 3, 4, and 5)
q What activities have we conducted to define
and evaluate all of our institutional goals,
with special emphasis on goals for student
learning? (Chapters 3 and 4)
q What existing evidence do we have
regarding student learning and achievement,
and what have we learned from that
evidence? (Chapters 3 and 5)
q What actions will we take to build on our
strengths and to address our weaknesses
regarding student learning? (Chapter 5)
The Organization of This Handbook
Readers of this handbook will be approaching the
task of evaluating student learning from many
vantage points. Some institutions will have existing
institution-wide assessment plans that need
refinement or adaptation. Other readers will be
embarking on cyclic evaluation of their assessment
plans at the course, program, or institutional level.
Still other readers will be just starting to help their
institutions set goals for student learning at the
course, program, and institutional level.
In order to address the needs of every institution
and every reader, this handbook starts with the
development of learning goals and individual
assessment strategies, builds to the creation of a
written assessment plan, and ends with a chapter on
using assessment results.
Even readers from institutions that have an existing
plan can benefit from the focus in this handbook on
setting learning goals and assessing the related
outcomes.
4
1
Motivating and Involving
The Campus Community
he purpose of assessment is to engage a
campus community collectively in a
systematic and continuing process to create
shared learning goals and to enhance learning.
Those who have direct instructional and supportive
contact with the students and those who lead
assessment initiatives are responsible for
motivating and involving the rest of the campus
community.
T
Partners in Teaching and Learning
The best way to motivate the community is to
promote an understanding of the benefits that
assessment brings to students, faculty, the
institution, and the public. The extent to which
learning goals and assessment processes that
already exist will form the core of a more clear and
integrated assessment process.
Faculty and Students
All campus members are partners in teaching and
learning and have a role in evaluating and
enhancing student learning. Those who have direct
instructional and supportive contact with students
include faculty, library and information literacy
professionals, and student support service
professionals.
Faculty members traditionally have had the primary
responsibility for facilitating student learning. They
determine what students should learn, both across
the curriculum and within individual courses or
programs, and how students should demonstrate
their learning. Faculty members devise methods of
Students, of course, want to attend the institution
gathering evidence of student learning and
that suits them best. Parents want the best value, or
collaborate with other faculty members in
perhaps the “cultural capital,” that an institution
evaluating student learning in their majors and
affords. Parents and students are interested in which
academic programs. They use this information to
institution will provide them with the education that
create a true partnership of learners with their
will result in a job, or acceptance to graduate or
students and to improve student learning.
professional school. Employers are interested in the
Huba and Freed (2000) provide examples of how to
“product” that a college or university produces.
develop this partnership and the benefits it offers.
Faculty members, too, have a vested interest in
Faculty members, who are trained as disciplinary
students being informed about their choice of a
experts, as scholars, and as researchers, can amplify
college or university to attend. It is much easier,
their skills by exploring further how students learn
and more enjoyable, for faculty members to teach
best.
students who are appropriately prepared for their
courses, either through earlier preparation or
through foundation courses at the institution.
5
First-year students arrive at their institutions eager
to embark on collecting the credits toward the
credential they believe they need to ensure their
long-term economic futures. If they are traditional
residential students, they also seek to experience
college life in its totality. Returning students or
continuing education students may have other, even
more pragmatic, reasons for attending college.
Often, however, neither group has had sufficient
prior experience in reflecting on how to learn,
evaluating the extent of what they have learned and
what they still need to discover, or using their new
knowledge and skills.
It is important for committed faculty members and
other institutional leaders to focus on the faculty,
staff, and student partnership, and to avoid
top-down or prescriptive rules for accomplishing
assessment. For example, it may be tempting for
institutions to pressure faculty members to orient
professional programs too heavily towards goals
that promote only practical skills. Conversely, the
need to define student learning goals might be
perceived as a potential threat to academic freedom.
Close partnership of faculty, librarians, student
affairs professionals, and students in defining
learning goals consistent with institutional mission
should avoid such extremes.
These faculty and students are partners who find
themselves engaged together in the pursuit of
knowledge, skills, and affective development.
They must cooperate to produce the best learning
possible.
Library and Information Literacy
Professionals
Not all learning occurs in the classroom. Therefore,
library and information literacy professionals also
Sometimes, however, faculty members engage in
this process automatically, without questioning the have a critical role in the process of enhancing
student learning. Together with faculty, students,
tacit assumptions underlying their concept of
and other staff members, they can address the full
teaching and learning.
range of learning in a student’s college career.
For instance, end of semester term papers are
Ideally, methods of facilitating student learning
regularly assigned as a means of evaluating student should exist in other divisions of an institution or
learning. Yet the typical term paper assignment is a should be integrated into coursework. A focus on
good illustration of how traditional approaches may information literacy is an important component in
not necessarily effectively foster learning.
achieving this objective, especially if it is integrated
into curricular and co-curricular facets of the
The paper may be assigned early in the semester,
without the requirement of an outline or draft of the institution.
paper during the course of the semester. The
The information literacy paradigm consists of five
professor’s concept of what an excellent paper
skills for learners (Association of College and
should be, including its form, its content, and the
Research Libraries, 2000):
process for completing it, may not have been
q Determining the nature and extent of needed
communicated effectively to the student when the
information;
paper was assigned. Therefore, the student may not
have engaged in a process designed to meet those
q Accessing information effectively and
expectations. Furthermore, the paper may be graded
efficiently;
after the semester is officially finished, may contain
q Evaluating critically the sources and content
no comments, and may or may not be returned to
of the information being sought, and
the student.
incorporating selected information in the
Assessment of student learning is not a means of
learner=s knowledge base and value system;
decreasing the autonomy of faculty members. It is a
q Using information effectively to accomplish
means of increasing the mutual engagement of
a specific purpose; and
faculty members, staff, and students in providing an
optimal learning experience.
q Understanding the economic, legal, and
social issues surrounding the use of
information and information technology, as
well as observing laws, regulations, and
6
institutional policies related to the access
and use of information.
Faculty and staff members who work with students
should have ownership of the assessment process as
well as full and continuing administrative support
The principles of information literacy are invoked
for its implementation. Such support is best gained
any time a student attempts to learn anything in any
through the public recognition of faculty and staff
discipline. To the extent that the expected learning
members’ ongoing efforts and accomplishments in
involves the use of resources available in or
assessing student learning. Clear recognition
through the library, librarians and faculty share
demonstrates an institution’s commitment to a
responsibility for various aspects of the process for
culture that values the enhancement of student
teaching information literacy.
learning.
The subject of information literacy is explained in
Palomba and Banta (1999) note that one or more
detail in a separate Commission publication of
leaders should take responsibility for leading the
guidelines on information literacy (Middle States
campus-wide assessment process, that resources
Commission on Higher Education, 2003) and
must be committed, and that the institutional
clarified even further in Standards 11 and 12 of
priority of assessment should be explicit. Activities
Characteristics of Excellence, 2006).
to involve faculty, staff, and students should be
sponsored by the academic leadership and, in
Student Support Service Professionals
particular, supported by the chief academic officer.
Those who administer services such as residential
Appendix 2 lists some self-reflection questions for
life, advising, career development, learning support,
chief academic officers to help them gauge their
service learning, and financial aid are partners with
own level of commitment to assessment activities
faculty members and students in developing
and to reflect on how they might enhance the
outcomes and assessing student learning. For
campus climate for assessment. Appendix 3 is a
example, student development personnel help
short “quiz” that can be used on campuses to
students to develop their own ethical values and
stimulate discussion about assessment. Although
to achieve the institution’s goal of graduating
the quiz is designed for true or false responses,
students who are responsible citizens in a
many of the questions are more ambiguous than
multicultural society.
they appear in order to generate a dialogue about
assessment; it is not a test of assessment
knowledge. Explanations of the “correct” answers
Leading Assessment Initiatives
may be found in Appendix 4.
Effective leadership is necessary to create a culture In addition to campus conversations, the
that values student learning assessment within the
institution’s leaders, and particularly the president
institutional context.
and the provost, can be partners with faculty to
introduce and to establish assessment by adopting
the following approaches:
Campus Leaders
Leaders should involve all constituencies in
understanding how assessment can be helpful, in
identifying the learning goals that are most
important to the community, in assessing outcomes,
and in using the results.
q Increase the awareness of assessment on
campus, articulate and define assessment
issues and priorities, and identify the
institution as an environment that supports
assessment practices.
Campus leaders might sponsor faculty-led
discussions of the issues and concerns related to
assessment, present workshops led by internal or
external experts, organize faculty and student
forums that provide an overview of assessment on
campus, address larger assessment issues, and
answer assessment questions.
q Acknowledge assessment activities that
already exist and promote fuller
participation in assessment activities by
facilitating communication and discussion
among the institution’s members, with the
goal of achieving shared responsibility for
assessment.
7
q Be a sponsor of assessment who shares
leadership in bringing about this change in
the campus culture.
A private comprehensive university, such as a
faith-based institution, may develop assessment
goals related to an ecclesiastical mission.
q Bring participating members into the
assessment process by identifying existing
coalitions and developing new coalitions,
with the goal of opening the process to as
many as possible.
Within the context of its mission and broad context,
each college or university will have subsidiary and
more specific purposes. Thus, one university might
stress the development of civic leadership or
technical expertise, and another institution might
stress pre-professional development and global
citizenry.
q Provide funding and other incentives for
participation in the assessment process,
institutionalizing assessment, and
integrating assessment into the faculty and
staff roles and rewards process.
The link between mission and the development of
goals—in this case, goals for student learning—is
clearly expressed in Standard 1 of Characteristics,
which requires that an “institution’s stated goals
and objectives, consistent with the aspirations and
expectations of higher education, clearly specify
how the institution will fulfill its mission.” For
example, training excellent teachers or insightful
managers may express one institution’s mission,
while training academic scholars and researchers
are desired goals for another institution.
q Provide a clear charge to an appropriate
campus assessment committee responsible
for communicating expectations for
assessment.
Institutional Context
The institutional context is grounded in the
institution’s mission, and it is shaped by the
institutional culture.
Failure to clarify the institution’s goals and
strengths may result in misallocated resources and
confused students and applicants. Students may
Mission. An institution’s mission, at both broad
come to an institution, for instance, with the goal of
and specific levels, serves as the context within
which to assess student learning, and it is important becoming nurses or biomedical technicians, only to
that mission serves as the backdrop for assessment find that the health sciences do not fit well within
the institution’s mission. The result is that they may
efforts at the institutional, program, and course
find a “disconnect” between the mission and goals
levels. An institution’s broad contexts will shape
of a college or university and the learning outcomes
overall goals for student learning and how that
that its students hope to achieve for themselves.
learning is demonstrated.
Leaders of institutional assessment initiatives, then,
should refer back constantly to the institutional
mission and should articulate to faculty,
administrators, board members, and donors the
fundamental importance of designing learning goals
that are consistent with the institutional mission in
order to serve both their students and faculty.
For instance, a comprehensive university in the
public sector will have a mission driven in large
part by the needs and interests of the state, while a
private comprehensive university’s mission may
focus on the interests of its founders or trustees. In
the case of the public university, accountability to
the state may include the demonstration of service
to the community, success in workforce
development, and the ability to keep intellectual
resources within the state, in addition to the
demonstration that students are liberally educated.
Private colleges may have missions that focus
solely on the liberal arts.
A comprehensive public sector institution may
articulate assessment goals through an emphasis on
natural resource management, mining or agronomy
programs, characteristic of that state’s economy.
Institutional Culture. As an institution begins to
plan for assessment, it is important that it consider
the particular aspects of institutional culture that
might affect the form and process of assessment
practice on its campus. Respect for how consensus
is achieved and recognition of existing structures,
both official and unofficial, will pave the way for
reaching the ultimate goal of improving student
learning. Following are some questions that leaders
8
or potential leaders of campus assessment
initiatives can ask themselves as they embark on
new or changed assessment activities:
describe the campus culture, bring objectivity to the
review, and offer a fresh perspective. This
consultant could work with the chief academic
officer or other campus leader to articulate the
underlying assumptions of the culture, reveal any
“surprises” lurking beneath the surface, and devise
strategies for change.
q What is the quality of communication on
campus? Can it be improved before
implementing an assessment plan?
q How is decision-making handled on
campus, both formally, and informally?
q What is the level of trust on campus? If trust
is a problem, how can it be earned?
q What person or persons on campus are
perceived to hold unofficial power? How
can those persons be convinced of the
benefits of assessment? How can they serve
as sources of support for assessment
initiatives?
q What is the system of apportioning
resources on campus? If there are concerns
about equity of resource distribution (or
perceived concerns), can they be addressed
before implementing an assessment plan?
q What is the process by which curricula and
programs are approved and revised? Does
the process present any impediments to
assessment, and can it be improved or
streamlined?
q Are there collective bargaining agreements,
union-wide or local, which could either
support or impede assessment practices?
For example, are union funds available
to support faculty development that could be
used for assessment? Do union rules restrict
“official” work during the summer?
Leaders of assessment on campus should consider
these and other factors that could influence the
institution’s ability to do assessment well. If the
campus culture is not functioning in a manner that
is likely to be supportive of assessment, it would be
useful to take steps to “heal” the culture before
instituting large-scale assessment initiatives. Schein
(1996) offers some excellent advice on assessing
campus culture and suggests that it may be
beneficial to an institution to enlist the advice of an
external consultant. A consultant can review and
9
2
Learning Goals
oals for student learning are the foundation
q Help students to understand the nature of
of meaningful assessment. Statements of
skills acquired for use in other
desired student outcomes can be derived
contexts—during and after college
through a variety of effective methods at the
institutional, program, and course levels. This
Benefits for Faculty and Staff
chapter describes the benefits of having clearly
Statements of student learning goals benefit faculty
articulated learning goals, explores the
and staff because they:
characteristics of learning goal statements, and
provides resources for implementing participatory
q Identify what to teach, including
processes for developing goals.
discipline-specific knowledge and skills,
as well as the discipline’s perspective
and values
G
Benefits of Clearly Articulated
Learning Goals
q Provide structure for co-curricular programs
q Determine what will be evaluated at the
conclusion of the course or program
Clearly articulated statements of what each
institution expects its students to learn at the course,
program, and institutional levels are important to
students, faculty, staff, the institution, and the
public for many reasons.
Benefits for Students
Statements of student learning goals benefit
students because they:
q Ensure that skills that should be taught
throughout the curriculum actually are
included in instruction and evaluation of
specific courses
Benefits to the Institution
Statements of student learning goals benefit the
institution because they:
q Explain the sometimes “hidden agenda”
(e.g., the expectation that students analyze
relationships between causes and effects,
rather than simply learn substantive
material)
q Publicize to the institution’s constituents
evidence that it can demonstrate the
accomplishment of clearly-defined student
learning goals
q Ensure that goals the institution itself values
are assessed, rather than those used by
external assessors (e.g., sophisticated
analytical math skills versus minimal
national competency levels)
q Prioritize which goals are most important
q Provide assurance that a student has not
“missed” an important goal
10
q Ensure that student learning outcomes are
suited to the mission of the institution
goals and assess them at varying levels, depending
on the needs of the institution.
q Ensure that core institutional values
(e.g., professional career development
and approaches of different cultural
perspectives) are sufficiently incorporated
Learning goals at the institutional, program,
and course levels
Students learn specific content and skills in each
course. In aggregate, those courses, together with
q Ensure that general education skills,
other program experiences such as academic
such as proficiency in oral and written
communication, the ability to think critically advising, internships, and faculty-directed research
by students, should result in the desired student
and analytically, and the ability to be
outcomes at the program level. Similarly, goals at
effective decision-makers and
the program level combine with general education
problem-solvers are included in
goals, extra- and co-curricular goals, information
programmatic plans
literacy goals, and other goals (for example, ethical
q Ensure that the personal growth and
and civil leadership goals) to create institutional
affective development of students are
goals. In other words, goals at the institution,
addressed
program, and course (or activity) levels are
q Focus attention on the use of direct methods interconnected, complimentary, and reciprocal.
of assessing student learning, supported by Institutions differ in the way that they characterize
meaningful indirect methods, instead of
the relationship between general education goals
potentially less meaningful indirect
and institutional goals. In one model, the institution
measures often used by:
develops a set of overall institutional learning goals
stemming from its mission; these goals serve as the
• external assessors (e.g., graduation
super-ordinate (highest level) goals from which
rates, cost efficiency, etc.)
program and course level goals flow. In this format,
• internal assessors (e.g., student
general education goals are essentially
evaluations of faculty)
programmatic goals; that is, the general education
program is one of the programs whose goals
Benefits for the Public
contribute to the achievement of overall
institutional goals.
Statements of student learning goals benefit the
public because they:
q Enable students to choose an institution
based on a particular mission
q Satisfy accountability needs of legislators,
funding agencies, and others
q Help the public to understand more clearly
what an institution seeks to accomplish
Relationship among Learning Goals
At All Levels
Before developing or revising learning goals
institution-wide, it is important to consider the
relationship among learning goals at the
institutional, program, and course levels.
In addition, different institutions might develop
In another model, the institution adopts general
education goals as overall institutional goals. In this
approach, academic and co-curricular program
goals would contribute to the achievement of the
umbrella-like general education goals, which are
essentially institutional goals.
Standard 14 of Characteristics, the Assessment of
Student Learning, includes language that is most
similar to the first model presented above—that is,
it describes the assessment of student learning at the
institutional, program, and course levels. Standard
12, General Education, also includes its own
“fundamental element” related to the assessment of
general education.
The Commission, however, is not concerned with
the language that an institution uses to describe
various levels of learning goals, nor is it concerned
with the specific type of hierarchical structure an
11
institution adopts when defining its goals. It is
concerned that the institution develops a coherent
set of goals, that those goals stem from the
institutional mission, and that goals at the
subordinate levels contribute to the attainment of
goals at the higher levels. The way in which a
particular institution defines general education
goals relative to institutional goals depends on the
institution’s mission (e.g., a specialized institution
is unlikely to adopt general education goals as
institutional goals). It also depends on how the
general education program is structured (e.g., Is it
“modular” or are its goals met in part through the
major? Are some of its goals met through student
affairs programs?) Finally, developing general
education goals depends on the institution’s history
and culture.
overlap and that not all programs work toward
meeting all institutional goals.
Figure 2 depicts the hypothetical relationship
between a subset of the program goals presented in
Figure 1 and sample goals from courses in each of
those programs. Notice, for instance, that one of the
goals in the course World Art History, to “identify
and analyze major works representing several
different cultures,” contributes to the general
education program goal to “recognize and
appreciate artistic and literary contributions of
diverse cultures,” which in turn contributes to the
institutional goal to prepare “global citizens.”
At the course level, the faculty member teaching
World Art History will have many additional goals.
Some will contribute further to the achievement of
general education goals, but others may contribute
Figure 1 and Figure 2 describe levels of learning
to the achievement of goals for the major in Art
goals at the institutional, program, and course
History. Because of the interdependence among
levels. Although some institutions actually create
goals and the course and program levels, it could be
matrices like these to aid them in formulating goals,
impractical, or perhaps impossible, to specify all of
this matrix is not presented as a model of how goals
the links between goals for each course and for
should be illustrated but, rather, as an abstraction to
each program in matrices. It is more important to
help the reader understand the relationships
strive for conceptual coherence, ensuring that
between levels of learning goals.
learning goals at the various levels are
understandable, meaningful, and accessible to
Institutional and Program Goals. Figure 1
faculty and students alike.
illustrates hypothetical relationships among
learning goals or statements of student outcomes at
Flexibility
the institutional and program levels. The desired
outcomes at the institutional level provide the
In Standard 14, the Commission recognizes that
outline or framework for connecting goals at the
institutions will be “flexible in their approach to
program level into a coherent whole.
defining student learning goals at these different
These illustrations include goals for disciplinary
levels, such as repeating goals (some general
and major programs, general education, and a
education goals, for example) across programs or
student-affairs oriented program, such as residence defining the goals at the institutional or program
life. Goals from various activities and initiatives
level as synthesis of the goals set at the program
contribute to overall student affairs goals. Because and course levels.”
student affairs and academic programs both
For instance, an institutional goal for undergraduate
contribute to the overall education of the student,
students to become proficient in information
goals from each of these programs work together
literacy may be achieved through a combination of
to fulfill institutional goals.
the active involvement of faculty in information
Program and Course Goals. Figure 2 illustrates
literacy instruction (see Chapter 1), a first-year
how program goals provide a framework for course introduction to library and learning resources
goals. It presents general education as a program,
presented by a librarian, required assignments in a
but one could well imagine the institutional goals
general education course, and/or a series of
cited here as general education goals instead.
substantial research papers required in the major.
Notice also that some of the goals for programs
The goals for student learning in each of these three
situations, when combined, may fulfill the
12
Figure 1
Relationship between Institutional and Program Goals
Note: Not all programs are designed meet all institutional goals. Some cells are left blank for
illustrative purposes only, not to imply that these goals cannot be met by the programs used
in this example.
Programs and Program-level Goals
Institutionallevel Goals
Leadership
Global Citizens
General
Education
Function
effectively as a
team member to
produce a
scholarly product
Recognize and
appreciate artistic
and literary
contributions of
diverse cultures
Residence
Life
Business
Develop
leadership skills
Apply conflict
resolution skills in
a living-learning
environment
Develop an
appreciation for
cultural and ethnic
diversity
Exhibit engaged
citizenry and value
community service
Technologically
Sophisticated
Individuals
Effective
Communicators
Critical
Thinkers
Use technology
effectively to
communicate and
analyze
information
Write and speak
proficiently
Develop
leadership
potential in self
and others
Value and exhibit
comfort with
cultural
differences in
business practices
Function
effectively as a
team member to
run a small
business
Use technology
effectively to
communicate and
analyze
information
related to business
Communicate
effectively in
social situations
Distinguish critical
from non-critical
information
Use critical
thinking to analyze
business case
studies
History
Chemistry
Analyze historical
perspectives on
individual and
political leadership
Recognize and
value
culturally-diverse
historical
perspectives
Demonstrate an
ability to work as a
team with a
diverse group of
students
Use technology
effectively to
communicate and
analyze
information
related to history
Use technology
effectively to
collect, analyze,
and display data
Communicate
effectively, orally,
and in writing
about historical
topics
Cogently present
research data and
analyses in
written, visual, and
oral formats
Critically analyze
historical events
and trends using
scholarly
techniques
Apply critical
thinking skills to
design an
experiment that
tests an hypothesis
Collect, analyze,
and interpret data
relevant to testing
an hypothesis
13
Figure 2
Relationship between Program and Course Goals
Program
General
Education
Residence Life
Program Goal
Recognize and
appreciate artistic and
literary contributions of
diverse cultures
Apply conflict
resolution skills in a
living-learning
environment
Business
Administration
Function effectively as
a team member to run a
small business
History
Communicate orally
and in writing about
historical topics
Chemistry
Collect, analyze, and
interpret data relevant
to testing an hypothesis
Course or Activity
Course Goal
World Art History
Identify and analyze major works representing several
different cultures.
Caribbean Literature
Demonstrate familiarity with themes and genres of
classic and contemporary Caribbean literature.
First Year Student
Orientation Program
Work effectively as part of a group to analyze and
resolve a hypothetical interpersonal conflict.
Seminar led by resident
assistants
Develop a plan, in cooperation with floor-mates,
for handling conflicts as they arise.
Introduction to Marketing
Develop a feasible marketing plan for a small business.
Capstone in Business
Administration
Work with a team of students to develop, plan, manage,
and market a small business.
Modern American
History
Present a cogent oral analysis of one long-term effect of
the Cold War.
Medieval History
Present a cogently-written, critical analysis of gender
and class roles in Medieval England.
Introductory Chemistry
Laboratory
Replicate chemical reactions using appropriate
laboratory techniques.
Introductory
Biochemistry
Collect, analyze, and interpret data relevant to an
hypothesis supplied by the instructor.
14
institutional goal. Thus, an institution need not
articulate specific means of achieving a particular
goal at the institutional level, or assess it at that
level, if it has chosen to assess it at the course and
program levels.
Learning goals are discussed throughout the
remaining three chapters of this handbook.
Chapter 3 discusses the means by which the
attainment of goals is assessed. Chapter 4 describes
Middle States expectations for asssessing student
learning and institutional effectiveness. Chapter 5
focuses on using assessment results and how
information about whether or not goals have been
attained can be used to adapt curricula and
programs.
Angelo, Ewell, and Lopez (2001) recommend that
institutions begin assessment by “rounding up
information you already have.”
Institutional Level. At the institutional level, an
audit may be accomplished easily by cataloging the
means used to assess the entire student body
through the activities of offices of institutional
research, student affairs, career services, the library,
and information management. Most institutions
have existing information from some or all of the
following:
q Surveys of student satisfaction and
engagement that are designed and
administered nationally and locally
q Alumni career and satisfaction surveys
q Tests: standardized and/or locally-created
First Steps Towards Developing
Learning Goals
The process of developing learning goals should
begin with a “situation audit” or inventory of what
exists and which practices have been successful.
Practices that are identified will provide
information for developing a plan for the
assessment of student learning, establishing goals,
and identifying assessment measures.
The Situation Audit: Taking an Inventory
and Starting with What Already Exists1
A basic tenet for evaluating student learning is to
begin with successful assessment activities already
in place. Whether the objective is to develop
learning goals and assessment techniques for an
individual course, an entire program, or the
institution as a whole, an inventory of what exists
provides a strong foundation for later success.
q Statistics, such as placement and retention
rates
q Program reviews of both academic and
support programs
q Reports by instructional librarians on
information literacy and collaboration with
faculty members
Program Level. At many institutions, each
department and program institutes evaluations of its
students that are independent from those of other
departments and programs. The choice of
instruments and assessment activities is often
idiosyncratic, grounded in the approach that is
typical of each discipline. A comprehensive and
well-designed institution-wide checklist of possible
types of assessment activities may help each
department to create an accurate description of its
assessment activities.
At the program level, the checklist for assessment
An excellent method of gauging the level of an
institution’s existing evaluation of student learning activities might include:
is to survey the assessment practices embedded at
q Senior capstone theses, papers, individual or
the course, program, and institutional levels.
group projects, and performances or other
Peter Ewell has referred to this as a “situation
presentations
audit”—i.e., an inventory of information already on
q Student portfolios
hand that may provide evidence of student learning.
1 This topic is placed here in the section on goals so that it appears early in the book, and it is referenced again in later
chapters. Its placement also emphasizes the point that institutions should examine existing learning goals and develop new
ones before making decisions about adopting previously-used measures.
15
q Student research participation
qualitative and quantitative information over
time and across situations.
q Departmental student and alumni surveys
A more thorough discussion of course-embedded
assessment techniques is presented in Chapter 3 of
this handbook. It describes the relative uses of
quantitative and qualitative information, as well as
direct and indirect methods of evaluating student
learning.
q Standardized tests of subject area or broad
skills
q Reports from student internship supervisors
Additional assessment activities may be suggested
by disciplinary accreditors who issue guidelines
and standards for intended student learning,
required or suggested educational experiences,
recommended evaluation methods, and
expectations for the use
of results.
Examining Existing Practices for Success
Angelo, Ewell, and Lopez (2001) advocate building
assessment plans and practices from those activities
on campus that are already successful. When a
“situation audit” of course-based, programmatic
and/or institution-wide assessment is complete,
local best practices will surface as models for
additional assessment initiatives.
A survey designed to document assessment
practices at the department or program level can
assist departments in identifying where there are
gaps in the learning goals they are assessing,
duplicative teaching efforts, and the usefulness of
existing assessment results.
Such an inventory also can offer departments a
basis for comparing themselves with other
departments, as well as creating an institution-wide
resource of where to find advice about instituting
assessment on their own campuses. Appendix 5 is
an example of such a survey.
Course Level. The commitment of individual
faculty and teams of faculty is essential. Reviewing
existing course-based assessment practices can help
faculty members to reflect on assessment practices
that have become routine. A review of course
materials can provide useful insights into what
students may or may not be learning.
A well-constructed course-level checklist might
include:
q Embedded assessment elements faculty
prepare, such as syllabi, curricula,
instructional materials and methods,
assignments, exams, and quizzes
q Direct evidence of student learning and
development, such as student products and
performances resulting from embedded
assignments, tests, and other educational
experiences
Faculty members and students probably already
have a good sense of what is working best on a
campus. For example, there may be anecdotal
evidence that graduates of one program have
particularly strong research skills, while students in
another program may be especially adept at using
and adapting what they have learned to solve
unforeseen problems while working as interns.
An audit of teaching and assessment practices used
by successful programs will produce models for
other departments.
Ideally, some of the faculty members or
departments that have been evaluating student
learning will have used the results of the evaluation
to change practices and to enhance student learning.
These efforts also can motivate and guide others in
the institution.
Data collected from a comprehensive audit can be
used to answer critical questions about which
existing assessment practices on campus can form
the core of the institution’s assessment program and
to identify the most critical gaps for which new
assessment techniques are needed. Perhaps the most
important benefit of conducting a situation audit is
that the data gathered become a foundation for
developing learning goals.
q Indirect indicators such as surveys,
placement, and other institutional research
data. These indicators can provide both
16
Starting with Successful Programs
Identifying successful programs and courses
early can help later when organized assessment
is started. Starting with the assessment of
successful programs offers several benefits:
Ø
Ø
Ø
Ø
characterize students after they have completed a
course or program, or after they have graduated
from the college or the university.
The data generated from actual tests, surveys, or
instruments used to gauge the outcome of the
educational experience are the actual assessments2.
For example, the learning goal might be to develop
analytical skill. After a student has taken a course
Effective teaching/learning efforts of faculty
intended to promote this skill, he or she should
members and students are validated.
have better analytical skill. This abstraction,
The admissions, public relations, and
analytical skill—the quality or attribute that
development offices have substantive research
students should possess after taking the course—is
information to use when publicizing the
a generalized notion of what should be achieved.
institution and its programs.
To evaluate the achievement of analytical skill, a
External stakeholders have concrete, rather than test of learning might include problems that can be
anecdotal, evidence of the quality of the
solved with syllogisms. The syllogistic problems
institution and its programs.
“operationally define”—i.e., make concrete the
Faculty members and administrators in other
abstraction “analytical skill.” Thus, success in
programs can learn from the successes of their
solving the problems (as indicated by scores above
colleagues.
the norm, scores that surpass pretest scores, or
scores that differ from those of similar students who
did not take the same course) would indicate
success in acquiring analytical skill.
Defining Learning Goals before
Selecting Assessment Methods
The most important step in developing successful
methods for evaluating student learning is to
develop meaningful, clear, and realistic goals for
student learning at the course, program, or
institutional level. These goals or statements of
expected student learning are different from the
actual evidence or the data gleaned from
evaluations of student learning. Goals are the basis
for determining how best to collect, assess, and
interpret the data in order to improve. Data
collection not tailored to goals will not provide
information about the achievement of desired
student learning, nor will it lead to new approaches
to teaching and learning.
The goals or statements of student learning are
hypotheses for what qualities or attributes will
Another way to move from the abstract to the
specific when articulating student learning goals is
to state those goals in terms of what, specifically, a
student should be able to do in order to demonstrate
that desired learning has occurred. In other words,
what observable student behaviors should result
from a learning experience? For example,
astronomy faculty members may agree that their
students will understand basic concepts about the
solar system, but they may have differing opinions
about what constitutes “basic concepts” and what it
means to “understand” them. Do “basic concepts”
refer to basic facts about each planet or also to
theories about how the solar system was created?
Should students memorize those basic facts, or
should they be able to use information about our
solar system to speculate about the characteristics
of other solar systems?
It is important, therefore, to understand that the
qualities or attributes that students should exhibit
after a learning experience should be operationally
2 In research terms, educational experiences are the “independent variable or treatment,” the assessments are the methods, and
their results would be called the “dependent variable.” The student learning outcomes, then, depend upon the educational
experiences.
17
defined in order to be assessed meaningfully.
For example, a statement of student learning (a
learning goal) might be that a student will think
critically after completing an introductory course in
philosophy. Another learning goal might be that,
after completing a service-learning course, a
student have greater appreciation for others who are
different. Each of these goals can be operationally
defined, and then learning can be documented by a
test or other instrument created to assess the
specific goal. The results of the assessment
demonstrate (or do not) the outcome one would
expect to see—i.e., What would a student’s
performance on this particular assessment look like
if he or she is a critical thinker? What would the
student’s performance look like if he or she is a
person with appreciation for differences between
people?
Appendix 6 is a worksheet for an exercise that
faculty and staff members can use to begin to
develop learning goals for courses and programs
and to begin to think about how those goals might
be achieved, how they might be assessed, and how
a course or program might be altered to ensure
greater student learning. The worksheet contains
space for only three goals, in order to emphasize
that the focus should be on important goals.
The remainder of this chapter 2 is devoted to
developing learning goals; Chapter 3
is devoted to evaluating those goals.
Ensuring the Quality and Relevance of
Learning Goal Statements
with tasks, provide too much information, and
dilute the focus on areas that need the most
attention.
The departmental, school, or institutional mission
statement, as described in Characteristics
(Standard 1), should provide the basis for
determining the most important goals at each level.
It is useful to concentrate statements of expected
learning outcomes by asking, “What are the most
important learning outcomes we seek for our
students in the context of the goals of our
institution/program?” For example, the programs
and learning outcomes of an institution whose
mission includes giving each student a strong
spiritual grounding may emphasize different
learning outcomes from those of an institution
whose mission includes teaching its students
technical skills.
Widely Agreed-upon Concepts
Statements of expected learning outcomes will not
be effective unless they are developed
collaboratively and widely accepted by
stakeholders: faculty members, students,
employers, alumni, and others affected by or
concerned with the program or institution. While it
is unlikely that there will be unanimous agreement
on expected learning outcomes, there should be a
shared sense among most members regarding which
learning is most important. The mission of the
institution and the subsidiary missions of
departments and programs serve as the natural
sources for shared expectations.
Communication of Learning Goals
The institution can ensure the quality and relevance
of learning goal statements by focusing on those
that are most important, widely accepted by the
various stakeholders, meaningful, sufficiently
explicit, and interconnected among the various
academic levels and curricula within the institution.
Key Learning Outcomes
Effective statements of expected student learning
are focused on the most important goals of a
course, program, or institution. They are not a
collective list of goals that are idiosyncratic to a
few faculty or staff members. Attempts to evaluate
every possible goal can overwhelm an institution
If the institutional community shares learning goals
and if they are expressed clearly, then the resulting
statements of expected learning outcomes can be
used by the entire campus community.
Clearly-expressed expectations for the learning
outcomes of courses and programs can help
students to focus their studies and, as a result,
to learn more effectively. Prospective students who
are aware of the specific types of expected learning
outcomes of a program to which they are
considering applying can make a better-informed
decision about whether the program meets their
18
needs, especially when evidence is available that
those goals actually are achieved.
Faculty members who teach prerequisite courses or
“service” courses can prepare students better for
later courses and programs if they are familiar with
the expected learning outcomes of subsequent
courses or courses in the target program.
For example, faculty members in the English
department who are familiar with the expected
learning outcomes of the theater department’s
programs and courses can better meet the needs of
theater students taking literature courses, and
physics faculty members can meet the needs of
engineering students.
Sufficiently Explicit Learning Goals
Although it is not helpful for statements of student
learning goals to be so specific that they focus on
unimportant or trivial outcomes, it is important for
statements to be sufficiently explicit for all
stakeholders to have a common understanding of
their meaning.
For instance, one goal for an undergraduate
psychology program might be for students to
exhibit proficiency in conducting research.
While faculty members may implicitly understand
what this goal might mean, increasing the
specificity of the goal would enhance its clarity and
allow for more direct assessment of the attainment
of the goal. For example, a statement of the goal
Meaningful Learning Goal Statements
might read: “Students will learn the statistical,
That Lead to Improvement
organizational, writing, and analytical skills
Meaningful statements of student learning goals
necessary to conduct meaningful and valid
address learning as a multidimensional and
scientific research.” Statements then could describe
integrated process, occurring over time. They do
the evidence needed to demonstrate that students
not focus on trivial learning outcomes. Stated
have achieved the kowledge and abilities related to
cogently and clearly, meaningful learning goals will each of these components.
lead to the improvement of teaching and learning at
the course, program, and institutional levels. The
Interconnectedness of Student Learning
importance of each learning goal should be obvious Goals
to students, faculty, and prospective employers.
Student learning can occur at many levels and in
Meaningful learning goals stress generalizable and many venues:
higher-order thinking skills rather than
q Course, program, and institutional levels
memorization of facts or very simple conceptual
(Standard 14);
understanding. For example, a goal to identify
grammatical forms (past participles, etc.) is, in most
q Undergraduate, graduate, and professional
cases, not as meaningful as a goal of being able to
program levels (Standard 11);
write and speak grammatically. Similarly, the
q General education curricula (Standard 12);
successful memorization of important historical
dates is not as meaningful as a goal for students to
q Related educational activities, such as basic
be able to place historical events within a social and
skills, certificate programs, experiential
political context, to draw meaningful comparisons
learning, non-credit offerings, and distance
between events, and to analyze current events
or distributed learning (Standard 13); and
within an historical framework. For both of these
q In co-curricular and extracurricular activities
examples of more meaningful or higher-order
(Standard 9).
goals, the more trivial goals of memorizing dates
An institution’s curriculum may address particular
and acquiring the names of parts of speech
probably will be achieved naturally in the course of learning outcomes in different complementary or
overlapping courses and programs. Statements of
achieving the larger goal.
learning outcomes for courses or programs should
recognize and clarify these relationships, and
student learning outcomes assessment plans should
be structured to avoid duplication.
19
Choosing Learning Goals
Start with success
Determine which learning goals are
already being assessed and what data may
be available to assess other goals
Ø Institutional Level
Ø Program Level
Ø Course Level
Ensure relevance of goals
Ø Identify key learning outcomes
Ø Use widely agreed-upon concepts
Ø Communicate goals
Ø Select important and meaningful goals
Ø Be explicit
Ø Integrate goals in different areas
and levels
Choose goals that can lead to
improvement
Emphasize higher-order thinking skills
Define learning goals before choosing
assessment methods
Ø Operationally define each goal
Ø Tailor data collection to defined goals
Resources for Creating Student
Learning Goal Statements
This section includes a discussion of several
specific resources for crafting actual statements of
student learning goals. Each of these resources
presents opportunities both for “brainstorming” and
for comprehensively and systematically reviewing
sets of possible goals.
Existing Learning Goals
An institution already may have explicit and clear
statements of student learning goals in place, and it
is important to resist the urge to redesign entire sets
of course, program, or institutional goals if they
already exist. The focus should be on developing
those that are missing, those that are unclear, those
that have changed, or those that are complementary.
For instance, many institutions developed learning
goals for general education programs when those
programs were initially created, and some may have
been substantially revised during subsequent
program review. However, faculty members may
determine that no measures are being used to assess
whether the goals have been achieved. The goals,
then, may not need to be re-written; instead, they
should be evaluated for their importance and
relevance, and they should be supplemented with
additional goals where appropriate. Of course, it is
still important to measure student outcomes in
these areas.
Existing Syllabi and Course Descriptions
Existing syllabi and catalogue descriptions provide
a logical starting point for developing learning
goals at the course level, because many faculty
members already include learning goals in their
syllabi, and many course descriptions include
statements about course content, goals, and/or what
the student should be able to do once the course is
completed. Existing goals such as these can be
reviewed for their relevance to programmatic
mission, and faculty members should be
encouraged to think about whether these goals
reflect the current or changing focus of their
disciplines. Examples of syllabi that already
address learning goals can serve as resources for
faculty members who have not previously
20
developed goals, and faculty members familiar with Analysis of Student Work
the process can serve as mentors.
The work products and performances of students
that are the result of existing assignments and tests
Leading Questions
may already embody the important learning
Leading questions also can serve as the basis for a
outcomes, even though the faculty member may not
brainstorming exercise in developing learning
have explicitly conceived or stated those goals.
goals.
A retrospective review can suggest how the original
The leading questions listed in Figure 3 can be
statements of goals might be revised, especially if
tailored to apply to any discipline and can be
the unstated learning goals are important and if the
refined to address more specific outcomes. In
evaluation methods are valid.
general, they focus attention on the most important
For instance, a political history course may not
learning goals for individual programs.
have the explicit goal of increasing awareness of
The following additional questions related to basic and participation in current political events.
improvement are adapted from the work of
Nevertheless, after taking the course, students may
Stufflebeam (2001):
report increased civic awareness, an increased
tendency to vote, and increased participation in
q What alternative educational practices and
experiences are available, and what are their local political activity. Therefore, it may make
sense to make the development of political
comparative advantages over current
awareness an explicit course goal and to revise the
practices at our institution?
course accordingly.
q What are the characteristics of a good
syllabus that can serve as a guide to
Inventory of Teaching Goals
teaching and learning?
Faculty members and students can use the Teaching
q What facilities, materials, and equipment are Goals Inventory (Angelo and Cross, 1993), shown
needed to ensure success in reaching our
in Figure 4, to identify the priority of various
educational objectives?
learning goals in courses and programs. For
example, individuals or groups of faculty members
q What are the roles of faculty members,
could use the inventory to understand better their
students, and others in the pursuit of
expectations for a single course, a course with
learning?
many sections, or a series of courses. They can use
q Is the course providing learning experiences the inventory to review faculty documents, to
for all of the students who are enrolled?
examine existing disciplinary accreditation
guidelines and standards, and
q Is a particular course worth the required
to analyze direct evidence of student learning and
institutional investment?
development. Students also could complete the
q Is the course meeting the minimum
inventory so that the institution can determine
accreditation requirements for the
whether students and faculty share the same
discipline?
perceptions about the relative priority of different
types of learning.
q Is the course equal to or better than
analogous courses at comparable
institutions?
21
Figure 3
Leading Questions for Developing Learning Goals
Questions for Faculty
q In general, what are the most important things a student gains from your field of study?
q What qualities and capabilities do you strive to foster in your students?
q What is the most important knowledge that your students acquire from your field of study or from
working with you?
q How does your field of study or your work change the way students view themselves?
q In what ways does your field of study or what you do contribute to a student’s well being?
q How does your field or what you do change the way a student looks at the world?
q What does your field of study or what you do contribute to the well being of society at large?
q How do people in this area of study differ from those in other areas (knowledge, skills, and/or
values)?
q How do we know the extent to which students are learning what we hope from our field of study?
q How do we use information about student learning and development to enhance student learning?
Questions for Students
q What is the most important knowledge you have gained from taking courses, minoring, or majoring
in this subject?
q What are the most valuable skills or abilities that have you developed as a result of taking courses,
minoring, or majoring in this subject?
q How has taking courses, minoring, or majoring in this subject changed the way you look at yourself?
q How has taking courses, minoring, or majoring in this subject changed the way you look at the
world?
q How has taking courses, minoring, or majoring in this subject changed the way you think about the
future?
q How do you know whether these changes have occurred?
q How do people in this area of study differ from those in other areas (knowledge, skills, and/or
values)?
q What changes might be made in course and programs of your major or minor to enhance student
learning?
Based on leading questions developed by Prof. C. Ewart, Department of Psychology, Syracuse University,
1998. Reproduced with permission.
22
Figure 4
Teaching Goals Inventory
Self-Scorable Version
Purpose
The Teaching Goals Inventory (TGI) is a self-assessment of instructional goals.
Its purpose is three-fold: (1) To help college teachers become more aware of what they want to
accomplish in individual courses; (2) To help faculty locate Classroom Assessment Techniques
they can adapt and use to assess how well they are achieving their teaching and learning goals;
and, (3) To provide a starting point for discussions of teaching and learning goals among
colleagues.
Directions
Please select ONE course you are currently teaching. Respond to each item on the Inventory in
relation to that particular course. (Your responses might be quite different if you were asked about
your overall teaching and learning goals, for example, or the appropriate instructional goals for
your discipline.)
Just to remind yourself, please print the title of the specific course you are focusing on below:
_____________________________________________________________________________
Please rate the importance of each of the 52 goals listed below to the specific course you have
selected. Assess each goal in terms of what you deliberately aim to have your students accomplish,
rather than in terms of the goal’s general worthiness or overall importance to your institution’s
mission. There are no “right” or “wrong” answers; only personally accurate or inaccurate ones.
For each goal, circle only one response on the 1 to 5 rating scale. You may find it helpful to
quickly read through all 52 goals before rating their relative importance.
In relation to the course you are focusing on, indicate whether each goal rated is:
(5) Essential
A goal you always/nearly always try to achieve (76% to 100% of the time)
(4) Very Important
A goal you very often try to achieve (51% to 75% of the time)
(3) Important
A goal you sometimes try to achieve (26% to 50% of the time)
(2) Unimportant
A goal you rarely try to achieve (1% to 25% of the time) or
(1) Not Applicable
A goal you never try to achieve.
Please note: This Inventory was developed with support from The Pew Charitable Trusts and the Ford Foundation by
K. P. Cross & T. A. Angelo, U. C. Berkeley School of Education, 1992. Reproduced with permission of the authors.
23
Ess
ent
ia l
Ver
y Im
po r
Im
ta n
po r
t
ta n
t
Un
imp
ort
a nt
No
tA
ppl
ica
b le
Rate the importance of each goal below in terms of
what you aim to have students accomplish in your
course.
1. Develop ability to apply principles and generalizations
already learned to new problems and situations
5
4
3
2
1
2. Develop analytic skills
5
4
3
2
1
3. Develop problem-solving skills
5
4
3
2
1
4. Develop ability to draw reasonable inferences
from observations
5
4
3
2
1
5. Develop ability to synthesize and integrate
information and ideas
5
4
3
2
1
6. Develop ability to think holistically:
to see the whole as well as the parts
5
4
3
2
1
7. Develop ability to think creatively
5
4
3
2
1
8. Develop ability to distinguish between fact and opinion
5 4
3 2
1
_____________________________________________________________________________
9. Improve skill at paying attention
5
4
3
2
1
10. Develop ability to concentrate
5
4
3
2
1
11. Improve memory skills
5
4
3
2
1
12. Improve listening skills
5
4
3
2
1
13. Improve speaking skills
5
4
3
2
1
14. Improve reading skills
5
4
3
2
1
15. Improve writing skills
5
4
3
2
1
16. Develop appropriate study skills, strategies, and habits
5
4
3
2
1
17. Improve mathematical skills
5
4
3
2
1
_____________________________________________________________________________
18. Learn terms and facts of this subject
5
4
3
2
1
19. Learn concepts and theories in this subject
5
4
3
2
1
20. Develop skill in using materials, tools, and/or
technology central to this subject
5
4
3
2
1
21. Learn to understand perspectives and values of this subject
5
4
3
2
1
22. Prepare for transfer or graduate study
5
4
3
2
1
K. P. Cross & T. A. Angelo, U.C. Berkeley School of Education, 1992
24
Ess
ent
ia l
Ver
y Im
po r
Im
ta n
po r
t
ta n
t
Un
imp
ort
a nt
No
tA
ppl
ica
b le
Rate the importance of each goal below in terms of
what you aim to have students accomplish in your
course.
23. Learn techniques and methods used to gain
new knowledge in this subject
5
4
3
2
1
24. Learn to evaluate methods and materials in this subject
5
4
3
2
1
25. Learn to appreciate important contributions to this subject
5 4
3 2
1
_____________________________________________________________________________
26. Develop an appreciation of the liberal arts and sciences
5
4
3
2
1
27. Develop an openness to new ideas
5
4
3
2
1
28. Develop an informed concern about contemporary
social issues
5
4
3
2
1
29. Develop a commitment to exercise the rights
and responsibilities of citizenship
5
4
3
2
1
30. Develop a lifelong love of learning
5
4
3
2
1
31. Develop aesthetic appreciations
5
4
3
2
1
32. Develop an informed historical perspective
5
4
3
2
1
33. Develop an informed understanding of the
role of science and technology
5
4
3
2
1
34. Develop an informed appreciation of other cultures
5
4
3
2
1
35. Develop capacity to make informed ethical choices
5
4
3
2
1
_____________________________________________________________________________
36. Develop ability to work productively with others
5
4
3
2
1
37. Develop management skills
5
4
3
2
1
38. Develop leadership skills
5
4
3
2
1
39. Develop a commitment to accurate work
5
4
3
2
1
40. Improve ability to follow directions, instructions, and plans
5
4
3
2
1
41. Improve ability to organize and use time effectively
5
4
3
2
1
42. Develop a commitment to personal achievement
5
4
3
2
1
43. Develop ability to perform skillfully
5
4
3
2
1
_____________________________________________________________________________
K. P. Cross & T. A. Angelo, U.C. Berkeley School of Education, 1992
25
Rate the importance of each goal below in terms of
what you aim to have students accomplish in your
course.
Ess
ent
ia l
Ver
y Im
po r
Im
ta n
po r
t
ta n
t
Un
imp
ort
a nt
No
tA
ppl
ica
b le
44. Cultivate a sense of responsibility for one’s own behavior
5
4
3
2
1
45. Improve self-esteem/self-confidence
5
4
3
2
1
46. Develop a commitment to one’s own values
5
4
3
2
1
47. Develop respect for others
5
4
3
2
1
48. Cultivate emotional health and well-being
5
4
3
2
1
49. Cultivate physical health and well-being
5
4
3
2
1
50. Cultivate an active commitment to honesty
5
4
3
2
1
51. Develop capacity to think for one’s self
5
4
3
2
1
52. Develop capacity to make wise decisions
5
4
3
2
1
_____________________________________________________________________________
Self-Scoring Worksheet
1. In all, how many of the 52 goals did you rate as “Essential”?
____
2. How many “Essential” goals did you identify in each of the six clusters listed below?
Goals
Total number of
Clusters Ranked
included “Essential” goals (1st to 6th) by number of
in cluster in each cluster
“Essential” goals included
Cluster Number
and Name
I. Higher-Order
Thinking Skills
1-8
______
______
II. Basic Academic Success Skills
9 - 17
______
______
III. Discipline-Specific
Knowledge & Skills
18-25
______
______
IV. Liberal Arts & Academic Values
26-35
______
______
V. Work and Career Preparation
36-43
______
______
VI. Personal Development
44-52
______
______
K. P. Cross & T. A. Angelo, U.C. Berkeley School of Education, 1992
26
3
Evaluating Student Learning
T
here are many ways to approach the
evaluation of student learning. The
characteristics of good evidence of student
learning include considerations of direct and
indirect methods for gathering evidence of student
learning, the appropriate use of quantitative
and qualitative evidence, and other methodological
considerations. First, however, it is important to
understand the fundamental assessment concepts of
formative and summative assessment and
benchmarking.
Formative and Summative Assessment
Formative assessment is ongoing assessment that is
intended to improve an individual student’s
performance, student learning outcomes at the
course or program level, or overall institutional
effectiveness. By its nature, formative assessment
is used internally, primarily by those responsible
for teaching a course or developing a program.
Ideally, formative assessment allows a professor,
professional staff member, or program director to
act quickly to adjust the contents or approach of a
course or program. For instance, a faculty member
might revise his or her next unit after reviewing
students’ performance on an examination at the end
of the first unit, rather than simply forging ahead
with the pre-designated contents of the course.
Many instructors also solicit repeated brief
evaluations of their teaching, and the data gleaned
from these can be used to make adjustments that
may improve learning, such as the introduction of
more discussion into a class.
In contrast, summative assessment occurs at the end
of a unit, course, or program. The purposes of this
type of assessment are to determine whether or not
overall goals have been achieved and to provide
information on performance for an individual
student or statistics about a course or program for
internal or external accountability purposes. Grades
are the most common form of summative
assessment.
Goals for student learning will be expressed
summatively when faculty members are describing
what they expect students to be able to do or what
skills they expect students to achieve when they
complete a course or a program or when they
graduate from the institution.
Formative and summative assessment work
together to improve learning. They should be
central components of assessment at the course
level, and where appropriate, at the program level.
Benchmarking
The term benchmarking is now common in
assessment plans and conversations about
assessment. Originally, benchmarking was a term
used in the corporate environment to define a set of
external standards against which an organization
could measure itself. The organization identifies
comparable, peer, or “reach” organizations and
systematically compares its practices or
achievements against those of the other
organization.
In higher education settings, a university might use
benchmarking techniques to define its comparison
group—its peer institutions—and to compare its
27
own outcomes to theirs. This benchmarking could
be based, for example, on retention rates, five-year
graduation rates, admissions yield data (the number
of enrollees as a function of the number of students
accepted), employment and graduate school
placement rates, and performance on national or
professional examinations. Theoretically, any
outcome for which there are data from peer
institutions and programs can be compared in a
benchmarking study.
Direct and Indirect Methods for
Assessing Student Learning
The concepts of direct and indirect methods of
evaluating student learning are often confused with
each other and with quantitative and qualitative
forms of information. Each of these has its merits
and drawbacks.
Direct and indirect methods of evaluating learning
relate to whether or not the method provides
Two other related forms of benchmarking are used
evidence in the form of student products or
in higher education settings. A college or university
performances. Such evidence demonstrates that
can compare itself to a national norm by reviewing
actual learning has occurred relating to a specific
the data from a published test or survey such as the
content or skill. Indirect methods reveal
National Survey of Student Engagement (NSSE).
characteristics associated with learning, but they
Alternatively or in addition, an institution can set
only imply that learning has occurred. These
for itself the goals or benchmarks that it hopes to
characteristics may relate to the student, such as
achieve within a specified time period (e.g., to
perceptions of student learning, or they may relate
increase job placement rates from 70% to 90% in
to the institution, such as graduation rates.
five years).
When a student completes a calculus problem
The benefit of inter-institutional comparison is that
correctly and shows her work, learning is
it can flag problem areas to investigate the causes
demonstrated directly. When the same student
of results that differ from the norm. For example,
describes her own calculus abilities as excellent,
two comparable liberal arts colleges with similar
she is demonstrating indirectly that she has learned
selectivity, similar student preparedness, similar
calculus. Both of these pieces of information about
socioeconomic profiles for their students, and
the student’s performance are important. For
similar science curricula, may discover that
excample, a student’s perception that she is doing
proportionately more students are accepted to
poorly in calculus when she is actually doing well
medical schools from one institution than from
would provide important information to both the
another. Further investigation may reveal that the
student and the professor. However, indirect
excelling college requires a hospital internship for
evidence—in this case, a perception—is less
all of its pre-med students.
meaningful without the associated direct and
tangible evidence of learning.
The discovery that an institution’s students are
below the norm on a national metric (e.g., amount
Figure 5 includes examples of direct and indirect
of time devoted to school work outside the
measures of student learning at the course,
classroom) challenges the institution to determine
program, and institutional levels. Many of the
the reason for this result. Similarly, an institution
examples presented in Figure 5 can be used as
that sets its own internal benchmarks must design
measures of student learning at more than one level.
and implement a program to achieve its goals.
For example, portfolios of student work and student
satisfaction surveys can be used at the course,
Before beginning to articulate goals for student
program, and institutional level, and internship
learning, program faculty and leaders of
performance ratings could be used at the course or
institutional assessment should consider how the
use of benchmarks could enhance their institution’s program level.
ability to achieve its goals and whether useful
measures from comparable peer institutions are
available.
28
Figure 5
Examples of Direct and Indirect Measures of Student
Learning (Course, Program, and Institutional Levels)
Direct Measures
¬ Course and homework assignments
¬ Course evaluations
¬ Examinations and quizzes
¬ Test blueprints (outlines of the concepts and
skills covered on tests)
¬ Standardized tests
¬ Percent of class time spent in active learning
¬ Term papers and reports
Course
Indirect Measures
¬ Observations of field work, internship
performance, service learning, or clinical
experiences
¬ Research projects
¬ Number of student hours spent on service
learning
¬ Number of student hours spent on homework
¬ Number of student hours spent at intellectual or
cultural activities related to the course
¬ Class discussion participation
¬ Grades that are not based on explicit criteria
related to clear learning goals
¬ Case study analysis
¬ Rubric (a criterion-based rating scale) scores for
writing, oral presentations, and performances
¬ Artistic performances and products
¬ Grades that are based on explicit criteria related
to clear learning goals
Program
¬ Capstone projects, senior theses, exhibits, or
performances
¬ Focus group interviews with students, faculty
members, or employers
¬ Pass rates or scores on licensure, certification,
or subject area tests
¬ Registration or course enrollment information
¬ Student publications or conference
presentations
¬ Employer and internship supervisor ratings of
students’ performance
¬ Department or program review data
¬ Job placement
¬ Employer or alumni surveys
¬ Student perception surveys
¬ Proportion of upper-level courses compared to
the same program at other institutions
¬ Graduate school placement rates
¬ Performance on tests of writing, critical
thinking, or general knowledge
Institutional
¬ Rubric (criterion-based rating scale) scores for
class assignments in General Education,
interdisciplinary core courses, or other courses
required of all students
¬ Performance on achievement tests
¬ Explicit self-reflections on what students have
learned related to institutional programs such as
service learning (e.g., asking students to name
the three most important things they have
learned in a program).
29
¬ Locally-developed, commercial, or national
surveys of student perceptions or self-report of
activities (e.g., National Survey of Student
Engagement)
¬ Transcript studies that examine patterns and
trends of course selection and grading
¬ Annual reports including institutional
benchmarks, such as graduation and retention
rates, grade point averages of graduates, etc.
Direct Methods
Direct methods of evaluating student learning are
those that provide evidence of whether or not a
student has command of a specific subject or
content area, can perform a certain task, exhibits a
particular skill, demonstrates a certain quality in his
or her work (e.g., creativity, analysis, synthesis, or
objectivity), or holds a particular value. Direct
methods can be used at the course level, the
program level, and, theoretically, at the institutional
level.
Course Level. Most familiar are direct evaluations
of learning that are applied at the course level.
Examinations,3 regardless of format, are designed
to be direct evaluations of student learning.
Similarly, evaluations of writing samples,
presentations, artistic performances, and exhibits
provide direct evidence of student learning, as do
evaluations of student performance in internships,
research projects, field work, or service learning
settings. As discussed later, grading linked to clear
learning goals is a valid and useful form of direct
measurement of student learning.
Program Level. At the program level,
examinations also are used frequently as direct
measures of student learning. Such examinations
would be more comprehensive than those
embedded within a course and would be designed
to evaluate cumulative, aggregate, or holistic
learning after the conclusion of a program or during
the course of the program.
For example, a writing examination might be given
after the first two years of a general education
program, with the goal of determining whether
students’ writing was enhanced as a function of the
program. Standardized tests of disciplinary content
might be administered to students after they have
completed all program requirements for the major
(e.g., American Chemical Society examinations).
Honors theses, senior theses, or senior projects are
other sources of direct evidence of student learning
within a program. Ratings by internship supervisors
of how well interns are demonstrating key learning
outcomes are important, direct program-level
evidence of student learning.
Institutional Level. Direct evaluations at the
institutional level are used less frequently and are
much more likely to take the form of an
examination. A college or university might use the
Academic Profile or the Major Field Tests from the
Educational Testing Service, the Collegiate
Assessment of Academic Proficiency from the ACT
(American College Testing) or other graduate-level
examination scores to demonstrate that learning has
occurred.
An institution may wish to demonstrate that certain
goals expressed in its mission were achieved
through exposure to the entirety of its curriculum
and co-curricular experiences. For example, it may
wish to show that regardless of program or major,
which co-curricular activities students have
participated in, and whether students were residents
or commuters, they exhibit cultural sensitivity and
global cultural and geographical awareness. It could
design an evaluation process to determine the
degree to which graduating students exhibited these
qualities (e.g., a rubric for reviewing an
examination or portfolio).
It may appear that such qualities are abstract and,
therefore, that the measurement of learning was not
direct, but in fact that is not the case. In this
example, the goal was to have students learn,
through curricular and co-curricular programs, to be
good global citizens, broadly speaking, and the
hypothetical examination was designed to measure
the degree to which this goal was achieved.
General education knowledge, competencies, and
skills gained across the curriculum might be
evaluated over the entire student experience,
whether before or after graduation.
3 For the purpose of clarity, the term “examination” is being used here to refer to what are commonly called quizzes, exams,
or tests designed to measure whether or not a student has learned something that he or she was taught prior to its
administration. The word “test” is a more generic term and can apply to any measure that may be direct or indirect, or
qualitative or quantitative.
30
that the institution’s learning outcomes are
consistent with its goals.
Fundamental Importance of Direct Forms of
Evaluation. The power of direct assessments of
student learning is that, if designed properly, they
answer the most important questions:
q What did students learn as a result of an
educational experience?
q To what degree did students learn?
q What did students not learn?
Institutional stakeholders and the public can
understand easily data gleaned from direct
evaluations of student learning. They can
understand, for instance, that students at Institution
A have higher scores on the American Chemical
Society examination than students at Institution B,
and those same data provide assurance that a certain
level of knowledge has been acquired by students at
both institutions.
Limitations and Considerations Related to
Direct Forms of Evaluation. Direct assessments,
however, do not tell the whole story of student
learning. There are two potential problems with
using only direct assessments of student learning.
The first problem relates only to direct methods,
and the second pertains to both direct and indirect
methods.
Direct assessments of student learning, while
providing evidence of what the student has learned,
provide no evidence as to why the student has
learned or why he or she has not learned. The
“why” of student learning is especially important
when students have not learned, because one of the
primary goals of assessment is to make future
learning experiences more effective.
If students perform poorly on a mathematics exam,
for instance, it is important for the instructor to
know whether the students’ performance resulted
from not having learned the material or from
having learned the material but also experiencing
anxiety during the examination. Other data are
needed
to answer this question.
It is important to consider that even direct forms of
evaluation do not necessarily indicate whether
“value-added” learning has occurred. The
Commission does not require that its member
institutions demonstrate value-added learning, only
In and of themselves, direct forms of evaluation do
not always provide evidence that the targeted
learning goal was achieved within the context of a
course, a program, or an entire college education, or
whether the demonstration of the learning goal was
influenced by or a product of prior learning or even
the result of innate abilities. If an institution or
faculty members in a program are concerned about
demonstrating that the learning occurred in a
particular context, then care should be taken to
design aspects of the assessment program to tap
“value-added” learning.
At the course level, the contrast between
value-added demonstrations of student learning and
absolute levels of student learning is rarely
meaningful. One can assume, for instance, that
knowledge of college-level organic chemistry,
elementary school teaching techniques, or
Spinoza’s philosophy was acquired within the
context of the course specifically designed to teach
that knowledge. The same reasoning applies to the
program level; students are likely to have acquired
the skills and knowledge specific to their programs
while taking courses within the program.
At the institutional level, the distinction between
student knowledge that was acquired before the
student arrived at the institution and what he or she
learned while in attending the institution may be a
more salient one. Some institutions may want to
demonstrate that the education they provide has
had a fundamental effect on students’ lives—i.e.,
changed them in a way that would not have
occurred if the student did not attend college or
attended a different type of college.
One college, for instance, may want to demonstrate
that a personal atmosphere that encourages
faculty-student mentoring relationships results in
better preparation for acceptance to graduate school
than a student might otherwise receive at a different
type of institution. Another may want to
demonstrate that it prepares its students for the real
world in a way that a different college experience
cannot. Yet another might use assessment data to
show that students have dramatically increased
their job marketability or their chances of
graduating by attending the college.
31
If institutions seek to demonstrate such
accomplishments, it is important to consider
whether the assessment design truly demonstrates
value-added learning rather than some other
phenomenon. For instance, students entering an
institution with very high SAT writing scores are
likely to write well after they have been exposed to
the college’s General Education program. In other
words, to the extent that high scores of graduating
students on tests of writing skills reflect pre-college
expertise, those scores reflect the effect(s) of one or
more “inputs” but are not necessarily value-added.
An instructor who regularly reviews his or her
teaching evaluations and who changes the course as
a result of those evaluations is engaging in
improvement based on hypotheses derived from the
indirect assessment of student learning. The same
instructor can use this indirect method in
conjunction with direct methods to improve student
learning in the course.
For example, students might use the narrative
portion of the evaluation to request more time for
class discussion and might give the professor only
moderate ratings for “engagement with the course
material.” The instructor decides to introduce more
discussion into his or her class and subsequently
students praise the use of discussion and give high
ratings for the instructor’s “engagement with course
material.” Most importantly, the instructor notices
that student grades on quizzes or exams and work
on assignments are higher in the semester after he
made the change. This simple illustration of how
indirect evidence can be used in conjunction with
direct evidence can be applied in more complicated
situations.
Value-added gains can be useful in assuring the
college community and the public that higher
education provides cognitive, affective, and social
growth beyond the levels the students had attained
when they entered college. However, devoting too
much attention to creating an assessment design
that rules out other causes for student learning can
take the focus away from the most important
question: Have students who graduate from this
college or university learned what the institution
hoped they would learn?
Indirect Methods
Indirect methods of evaluating student learning
involve data that are related to the act of learning,
such as factors that predict or mediate learning or
perceptions about learning but do not reflect
learning itself. Indirect evidence often is acquired
through the use of self-report format surveys,
questionnaires, and interviews. Indirect evidence
also is provided in the form of “demographic”
statistics about the student population of an
institution, such as overall GPA, student retention
rates, graduation rates, and job placement rates.
Qualitative information about graduates, such as
names of employers, graduate schools attended, and
alumni achievements are also common forms of
indirect evidence.
Course Level. The most familiar indirect
assessment of student learning is the course and/or
teaching evaluation given at the end of the
semester. These instruments usually have a
quantitative section in a Likert (numerically-scaled)
format, in which the student rates the quality of
teaching and of the course, as well as a narrative
section in which the student offers additional
qualitative comments.
Program Level. At the program level, student
satisfaction surveys may reveal that students want
more one-on-one contact with faculty members.
Upon reflection, faculty members may decide to
offer more independent study experiences;
consequently, scores on Graduate Record
Examination subject area exams improve (direct
evidence of student learning), as do graduate school
admission rates (indirect evidence of student
learning).
Institutional Level. Indirect means of evaluating
student learning are important at the institutional
level as well. National surveys, such as the National
Survey of Student Engagement (NSSE), provide
benchmarking opportunities for the institutions to
gauge the qualities of their student populations
relative to other institutions so that they can
determine whether changes in programming affect
students’ perceptions and behavior inside and
outside the classroom. Ultimately, such assessments
can affect performance in the classroom.
For instance, if an institution finds that its students
spend less time studying than the national average
for study time, it might introduce curricular
changes that link student evaluation (i.e., grades)
32
more directly to the amount of time studied,
perhaps by providing assignments that demand
more out-of-class time and by using class
examinations which test areas that are not learned
simply by attending class. The greater engagement
that these changes create might serve to improve
student performance on direct measures of student
learning.
Indirect evidence often focuses on the learning
process and the learning environment. Alexander
Astin’s input-environment-output assessment
model, based on research from the past several
decades (e.g., Astin, 1991; Chickering & Gamson,
1991; Pascarella & Terenzini, 1991) indicates that
students learn most effectively when, in general,
they are engaged in the learning process and they
can see a connection among course goals, course
content, and evaluation.4
As noted above, the Commission does not require
proof of value-added student learning.
Nevertheless, an institution should consider
whether value-added data are necessary to
demonstrate that it fulfills its own mission. If so,
it should ensure that data collection procedures
warrant conclusions about the effectiveness of
programs in teaching students.
Quantitative vs. Qualitative Evidence
In every example of direct and indirect assessment
cited above, the evaluation of student learning
could provide either qualitative or quantitative
information. Both qualitative and quantitative
information are valuable forms of evidence about
student outcomes.
Quantitative evidence consists of data that are
represented numerically. For instance, performance
on a test or responses to a questionnaire may be
scored so that a number represents the degree to
which an individual performed or agreed/disagreed
with a certain concept. Because quantitative data
are expressed in numbers, they can be compared
directly or subjected to statistical analysis, and they
can enable the researcher to make certain
assumptions when comparing one data point to
another. Quantitative data also may peremit one to
express numerically meaningful changes in
performance (given certain conditions). One may
Limitations and Considerations Related to
claim, for instance, that a change in a test score
Indirect Methods of Evaluation. The most
important limitation of indirect methods is that they from 50 to 60 represents a 10-point or a 20 percent
gain in an individual’s performance, expressed as a
do not evaluate student learning per se, and
therefore should not be the only means of assessing percentage of his or her original score. Quantitative
data, therefore, are valued for the ease with which
outcomes.
calculations and comparisons can be made, and for
As with direct measures of student learning, it is
the easily understandable representations of
important to consider that indirect measures do not performance that they produce.
necessarily imply that value-added learning has
Qualitative evidence typically comes in two forms.
occurred. Students who express indifference to
co-curricular activities after their first year may be The first form involves simple categorization of
individuals into discrete groups (e.g., employed or
expressing an indifference that is the result of
dissatisfaction with campus programs, or they may unemployed; participates in athletics or does not
participate in athletics). The second form of
have arrived on campus disinclined to spend time
qualitative evidence is data expressed in prose or
on anything but course-related work.
narrative. A question is asked of an individual and
The extent to which these inputs and processes
exist may support the inference that student
learning is taking place. Each of these discoveries
about student learning was gained through indirect
methods of assessment, such as surveys of student
perceptions and opinions. The results of these
surveys then were correlated with actual student
learning outcomes (measured directly),
demonstrating that when the existence of
specified inputs and processes correlates with
student learning.
4 See Chapter 5 for a further discussion of this topic. See also Figure 21 for a list of conditions under which students learn
most effectively.
33
he or she responds in a free-form manner,
expressing, for instance, an idea, opinion, or
evaluation. Because of their non-numerical nature,
qualitative data cannot be subjected directly to
statistical analyses, nor can easy direct comparisons
be made without engaging in an intervening process
to categorize or interpret the data. Qualitative data,
however, can be “richer” than quantitative data,
because they provide a more extensive variety of
information related to a particular learning goal.
Many faculty members, for instance, use the
numerical scores (quantitative data) from their
teaching evaluations to make overall judgments of
their own performance, but they value the
qualitative, narrative comments from students as
more useful in revealing students’ personal
perceptions of a course.
A common misconception is that qualitative
assessments are not as reliable, valid, or objective
as quantitative ones. This is not necessarily the
case. There are well-designed and statistically
reliable means of interpreting and analyzing
qualitative data and numerous resources for
learning to use qualitative methods (see Silverman,
2001; Maxwell, 1996). For example, an instructor
might assess the same learning goals using a
multiple-choice test or an essay test. Similarly, an
instructor might grade a senior project presentation
quantitatively with a standard set of evaluation
criteria (i.e., a rubric). Alternatively, he or she
might provide the student with a prose evaluation,
in a non-scaled format, citing the strengths and
weaknesses of the presentation. However, it is best
if this evaluation is organized around a standard set
of criteria that were shared with the student
beforehand.
form of a readily-analyzable numeric score. In
contrast, qualitative data must be sorted,
categorized, and interpreted (most often by humans
rather than by computer programs) before a final
judgment can occur. Methods of ensuring the
reliability of qualitative data are time-consuming.
For instance, to ensure that portfolio assessment is
reliable, at least two raters are used to review each
portfolio, providing a form of “inter-rater”
reliability. Focus groups, another commonly used
form of qualitative data collection, require large
investments of time to gather data from
comparatively few students.
A good use of qualitative evaluation is to help
develop quantitative evaluation criteria (rubrics).
For instance, one might conduct focus groups for
the purpose of designing questions for a satisfaction
questionnaire or use a scoring rubric for portfolios
to determine what characteristics of students’
writing might be evaluated.
For assessing student learning, Characteristics
encourages the use of multiple approaches—both
quantitative and qualitative—but it does not require
the use of both approaches (see Standard 14).
Institutions and faculty members in different
programs should be thoughtful about which
approach, or combination of approaches, best suits
the student outcomes that are being assessed in each
unique situation.
Other Methodological Considerations
Some of the other methodological considerations
often raised with regard to assessment include
reliability and validity; pretests, posttests, and
A student survey designed to gather information on longitudinal design; the role of grades, self-report
measures, and statistical versus practical
student satisfaction may elicit data that are
significance.
quantitative (i.e., “On a scale of 1 to 7, how
satisfied are you with the quality of advising?”) or
qualitative (“How would you describe your
experience with academic advising?”). Similarly,
employers asked to assess the strengths and
weaknesses of alumni may be asked to assign
“scores” to, or to describe, alumni characteristics.
Validity and Reliability
In general, the terms “validity” and “reliability”
refer to the extent to which assessment tools and
methods provide accurate, fair, and useful
information. Both concepts are important factors in
Most beginning assessment initiatives are likely to choosing standardized assessment instruments and
should be considered seriously when developing
rely more heavily on quantitative, rather than
qualitative, forms of assessment for several reasons. locally-created instruments for summative
Quantitative data are easier to collect and are in the assessment.
34
Validity refers to the integrity of the instrument.
Does the instrument measure what it was designed
to measure, or does it actually measure something
else? An instrument designed to assess student
sensitivity to the cultural norms of others, for
instance, may actually be measuring a student’s
sensitivity to detecting those responses desired by
the professor or the institution that values such
sensitivity. Obviously, the instrument would not
provide a valid assessment of cultural sensitivity.
Three forms of validity are especially relevant to
assessing student outcomes. An instrument with
“construct validity” adequately taps the “construct”
or conceptual framework that it is designed to
measure because its questions have been developed
specifically for that purpose. The test of cultural
sensitivity described above lacks construct validity
because it assesses student perceptions of faculty
beliefs, not cultural sensitivity.
Content validity, and in particular “face validity,”
refers to the content and structure of an evaluation
instrument: On the face of it, does it appear to
assess what it is designed to assess (Gall, Borg &
Gall, 1998). The cultural sensitivity instrument
described above may appear to have face
validity—the questions appear to be about cultural
sensitivity—even though it lacks construct validity.
In general, however, the content and structure of an
instrument should make sense to those who are
using it. Several methods are employed by test
designers to ensure that instruments have both
content and face validity.
A third important form of validity is referred to as
“concurrent” or “criterion validity.” Criterion
validity means that a test or assessment instrument
will yield results that are similar to those of other
instruments designed to assess the same outcome.
Two tests of college mathematics ability, for
instance, should yield similar results when
administered to the same students; if one measure
of student satisfaction demonstrates that students
are very satisfied, another should as well. This
result also could demonstrate “predictive validity”
if the strength of the correlation between the two
measures was great. A test or other evaluation
instrument with good criterion validity also will
predict performance on other measures of
constructs that should be related. For instance,
student satisfaction should predict retention, and
high scores on a test of ethical decision-making
should predict ethical behavior. Additional
concepts and examples related to reliability and
validity are discussed in the section below entitled,
“Key questions for choosing assessment
instruments.”
Reliability refers to the consistency of results for
a test or assessment instrument over repeated
administrations to the same individuals.
For instance, an aptitude test for mechanical
engineering, given twice to the same person, should
yield similar results each time. Otherwise, it fails in
its purpose of providing an accurate prediction of
future success in mechanical engineering.
Reliability is established during the development of
the test, when special populations are recruited to
take the test more than once, before the test is used
for its intended purpose. Reliability information
about standardized tests is presented in the form of
statistical correlations (which should be very high)
among repeated administrations of the test in the
same population.
The concepts of validity and reliability apply
primarily to summative assessment, and not as
directly to formative assessment, because
instructor-created examinations and measures
usually only exhibit “face validity,” not the other
forms of validity discussed here, and they are not
usually subjected to rigorous pre-administration
tests of reliability.
Pretests, Posttests, and Longitudinal Designs
A common misconception is that, in order to make
any claims about “value-added” changes in student
learning, one must use a pretest-posttest format. For
instance, in order to demonstrate that a general
education program has improved the writing skills
of students, it appears that it would be necessary to
have data on the writing skills of the same students
before they began the program. This notion could
thwart attempts to assess writing skills, and in a
large institutional setting, it could be so daunting
that it could short-circuit any attempt to
demonstrate that writing skills have improved.
Two conceptual alternatives to a pretest-posttest are
discussed briefly below. Research methods experts
on most campuses could further explain these and
suggest additional alternatives.
35
The first option would be to identify which general
education experiences were designed specifically to
enhance writing skill. Perhaps these experiences
include courses in introductory composition,
rhetoric, and an initial writing-intensive course in
the major. One then could compare two populations
of first-year students or two populations of
sophomores—those who had completed these
courses with those who had not. The group that has
not completed the courses can serve as the
comparison or “control” against the group that
competed the courses.
A second option is to compare students against a
national norm on a standardized test or against a
targeted “benchmark” population. Suppose the
learning goal in question is that students have
gained a certain level of mathematical proficiency
as a consequence of taking a two-course
mathematics sequence in a general education
program. One can administer a standardized test of
college-level mathematics after students have
completed the sequence and compare students’
scores to national norms. In this case, no pretest
was necessary; the national norm serves as the
comparison or “control” group. This method is
problematic if students at the institution are not
drawn from an average population, as would be the
case if the institution is highly selective or
open-access. However, it does produce meaningful
comparisons if an institution’s student population
roughly approximates an average population.
Scholastic Achievement Test scores, for instance,
might be used as a measure of the level of
selectiveness used in admitting students.
institution is interested in demonstrating that its
graduates are successful in their careers, a
longitudinal survey administered to graduates
repeatedly over several years would be appropriate
for several reasons. Demographic data tell us, for
instance, that people change careers multiple times
during their lives, so examination of a single
“window” of time may not be an accurate
assessment. In addition, the population of graduates
offers the benefit that its members will be around
long enough to be surveyed repeatedly over time.
Most importantly, however, a longitudinal design
guards against “cohort effects” that could intrude if
graduates from one generation were compared with
graduates from another generation. Career
trajectories may change historically, and the
character of the institution may have been markedly
different in the past. Thus, 1950s female graduates
may hold a lower percentage of professional
degrees than 1980s female graduates. This finding
tells us more about historical context than
institutional outcomes. However, the same
question, asked of the same individuals at several
different points in time yields meaningful
information. A finding that female students from
the same cohort had a greater percentage of
graduate degrees 20 years after college than
they did 10 years after college could be used
(in conjunction with other outcomes data)
to demonstrate that the institution produces lifelong
learners.
In most cases, when student outcomes during or at
the end of a higher education experience are being
evaluated, longitudinal data are not necessary and
If the institution’s population is not average, a
may not yield meaningful information. Pre-test and
benchmarking strategy would be a more
post-test assessments, as well as alternatives which
appropriate alternative. Students’ scores on a test of are discussed above, are more practical alternatives
college mathematics could be compared to the
and provide answers to the same general question:
scores of students at institutions with comparable
“Has meaningful learning occurred as a result of an
populations. Scores higher than those of the
educational experience?”
benchmarked school would be convincing evidence
that the math curriculum of the target institution is Where Do Grades Fit into the Picture?
successful.
Faculty members and others often ask whether
A common assertion is that longitudinal research
grades are appropriate and sufficient for assessment
designs (those that follow the same individuals over of student learning after the learning goals are
time) are necessary to draw meaningful conclusions defined. The answer is that grades have been, and
about what students have learned. Sometimes
will continue to be, an excellent indicator of student
a longitudinal perspective is warranted because
learning if they are appropriately linked to learning
other approaches are less valid. For example, if an
goals. The Commission recognizes that grades are
36
an effective measure of student achievement if
there is a demonstrable relationship between the
goals and objectives for student learning and the
particular bases (such as assignments and
examinations) upon which student achievement is
evaluated (Standard 14).
students would do well on the essay exam, but their
performance probably would not be related to what
they learned in the course. In this example, a bad
grade could not be attributed to a student’s failure
to learn the material or to prepare for the
examination. Thus, even the use of grades as a
summative assessment warrants a careful approach.
In and of themselves, however, grades are not
direct evidence of student learning. That is, a
numeric or a letter grade alone does not express the
content of what students have learned; it reflects
only the degree to which the student is perceived to
have learned in a specific context.
Thoughtfully-constructed syllabi and “test
blueprints,” which are discussed later in this
chapter, are two of several possible approaches to
connecting grades directly to desired course goals.
One reason “grade inflation” is seen as a problem is
that grades alone cannot be relied on to reflect
student performance accurately. One could ask:
“Does one grade of ‘A’ equal another?” If
instructors were to match grades explicitly with
goals, it would become easier to combat grade
inflation, because high grades must reflect high
performance in specified areas.
Grades, however, can provide an excellent means
for improving teaching and learning both during a
course (formatively) and at the conclusion of a
course (summatively). When grades serve as the
final judgment of performance in a course, they
provide a summative evaluation of students’
performance as individuals and as a class. If the
grades of individual students can be traced directly
to their respective competencies in a course, the
learning achievements of those students are being
assessed in a meaningful fashion. If, however,
examinations or homework assignments are not
designed to test the skills and competencies that the
course was designed to teach, then grades for that
course are measuring something other than student
attainment of the course goals.
Grades and grading practices can be a component
of formative assessment as well. For example,
many faculty members use drafting and revising
processes to teach writing. Students mimic the “real
world” by writing multiple drafts, submitting them
to critiques by the professor or their peers, and
revising them for resubmission. Each draft may be
assigned a grade in association with critical
comments. Depending on the instructor’s preferred
strategy, all or only some of the interim grades may
be used to determine the final grade. In this case,
the grade for each draft, in conjunction with critical
comments, gives the student an indication of his or
her performance, what might be done to improve
the product, and how the quality of the product
changes over time.
Grading can be formative when there are multiple
means and formats for assessing student learning
and when there are repeated opportunities to
demonstrate improvement within the context of one
class. For instance, a professor might assign two or
three papers (with required drafts), two class
presentations, two objective format exams, and a
journal that must be reviewed repeatedly by the
professor during the semester. Each of these
“assessment events” could be graded, providing
Suppose, for instance, an instructor presents the
students with at least two distinct types of
content of an anatomy and physiology course that
opportunity to learn more or learn better. A student
focuses on identifying and labeling anatomical
can compare his or her performance on the various
structures and physiological processes.
assessment formats, thereby learning which skills
An appropriate evaluation of student mastery of the
he or she has mastered and which should be
course content would be an objective final exam
improved. In addition, a grade on the first test
requiring students to label diagrams, answer
administration or the first paper or presentation
multiple-choice definitional questions, and fill in
serves as feedback (a formative assessment) that
the blanks. In contrast, an examination that required
provides information on how to improve. This
students to evaluate a physiology experiment on its
learning experience can be applied toward adapting
methodological merits would not be an assessment
study skills or work habits before the next attempt.
of student learning of the course content. Some
37
For example, imagine two groups of students, each
of whom has completed an introductory calculus
Concerns are often expressed about the use of
course. Assume that members of each group were
self-report measures for answering questions about randomly assigned to two different teaching
student learning. Sometimes these concerns relate
formats—one problem-based and the other
to the use of indirect methods of assessing student
traditional lecture—and that the same professor
learning and the concerns about qualitative versus
taught each course. At the completion of the course,
quantitative assessment discussed previously.
both students are given the same standardized
Often, however, concerns are related most directly calculus examination. The average grade for the
to the validity and reliability of self-report
students in the problem-based course was 10 points
measures. Self-report measures can be designed to higher than the average grade for the students in the
be valid and reliable and can be assessed by
traditional lecture course. Is a 10-point difference
applying the characteristics of reliability and
enough to make the claim that the problem-based
validity described above.
course is a better form of teaching? Would a
2-point difference have been enough? Would
Both common sense and face validity should be
used to determine the value of a specific self-report 20 points be enough to make the claim? A test of
statistical significance would reveal whether the
measure. For example, if the goal is to determine
10-point difference could have happened by
whether students are satisfied, it seems that a
accident in a normally distributed population of
self-report measure is the only means of gathering
students (i.e., the difference could have been caused
such data. Satisfaction, by definition, is one’s
feeling of liking, comfort, and fulfillment resulting by other factors, unrelated to the course, of which
we are unaware), or whether the 10-point difference
from a specific event or situation. Similarly, it is
was large enough that in all likelihood it was
appropriate to gather data on affective states
(emotions) and social perceptions with a self-report caused by differences in the courses.
instrument (assuming that it meets the criteria for
Judgments of statistical significance only become
reliability and validity).
reliable when there are sufficient numbers of
Self-report Measures
It is possible to collect direct evidence of student
learning using self-report measures, but these must
be designed carefully to elicit evidence of student
learning. For example, students may be asked to
reflect on the most important thing they learned in a
specific course, or what else they would like to
learn on the same subject. In doing so, they would
reveal the actual content of what they had learned.
However, self-report questions such as, “Did you
learn a lot in this class?” would not elicit such
information. Self-report measures are most
frequently used to provide valuable indirect
evidence of student learning.
student test or survey results from which to draw
inferences about a population of students. In many
cases, faculty members will be studying outcomes
data from small groups of students or engaging in
formative assessment for which ongoing
improvement in a class or a program is the goal.
In these situations, faculty and staff members
should make judgments and introduce changes
based on “practical significance.” Do the students’
scores, or their change in scores from one time to
another, reveal a pattern or appear to be meaningful
or informative enough to support changes in a
course or program?
In general, when large-scale assessments are being
used, or when standardized tests are administered
program-wide or institution-wide, statistical tests
Data related to student outcomes are often
described as being “statistically significant” or “not should be used to analyze the data. Guidance may
statistically significant.” The concept of statistical be found in social science, education, mathematics
and statistics, and other departments on campus that
significance relates to the probability of a given
use empirical methods.
result occurring by chance. If the result is too
unlikely to have occurred by chance, it is said to be
statistically significant.
Statistical versus Practical Significance
38
Judgments of student outcomes based on practical
significance are equally valid when the number of
students being evaluated is small, when data are
qualitative rather than quantitative, and when the
purpose is to engage in formative assessment.
are designed to evaluate affective and social
development are especially likely to incorporate a
series of questions that seem irrelevant, but that
actually enhance the instrument’s validity.
q Is a standardized instrument appropriate
Key Questions When Choosing and
Implementing Evaluation Instruments
One should ask several questions when choosing
assessment instruments:
q Is the evidence provided by the evaluation
method linked to important learning
outcomes?
This is perhaps the single most important way to
determine the quality of most evaluation tools and
methods. Regardless of whether an evaluation
instrument is standardized (previously published
and tested for validity and reliability) or “home
grown” (created locally for a specific purpose), it is
important to ensure that the instrument is designed
to provide evidence of the desired learning
outcomes. In research design terms, this involves
determining whether the operational definition (the
aggregate instrument or items on the instrument)
actually assesses the construct (the learning goal)
that it is intended to assess (construct validity).
For many standardized instruments, the intended
purpose will be apparent immediately.
A disciplinary test, for example, such as the
American Chemical Society (ACS) test, evaluates
students’ knowledge of facts, skills, and procedures
that should have been acquired as a function of the
undergraduate curriculum in an ACS-accredited
program. Subject-area Graduate Record
Examinations (e.g., the psychology GRE) evaluate
content knowledge in the respective disciplines
they represent. Publishers of other standardized
tests with other less readily obvious content will
explain, in the test information materials, what the
test is designed to assess.
It is important, however, not to assume that the
linkage between every item on a standardized
assessment instrument and the construct it is
designed to assess will be readily apparent. Many
standardized instruments have built-in reliability
checks and so-called “lie-scales.” Measures that
for the learning goals of the institution?
It certainly is not necessary to use standardized
assessment instruments. In fact, for most learning
goals, none will be available. Although a test
created locally may not have the same statistical
validity and reliability as a standardized instrument,
its relevance to the specific learning goals in
question may make it a more appropriate and
effective instrument. A “test blueprint” (an outline
that matches test items with the learning outcomes
they are intended to assess) can be used to construct
a test or instrument or to evaluate how well an
existing “home-grown” instrument assesses key
learning outcomes.
q Is the evaluation method appropriately
comprehensive?
No assessment tool or method can assess every
important learning outcome, but the best ones
assess a comprehensive and/or representative
sample of key learning outcomes. It is not
financially feasible to use several published
instruments to assess multiple outcomes, nor is it
feasible to subject students to multiple tests or
surveys. (The latter has its own measurement
problems.) Regardless of whether an assessment
instrument is standardized or specially created, it
should be as comprehensive as possible.
q Are important learning outcomes
evaluated by multiple means?
Few evaluation methods are perfect. It is important
to triangulate around important learning goals,
assessing them through various means, and through
tests of various formats. For instance, a
standardized test of disciplinary knowledge may be
an adequate form of assessment of students’ content
knowledge of a discipline, but it may provide no
indication of his or her preparedness to be a good
practitioner in that discipline.
39
q Are the questions clear and interpreted
consistently?
In addition to examining the correspondence
between learning goals and the assessment
measures being used, it is important to assess
whether its “non-content” properties are adequate.
For example, a test should not be culture-specific,
its vocabulary and sentence structure should be at
an appropriate level, and it should not contain
ambiguous, unclear, or double-barreled questions
(i.e., questions that actually contain two questions).
Questions should be phrased carefully to ensure
meaningful responses. For instance, imagine that a
targeted learning goal is that students’ desire
to engage in community service increases after
exposure to a service-learning program. Imagine
also the following two questions asked of a
graduate:
The instrument also should ask how the office
might be improved. For example, the respondent
might be asked to name the three most useful
activities of the Career Services Office for helping
students find jobs and to name three ways in which
the functions of that office could be improved.
The concept of creating questions that are useful for
making improvements can be applied to direct
assessments of student learning as well. For
instance, a complicated problem in a physics class
can be divided into subsections to help the
professor determine which content or skill areas
need additional reinforcement.
q Does everyone interpret the responses the
same way?
When assessments of student outcomes are
subjective—that is, if they do not require discrete or
q “On a scale of 1 to 7, how likely are you to quantifiable or unambiguous answers—it is
participate in community service activity?” important to develop a rubric (criteria used to score
or rate responses) to ensure comparability of
q “On a scale of 1 to 7, how much influence
review. There should be collegial agreement on
did your community service during college what constitutes acceptable, inadequate, and
have on your desire to participate in
exemplary responses or performance for each
community service in the future?”
assessment instrument to be used, whether it is a
paper, a project, a presentation, or an artistic
Both of these questions are indirect measures of
offering. A rubric created to reflect the agreement
learning or development, but only the second
provides information that would help the institution should be pre-tested by having colleagues
independently score the same work samples to see
to improve the service-learning program.
if their scores are consistent. The strategy of
A specially created instrument should be reviewed inter-rater reliability can be used as well, by
by several colleagues and students to ensure clarity, enlisting two or more colleagues to “grade” each
and it then should be pre-tested by some students
student’s work or performance.
who have diverse backgrounds and characteristics
in order to clarify ambiguous items.
q Do the results make sense?
q Do questions elicit information that will be It is important to use common sense when
developing assessment instruments, designing a
useful for making improvements?
scoring system or rubric, or interpreting data
Questions should be designed so that, when
resulting from assessment instruments. One would
possible, they yield responses that both evaluate an expect honors students to outperform other students
aspect of the educational experience and suggest
on their senior theses presentations. One also might
options for improvement. For instance, a survey
expect those same students to fare better in
designed to evaluate student experiences with the
applying to graduate school, but not necessarily in
Career Services Office should ask about
being hired to entry-level positions in corporations.
perceptions of its efficacy:
Students who have completed a general education
q “On a scale of 1 to 7, how important was the sequence should score better on tests of general
knowledge and skills related to specified general
Career Services Office in helping you find
education outcomes than students who have not
employment upon graduation?”
40
completed the sequence. Unexpected results should q Is evidence gathered over time and across
trigger further inquiry.
situations?
Assessment is not a once-and-done process.
As students, faculty members, curricula, and
evidence?
teaching methods evolve over the years, even
institutions with very positive assessment results
It is always important to use multiple means of
assessment to determine if a particular learning goal should undertake repeated assessments to ensure
that students are learning as effectively today as
has been met. It also is necessary to compare
they were a few years ago. Because each evaluation
assessment results for related goals for student
learning and even for goals that would be expected technique has relative strengths and weaknesses,
to be mutually exclusive. For instance, rubric scores there is no single perfect assessment that will yield
absolutely accurate information and that is relevant
for the writing quality of senior theses should be
to every situation. In order to have support the
corroborated by students’ grades in composition
classes. Faculty ratings and students’ self-ratings of findings that each evaluation yields, more than one
assessment strategy should be used to corroborate
performance should correlate with each other.
findings.
Focus group results should support survey results
on the same topic. Conversely, students who
demonstrate an increased personal emphasis on
q How much should be assessed?
wellness by their attendance at the gym and by
Plunging immediately into assessing a large
participation in athletics should not be engaging in
number of students on a full range of learning
increased alcohol and drug consumption. The latter
outcomes will overwhelm faculty members and
finding would warrant re-evaluation of the campus
institutional resources. It will produce an
wellness program.
overwhelming amount of information that may be
impossible to interpret or to use in enhancing a
q Are efforts to use “perfect” research tools program. It makes more sense to begin with a more
balanced with timeliness and practicality? limited approach. For example, faculty members
assessing student writing skills might gain more
Although institutions will do their best to ensure
that the research designs they use yield meaningful from a thorough analysis of a sample of 30 papers
than from a more perfunctory review of 300, as
results, they should remember that assessment
well as by assessing only a few key goals.
cannot wait for the perfect research strategy.
q Are the results corroborated by other
Indeed, there probably is no perfect strategy. For
the purpose of managing the quality and change of
an academic curriculum, assessment is a form of
systematic inquiry—i.e., “action research” or
“applied research,” based on the collection and
analysis of data about student learning that is
undertaken with the best knowledge and resources
permissible and within the time available. The
resulting information guides decision makers in
choices related to the curriculum, faculty, the use
of physical space, and other areas that may have an
effect on learning.
Just as every possible outcome need not be
measured, it is not necessary to collect data about
each student’s performance. The Commission is
interested in the institution’s ability to graduate
students with appropriate knowledge, skills, and
behavior, not in a demonstration that every student
was tested. Meaningful and representative
sub-populations (randomly chosen when
appropriate) can provide the basis for
demonstrating that students across the institution
are achieving learning goals.
41
Evaluating Student Learning
Ø Use indirect measures to explain or
support findings from direct measures.
Ø Choose the most relevant level for
evaluation of the learning goals:
institution, program, or course.
Ø Select quantitative or qualitative
measures based on type of student
learning goals.
Ø Ensure that grades are related directly
to goals.
Ø Choose appropriate research design.
to serve as internal consultants, reviewers,
statisticians, and mentors in the assessment process.
Easy-to-Implement
Tools and Techniques
The assessment tools and techniques presented
below yield useful information and are relatively
easy to implement. They are not meant to be an
exhaustive selection of tools but, rather, an
overview of available options.
Rubrics or Rating Scales
A rubric is an instrument based on a set of criteria
for evaluating student work. Rubrics help a
professor or other evaluator to make explicit,
Ø Use common sense: Is the result logical?
objective, and consistent the criteria for
performance that otherwise would be implicit,
subjective, and inconsistent if a single letter grade
were used as an indicator of performance. Rubrics
delineate what knowledge, content, skills, and
behaviors are indicative of various levels of
learning or mastery. Ideally, “grading” rubrics are
q Are faculty and staff members who are
shared with students before an exam, presentation,
knowledgeable about measurement
writing project, or other assessment activity.
serving as resources for developing
Conscious awareness of what he or she is expected
assessment instruments?
to learn helps the student organize his or her work,
encourages self-reflection about what is being
The work of assessing student learning is
learned and how it is being learned, and allows
essentially systematic inquiry in the tradition of
opportunities for self-assessment during the
social science or evaluation research, with its
learning process. Huba and Freed (2000) suggest
attendant need for validity, reliability, control,
analysis, and interpretation, to the extent that these that instructors consider involving students in the
development of rubrics as a class progresses as a
are possible. Although everyone involved in the
enterprise is an expert in the content base of what is way of helping students to develop their own
being researched (i.e., teaching and interacting with conceptions of what constitutes good and poor
work. Both Huba and Freed (2000) and Walvord
students in a higher education setting), few are
and Anderson (1998) offer extensive information
expected to be experts in conducting research.
on developing rubrics.
While much of the work of assessing student
learning has a common-sense base, it is also true
Figure 6 includes a description of the characteristics
that meaningful analysis of student learning,
and components of rubrics. Huba and Freed (2000)
especially beyond the course level, requires
present a thorough description of the uses and
expertise. There are usually faculty members on
purposes for rubrics, along with a comprehensive
campus who are trained as social science,
primer on how to construct them.
education, or other researchers. They can conduct
There are four basic types of rubrics: simple
careful, meaningful research and can construct
checklists, simple rating scales, detailed rating
measures. These faculty members, who can be
scales, and holistic rating scales.
found in psychology, sociology, education,
business, and other departments, may be enlisted
Ø Use formative assessment “mid-course”
to improve teaching and learning.
42
Figure 6
Criterion-based Rating Scales (Rubrics)
What is a rubric? A rubric is a criterion-based rating scale that can be used to evaluate student performance
in almost any area. A rubric establishes the “rules” for the assignment (Huba and Freed, 2000). It contains
a priori criteria for various levels of mastery of an assignment.
How is a rubric used? The person evaluating student performance uses a rubric as the basis for judging
performance. Ideally, rubrics are available to students prior to their completion of the assignment so that
they have clear expectations about the components of the evaluation and what constitutes exemplary
performance.
What are some of the criteria that may be used within a rubric to evaluate student work? Criteria can
include sophistication, organization, grammar and style, competence, accuracy, synthesis, analysis, and
expressiveness, among others.
What are the components of a rubric? Huba and Freed (2000) describe the following elements of rubrics:
Ø Levels of mastery (e.g., unacceptable through exemplary)
Ø Dimensions of quality (see criteria above)
Ø Organizational groupings (macro categories for criteria)
Ø Commentaries (the junctures between levels of mastery and dimensions of quality; e.g., a description
of the characteristics of an exemplary organization)
Ø Descriptions of consequences (components of commentaries that relate to real-life settings and
situations).
Where can I see examples of rubrics and learn more? Walvoord and Anderson (1998) and Huba and
Freed (2000) are both excellent sources of information about the characteristics of rubrics and how to
develop them. They also provide examples of various forms of rubrics.
Simple Checklists. This form of rubric can be used
to record whether the relevant or important
components of an assignment are addressed in a
student’s work. For instance, a rubric might be used
to assess whether a laboratory report contained
required sections or whether a writing sample
contained all of the assigned parts. A checklist of
this sort is categorical, that is, it records whether or
not a required aspect of an assignment is present,
but it does not record quantitative information
about the level of competence a student has
achieved or the relative skill level he or she has
demonstrated.
Simple Rating Scales. This form of rubric records
the level of student work or categorizes it
hierarchically. It is used, for instance, to indicate
whether student work is deficient, adequate, or
exemplary, or to assign a numerical “code” to
indicate the quality of student work.
In most cases in which a numerical scale is used, it
should contain a clear neutral midpoint
(i.e., the scale should contain an odd number of
rating points). However, survey designers should
determine when this might not be appropriate.
Occasionally, such scales are intentionally designed
without a midpoint in order to force a non-neutral
response.
Figure 7, an excerpt from an employee rating scale,
is an example of a simple rating scale that does not
provide information about the “value” of different
points on the scale.
Detailed Rating Scales. Detailed rating scales
describe explicitly what constitutes deficient,
adequate, or exemplary performance on each
criterion. Detailed rating scales are especially
43
Figure 7
Excerpt from a Simple Rating Scale
Employer’s Final Performance Evaluation of
Knowledge, Skills, and Attitudes (KSAs)
of: _______________________________
Dear Employer:
The College of Business Economics (CBE) understands the need for its graduates to be broadly trained and
ready to perform immediately upon entering the job market, both as individuals and in teams. Therefore, its
curriculum contains concrete, measurable, and attainable objectives throughout. As a result, each CBE
graduate is expected to perform successfully in the following areas of Knowledge, Skills, and Attitudes.
Please rate your intern or employee’s performance only on the areas that apply to his/her job.
The rating scale is: 5=Excellent; 4=Good; 3=Satisfactory; 2=Fair; 1=Poor; N/A=Not Applicable.
Excerpt:
COMMUNICATION: WRITTEN, SPOKEN, GRAPHIC, AND ELECTRONIC
5
4
3
2
1
n/a
5
4
3
2
1
n/a
1. Write articulate, persuasive, and influential business reports, proposals, and letters
2. Make articulate, persuasive, and influential individual and team presentations
3. Develop graphic, spreadsheet, and financial analysis support for position taken
4. Display presentation skills
5. Generate appropriate visual aids
6. Use correct written structure, spelling, grammar, and organization
7. Articulate another’s viewpoint through verbal and non-verbal cue interpretation
8. Resolve interpersonal and team conflicts
9. Negotiate effectively
THINKING: CRITICAL, CREATIVE, AND INTEGRATED
10. Use problem-solving techniques
11. Use adaptable, flexible thinking
12. Use critical thinking to produce comprehensive, supported, integrated conclusions
13. Use creative thinking methods to produce ideas
14. Distinguish fact from opinion, and critical from non-critical information
15. Develop several workable solutions to a problem
16. Show common sense
17. Demonstrate continuous learning (learning to learn)
Source: College of Business and Economics, Towson University, November 2001. Adapted with permission.
Some of the other characteristics that could be evaluated in the manner shown in Figure 7 include:
¬Technology
¬Ethics and Values
¬Business Disciplinary Content
¬Leadership, Entrepreneurship,
¬ Diversity - International and Demographic
¬ Practical Excellence
¬ Job Experience and Career Development
44
useful when several faculty members are scoring
student work, because they communicate common
performance standards and therefore make the
scores more consistent. Detailed rating scales are
useful to present to students when an assignment
is given or at the beginning of a semester or even
a program. They provide students with a clear
description of what they are expected to learn and
the criteria upon which their learning will be
judged.
Ratings/Comments from Internship or
Research Supervisors
Programs that place students in practica, such as
internships, cooperative education, and student
teaching assignments, usually require that the
on-site supervisor rate the student on essential
knowledge, skills, and attitudes. Such scales are
relatively simple to construct (see Figure 7.)
Because these experiences require students to
integrate and use much of what they have learned in
Figure 8 is an example of a rubric designed as a
a program, these rating scales are evidence of what
detailed rating scale.
students have learned during the program. Brief
comments from supervisors also provide valuable
Holistic Rating Scales. Holistic rating scales
insights into the overall strengths and weaknesses
define deficient, adequate, or exemplary student
work as an aggregate, by assigning a single score to of a program.
a constellation of characteristics that have been
Placement Rates
fulfilled to a substantial degree, rather than rating
each criterion separately. Holistic rating scales
For professional programs whose goals include
often are used when evaluating student work that
preparing students for a particular career, the
may vary so widely in form and content that the
proportion of graduates who find positions in that
same criteria may not apply to all. Capstone
career is important indirect evidence of whether
projects in an art program, for example, might vary students are learning essential knowledge and
so that they cannot all be judged using the same
skills. If a large proportion of graduates from a
specific criteria. However, a faculty member could teacher education program is successful in finding
create a generic description of what constitutes
teaching positions, for example, it is likely that
exemplary work, adequate work, and so on,
those graduates have the knowledge and skills that
regardless of the medium or focus of the work.
school administrators consider important for
successful teachers. Similarly, if a program aims
Figure 9 is an example of a holistic rating scale.
to prepare students for graduate study or
professional programs—pre-medicine and pre-law
Self-reflection
programs are examples—the proportion of
Asking students to reflect on what and how they
graduates who are admitted into graduate or
have learned—in other words, to engage in
professional programs is important evidence that
metacognition—has several benefits. Student
students have learned what graduate programs
self-assessments give faculty members useful
consider important for success in their programs.
insights into the learning process, help students
Note, however, that placement rates alone do not
integrate what they have learned, and provide
provide insights into exactly what students are
students with an understanding of the skills and
learning. Therefore, they are usually insufficient
strategies they need to learn most effectively.
evidence of student learning if used alone.
Classroom assessment techniques suggested by
Angelo and Cross (1993) and other similar
self-reflection strategies have the added advantage
of taking very little faculty or student time. The
student often is asked to write simply a phrase or
sentence. Examples of self-reflection questions that
might be a useful part of an assessment program are
provided in Figure 10.
45
Figure 8
Example of a Detailed Rating Scale
This scale is adapted from one used to evaluate a “book journal and review” for a cognitive psychology
class. For the assignment, students were expected to read one full-length book, chosen from a list provided
by the instructor and related to the content of the courrse but not included on the required course reading list.
The purpose of the assignment was to provide a basis for making connections between the course content,
other professional or popular work in the field, and students’ daily exposure to topics or situations related to
cognitive psychology in their personal lives and in their other courses. A further purpose of the assignment
was to enable students to develop skills in describing research in cognitive psychology to the lay public.
The assignment involved reading the chosen book during the course of the semester and keeping a journal of
reflections related to the purpose of the assignment. Students also were expected to write a professional style
book review (of the type that might appear in the New York Times Review of Books). The rubric is
abbreviated for inclusion here.
Unacceptable
Fair
Proficient
Exemplary
Grammar and style that
interfere with a reader’s
ability to understand
the ideas presented
Grammar and style
adequate for the reader
to grasp the main
concepts presented
Grammar and style
allow the reader to
understand easily the
concepts presented
Grammar and style
enhance the reader’s
ability to understand
the concepts presented,
including nuances of
thought; May provide
a pleasurable reading
experience
Author’s ideas are
simply repeated,
indicating that
engagement was at or
below a surface level
Occasional discussion
of the author’s ideas,
suggesting ability to
engage
Frequent discussion
and analysis of the
author’s ideas,
including expression of
well-supported
opinions about those
ideas, suggesting
almost constant
engagement
Rich, mature grasp of
the author’s ideas,
coupled with analysis
and synthesis with own
ideas and ideas of other
writers and scholars,
suggesting constant and
sophisticated
engagement
Very few connections
with course material
Sporadic but
meaningful connections
with course material
Regular and meaningful
connections to course
material
Continual connections
to course material and
sophisticated discussion
of those connections
Very few connections
with other experiences
Sporadic but
meaningful connections
with other experiences
Regular and meaningful
connections with other
experiences
Continual connections
to other experiences
and sophisticated
discussion of those
connections
Book Journal
Use of grammar and
style to communicate
ideas effectively
Engagement with the
author’s ideas
Connections between
the course and
the book
Connections between
other experiences and
the book
46
Book Review
Grammar and form
Communication of
cognitive psychology
concepts to the reader
Grammar and style
impede understanding
of the “plot” or thesis
of the book; not
consistent with the
form of a professional
book review
Grammar and style are
adequate for the reader
to grasp the “plot” or
thesis of the book; the
form is consistent with
that of a professional
book review
Grammar and style
allow the reader to
understand easily the
“plot” or thesis of the
book; closely adheres
to the style and form of
a professional book
review
Grammar and style
enhance the reader’s
ability to understand
the “plot” and thesis of
the book;
indistinguishable from
a professional book
review
Ignores the reader’s
perspective of the
reader and/or
communicates
cognitive psychology
concepts inaccurately
or without scientific
analysis
Sometimes considers
the perspective of the
reader and occasionally
communicates
cognitive psychology
concepts well
Consistently addresses
the perspective of the
reader and
commuinicates
cognitive psychology
concepts accurately and
usefully
Engages the reader and
“forces” the reader to
be interested in the
topic of the book;
describes cognitive
psychology concepts
accurately and usefully
Test Blueprints
Other Assessment Tools
The creation of local examinations—“traditional”
examinations at the course level, or comprehensive
examinations at the program level—ideally begins
by writing a test blueprint before developing the
actual test questions. Often called a table of
specifications, a test blueprint is a list of the key
learning outcomes to be assessed on the test, with
the number of points or test questions to be devoted
to each goal.
Some other assessment tools may be valuable
components of many successful assessment
programs, but they are more difficult or
time-consuming to implement than the tools
suggested above, and they also may require
significant financial resources to purchase or
administer. Careful consideration is warranted to
determine whether information yielded from these
strategies justifies the time and effort they require.
An example of a test blueprint appears in Figure 11.
Note that in a test blueprint, an essential learning
Multidimensional or Comprehensive Tests
outcome might be represented by questions worth a
As most faculty members are already aware, valid
total of 20 points, while a lesser learning outcome
and reliable tests can be very difficult to design,
might be represented by only 5 points.
especially those meant to assess higher-order
The test blueprint itself is important evidence of the thinking skills, attributes, or values. Tests of this
test’s validity. When matched with test scores, it
type should be administered, analyzed, and revised
offers clear evidence of what students have learned over several semesters to eliminate poorly written
because it covers all learning goals. One could say items and to ensure optimal quality. It is best to
with confidence, for instance, that a student earning seek the advice of a colleague who is an expert in
an “A” on the test has mastered all or most of the
“tests and measurements” before embarking on the
important learning outcomes for a course or a
construction of a comprehensive test of multiple
program.
student learning goals. Several books are primers
on test construction. At the very least, they will
provide the reader with an overview of the best
47
Figure 9
Example of a Holistic Scoring Guide
(For Critical Thinking)
by Facione and Facione
[Ed. Note: The criteria below are shown from the highest score to the lowest.]
4
Consistently does all or almost all of the
following:
2
Does most or many of the following:
q Misinterprets evidence, statements, graphics,
q Accurately interprets evidence, statements,
questions, etc.
graphics, questions, etc.
q Fails to identify strong, relevant
q Identifies the salient arguments (reasons and
counter-arguments
claims) pro and con
q Ignores or superficially evaluates obvious
q Thoughtfully analyzes and evaluates major
alternative points of view
alternative points of view
q Draws unwarranted or fallacious conclusions
q Draws warranted, judicious, non-fallacious
q Justifies few results or procedures, seldom
conclusions
explains reasons
q Justifies key results and procedures, explains
q Regardless of the evidence or reasons,
assumptions
maintains or defends views based on
self-interest or preconceptions
q Fair-mindedly follows where evidence and
reasons lead
3
1
Does most or many of the following:
Consistently does all or almost all of the
following:
q Accurately interprets evidence, statements,
q Offers biased interpretations of evidence,
graphics, questions, etc.
statements, graphics, questions, information,
or the points of view of others
q Identifies relevant arguments (reasons and
q Fails to identify or hastily dismisses strong,
claims) pro and con
relevant counter-arguments
q Offers analyses and evaluations of obvious
q Ignores or superficially evaluates obvious
alternative points of view
alternative points of view
q Draws warranted, non-fallacious conclusions
q Argues using fallacious or irrelevant reasons,
q Justifies some results or procedures, explains
and unwarranted claims
reasons
q Does not justify results or procedures, nor
q Fair-mindedly follows where evidence and
explain reasons
reasons lead
q Regardless of the evidence or reasons,
maintains or defends views based on
self-interest or preconceptions
q Exhibits close-mindedness or hostility to
reason
© 1994, Peter A. Facione, Noreen C. Facione, and The California Academic Press.
For further information, contact the authors at Insight Assessment (info@insightassessment.com; Phone: 650-692-5628)
or visit the website at http://calpress.com/rubric.html for a reproducible version and instructions.
48
Figure 10
Student Self-reflection Questions
for a Course or Program
1.
How do you feel about writing/teaching/biology/sociology/etc.?
2.
What will you say to your friends about this course/program?
3.
What suggestions would you give other students on ways to get the most out this course/program?
4.
How do you feel about yourself as a writer/teacher/biologist/sociologist/etc.?
5.
What are your strengths as a writer/teacher/biologist/sociologist/etc.?
6.
What makes a person a good writer/teacher/biologist/sociologist/etc.?
7.
What was the one most useful or meaningful thing you learned in this course/program?
8.
What was your biggest achievement in this course/program?
9.
In what area did you improve the most? What improvement(s) did you make?
10. What one assignment for this course/program was your best work? What makes it your best work?
What did you learn by creating it? What does it say about you as a writer/teacher/biologist/sociologist/
etc.?
11. Describe something major that you have learned about yourself in this course/program.
12. List three ways you think you have grown or developed as a result of this course/program.
13. In what ways have you improved as a writer/teacher/biologist/sociologist/etc.?
14. What have you learned in this course/program that will help you continue to grow as a
writer/teacher/biologist/sociologist/etc.?
15. What was your favorite aspect of this course/program? Why?
16. What goals did you set for yourself in this course/program? How well did you accomplish them?
17. If you were to start this course/program over, what would you do differently next time?
18. What strategies did you use to learn the material in this course/program? Which were most effective?
Why?
19. What risks did you take in this course/program?
20. If you could change any one of the assignments you did for this course/program, which one would it be?
What would you change about it?
21. What problems did you encounter in this course/program? How did you solve them?
22. What one question about this course/program is uppermost on your mind?
23. What would you like to learn further about this subject/discipline?
24. In what area would you like to continue to strengthen your knowledge or skills?
25. Write one goal for next semester/year and describe how you plan to reach it.
© 2002, Copyright by Linda Suskie. Reproduced with permission.
49
Figure 11
Example of a Test Blueprint
Educational Research Methods: Final Exam Outline
The final exam will consist of 25 multiple-choice items, each worth 2 to 4 points, and five short essay
questions, each worth 3 to 5 points. The items will cover most of the concepts listed below.
q Understand what r, R2, and partial correlations are and
Validity and Reliability (Up to 16 points)
what they tell us.
q Demonstrate an understanding of reliability and
q Understand what multiple regression analysis is used for
validity.
and what it tells us.
q Correctly identify the type of reliability and validity
evidence being provided by given information on an
instrument.
Qualitative Research: Observation, Interviews, and
Ethnographic Research (Up to 16 points)
q Recognize examples of measurement error in a given
q Describe and discuss qualitative research and its key
situation.
characteristics.
q Assess the meaning and implications of measurement
q Identify the pros and cons of qualitative research.
error.
q Describe the concept of a focus groups.
q Apply general principles for ensuring validity.
q Identify the pros and cons of focus group research.
Inferential Statistics (Up to 16 points)
q Understand the key principles in conducting focus
q Demonstrate an understanding of the concept of
groups.
a null hypothesis.
q Define ethnographic research is and identify or describe
q Select the most appropriate inferential statistics
examples of it.
(t, F, or c2) for a given research situation
q Know the most common “cut-off” point that statisticians Historical Research (Up to 10 points)
use in deciding whether two means differ statistically
q Describe the need for historical research.
significantly from one another.
q Identify kinds of historical research sources.
q Correctly interpret the results of t, F, and c2 tests as
q Recognize examples of primary and secondary
presented in research articles.
resources.
q Interpret the effect of standard deviation and sample
q Understand how to evaluate historical research.
size on the results of a statistical test.
Content Analysis (12 points)
Experimental Research (Up to 12 points)
q Demonstrate an understanding of content analysis.
q Interpret correctly the symbolic representations of
q Understand the pros and cons of content analysis.
experimental designs.
q Recognize examples of different kinds of content
q Describe the benefits and limitations of each
experimental and quasi-experimental design covered in
class.
q Identify the appropriate research design for a given
analysis.
q Explain how to analyze content analysis data.
Multiple Units (Up to 6 points)
research situation.
q Identify the most appropriate research method for a
given situation.
Correlational Research (Up to 12 points)
q Demonstrate an understanding of regression and the use
of regression equations.
50
questions to ask when seeking expert advice
(Anastasi and Urbina, 1996; Haladyna, 1999).
Adding a published test to an assessment program
will require time to identify, evaluate, and
experiment with potential tests. Unfortunately,
many published tests aimed at the higher education
market offer limited evidence of quality (i.e.,
validity and reliability) and have been normed with
relatively small groups of students.
It is most important to compare the test blueprint
against the key learning outcomes of the course or
program in question to see how well they match.
A biology test that focuses on ecological concepts,
for example, probably would not be appropriate as
a key assessment instrument for a biology program
that aims to prepare students for careers in health
professions.
are intrinsic parts of course and program
requirements. Graduating senior surveys, for
instance, could be administered as part of a
capstone course offered in every major.
If an institution determines that a survey is a key
element of an assessment strategy, it should help to
ensure useful results by conducting a pilot test of
the survey. A draft should be administered to a
small group, the responses analyzed, and unclear
questions identified. Strategies to maximize the
response rate should be included in the plans to
administer the actual survey.5
Focus Groups
A focus group interview often is viewed as another
quick way to collect assessment information, but
the relatively small number of participants and the
Figure 12 contains a list of published tests designed free-flowing format can reduce the credibility and
to test critical thinking and general education goals. value of the results. Focus groups are usually most
appropriate as tools to help illuminate other
It is presented here as an example of the various
test characteristics that should be considered when assessment results, rather than as stand-alone
assessment strategies.
choosing an appropriate published assessment
instrument.
Successful focus groups require time for planning,
testing, and analysis to ensure a balanced
Ad Hoc Surveys and
discussion among a sufficient number of
participants and to assure that the results have
Pre-graduation Surveys
credibility and value. One should learn how to plan
Many people view surveys as a quick way to
and conduct focus groups, hire a consultant, or
collect assessment information. Unfortunately,
enlist the aid of an on-campus expert before using
surveys that are designed and administered quickly
focus groups as an assessment strategy.
often have low response rates and poorly-phrased
Several sources introduce the science of conducting
questions that yield information of questionable
focus groups and their use as a source of
value.
information. For example, see Morgan (1997); and
Indirect assessments of student perceptions and
Krueger and Casey (2000).
satisfaction that are administered at the institutional
level and are not embedded in course and program
Portfolios
requirements—such as voluntary graduating senior
Portfolios are structured, focused, and purposeful
surveys—take extra time and effort for both
collections of student work. They are increasingly
students and faculty members, and they present
popular assessment strategies, because they provide
sampling problems. It also can be difficult to
motivate students to participate in such extraneous an exceptionally comprehensive, holistic picture of
student learning.
assessment efforts, or to give their best possible
effort and thought, thus reducing the validity of the Figure 13 offers some questions that may help in
assessment itself. It is often simpler, more efficient, a decision on whether or not to use portfolios.
and more effective to use assessment strategies that If the decision is made to use portfolios, it is best to
5 For a discussion of effective survey use, see Suskie (1996).
51
Figure 12
Commonly-administered Measures of Critical Thinking
Measure
Critical
Thinking
Definition
Subscales
Design
Appropriate
Participants
Comprises
attitudes,
knowledge,
skills
Inference,
Recognition of assumptions,
Deduction,
Interpretation,
Evaluation of arguments
Parallel forms
A & B; 80 multiplechoice items, based
on readings; 40 mins.
to complete
California
Critical
Thinking Skills
Test
Purposeful,
self-regulatory
judgment
Analysis, Evaluation, Inference,
Inductive, Deductive
Parallel forms A
and B; 34 items;
40 mins.to complete
College age
California
Critical
Thinking
Dispositions
Inventory
Attitudinal
inclination to
apply critical
thinking skills
Truth seeking, Open mindedness,
Analyticity, Systematicity,
Critical thinking self-confidence,
Inquisitiveness, Cognitive
maturity
Likert-type scale; 75
items; Response
ranges from Agree to
Strongly Disagree;
40 mins.to complete
College age
Watson-Glaser
Critical
Thinking
Appraisal
Ennis-Weir
Critical
Thinking
Essay Test
Cornell Critical
Thinking Test
Reasonably
deciding what
to do or what
to believe
Reasonably
deciding what
to do or
believe
Getting the point; Seeing reasons
and assumptions; Stating one’s
point; Offering good reasons;
Seeing other possibilities;
Equivocation; Irrelevance;
Circularity; Reversal of
conditional relationships;
Straw person fallacy;
Overgeneralizations; Excessive
skepticism; Credibility; Using
emotive language to persuade
Level X: Induction; Deduction.
Credibility, Assumptions, Value
judgment; Meaning
Level Z: All level X subscaled
plus semantics, prediction,
definition
Essay format;
Responses written to
questions about
scenarios; 40 minutes
to complete
Level X: 71 multiplechoice items based
on scenarios; Level
Z: 52 multiple-choice
items based on
scenarios; 50 mins.
to complete
9th grade and
higher
Grade 7 to
College
Level X: 4th
grade-college
sophomore
Level Z: gifted
high school
and collegeaged adults
References: Adams, M., Whitlow, J., Stover, L., and Johnson, K. (1996). Critical thinking as an educational outcome: An
evaluation of current tools of measurement. Nurse Educator, 21 (3), 23-31.
Facione, N.C. (1997). Critical thinking assessment in nursing education programs: An aggregate data analysis. Millbrae:
The California Academic Press.
Ennis, R. H. & Millman, J. (1985). Cornell critical thinking test, Level 2. Pacific Grove, California: Midwest Publications.
Ennis, R. H. & Millman, J. (1985). Cornell critical thinking test, Level X. Pacific Grove, California: Midwest Publications.
Rane-Szostak, & D. Robertson, J. F. (1996). Issues in measuring critical thinking: Meeting the challenge. Journal of
Nursing Education, 35(1), 5-11.
Note: This table contains information on the most commonly used measures of critical thinking only. It is not meant to be
exclusive. There are many more measures available, including several domain-specific measures. Table prepared in 2001
by D. A. Redding,Ph.D., Instructional Assistant Professor, Mennonite College of Nursing, Illinois State University.
Reproduced with permission..
52
start on a small scale. Portfolios may be especially
appropriate for programs that enroll only a handful
of students. Such programs would be ideal for
piloting portfolio projects for later use with larger
programs.
Figure 13
Considerations when
Deciding to Use
Portfolios
Portfolios can present significant logistical
problems related to sampling, storage, development
of evaluation criteria, and the allotment of
sufficient faculty and staff time for review. These
issues can be resolved, but the solutions may take
time to identify and implement. For example, a
number of institutions use electronic portfolios to
solve the storage problem. Huba and Freed (2000)
provide an excellent discussion of the development
and assessment of portfolios.
1. What are the goals of the portfolio?
q What do you want your students to learn
by the act of creating a portfolio?
q What processes or outcomes are to be
evaluated by the portfolio?
Retention/Graduation Rates
2. How will students choose what to include in
the portfolio?
Retention and graduation rates that do not meet the
institution’s goals may be signs of problems with
student learning. However, they do not necessarily
reveal what students actually have learned. They
can be useful to the extent that they correlate with
and illuminate direct learning assessments, or that
they assess directly such institutional outcomes as
cost effectiveness, diversity, student achievement,
and other evaluations of institutional effectiveness.
3. How and when will work be inicluded in
the portfolio?
4. How will student and faculty reflection occur
in the portfolio process?
5. How will the portfolios be reviewed and
evaluated? What would a successful
portfolio in your program look like? What
are your criteria for deciding if a portfolio is
a “success”?
6. Will the portfolios be graded? If so, how?
7. How and where will portfolios be stored?
8. Will the portfolios be passed one faculty
member to another? Will students retain
ownership of portfolios?
9. What are the benefits of moving toward
portfolio assessment? What are the areas of
concern?
10. Is the collection of student work a feasible
practice in your program?
© 2002, Copyright by Linda Suskie. Reproduced with
permission.
53
4
Understanding Middle States Expectations
for Assessing Student Learning and
Institutional Effectiveness
I
n 2002, the Middle States Commission on
Higher Education introduced updated
accreditation standards that simplified
requirements for resources and processes and
concentrated instead on assessment: evidence
that the institution is achieving its goals. Every
accreditation standard now includes an assessment
component; the assessment of student learning is
addressed in Standard 14 (Assessment of Student
Learning); and the assessment of all key
institutional goals, including those assessed in the
other thirteen standards, is addressed holistically in
Standard 7 (Institutional Assessment).
Because Standards 7 and 14 are a significant
change from prior standards, and because the
Commission gives institutions great latitude in
choosing approaches to comply with them, these
two standards have engendered many questions.
This statement is intended to address these
questions and to clarify the Commission’s
expectations regarding these standards and their
relationship to other standards such as Standard 2
(Planning, Resource Allocation, and Institutional
Renewal).
What is the Assessment of Institutional
Effectiveness (Standard 7)?
Assessment may be characterized as the third
element of a four-step planning-assessment cycle:
1. Defining clearly articulated institutional and
unit-level goals;
2. Implementing strategies to achieve those goals;
3. Assessing achievement of those goals; and
4. Using the results of those assessments to
improve programs and services and inform
planning and resource allocation decisions.
The effectiveness of an institution rests upon
the contribution that each of the institution’s
programs and services makes toward achieving the
goals of the institution as a whole. Standard 7
(Institutional Assessment) thus builds upon all
other accreditation standards, each of which
includes periodic assessment of effectiveness as
one of its fundamental elements. This standard
ties together those assessments into an integrated
whole to answer the question, “As an institutional
community, how well are we collectively doing
what we say we are doing?” and, in particular,
“How do we support student learning, a
fundamental aspect of institutional effectiveness?”
(Standard 14). Self-studies can thus document
compliance with Standard 7 by summarizing the
assessments within each accreditation standard into
54
conclusions about the institution’s overall
achievement of its key goals.
q Institutional and program-level goals are
clear to the public, students, faculty, and
staff.
What is the Assessment of
Student Learning (Standard 14)?
q Institutional programs and resources are
organized and coordinated to achieve
institutional and program-level goals.
Assessment of student learning may be
characterized as the third element of a four-step
teaching-learning-assessment cycle that parallels
the planning-assessment cycle described above:
q The institution is indeed achieving its
mission and goals.
1. Developing clearly articulated learning
outcomes: the knowledge, skills, and competencies
that students are expected to exhibit upon
successful completion of a course, academic
program, co-curricular program, general education
requirement, or other specific set of experiences;
2. Offering courses, programs, and experiences
that provide purposeful opportunities for students
to achieve those learning outcomes;
3. Assessing student achievement of those learning
outcomes; and
q The institution is using assessment results to
improve student learning and otherwise
advance the institution.
What Are the Characteristics of
Assessment Processes that Meet
Middle States Expectations?
Effective assessment processes are useful,
cost-effective, reasonably accurate and truthful,
carefully planned, and organized, systematic, and
sustained.
1. Useful assessment processes help faculty and
4. Using the results of those assessments to
staff make appropriate decisions about improving
improve teaching and learning and inform planning programs and services, developing goals and plans,
and resource allocation decisions.
and making resource allocations. Because
institutions, their students, and their environments
Because student learning is a fundamental
are continually evolving, effective assessments
component of the mission of most institutions of
cannot be static; they must be reviewed periodically
higher education, the assessment of student
and adapted in order to remain useful.
learning is an essential component of the
assessment of institutional effectiveness (Standard 2. Cost-effective assessment processes yield
7) and is the focus of Standard 14 (Assessment of dividends that justify the institution’s investment in
Student Learning).
them, particularly in terms of faculty and staff time.
Why Does the Commission Expect
Student Learning and Institutional
Effectiveness to be Assessed?
The fundamental question asked in the
accreditation process is, “Is the institution fulfilling
its mission and achieving its goals?” This is
precisely the question that assessment is designed
to answer, making assessment essential to the
accreditation process. Assessment processes help to
ensure that:
To this end, institutions may begin by considering
assessment measures, indicators, “flags,” and
“scorecards” already in place, such as retention,
graduation, transfer, and placement rates, financial
ratios, and surveys. New or refined measures may
then be added for those goals for which evidence of
achievement is not already available, concentrating
on the institution’s most important goals. Effective
assessments are simple rather than elaborate, and
they may focus on just a few key goals in each
program, unit, and curriculum.
55
3. Reasonably accurate and truthful assessment What Should Institutions Document
processes yield results that can be used with
Regarding Assessment?
confidence to make appropriate decisions. Because
there is no one perfectly accurate assessment tool or
When submitting information on their assessment
strategy, institutions should use multiple kinds of
efforts to the Commission, institutions are expected
measures to assess goal achievement. Assessments
to document:
may be quantitative or qualitative and developed
q clear statements of key goals, including
locally or by an external organization. All
expected student learning outcomes;
assessment tools and strategies should clearly relate
to the goals they are assessing and should be
q an organized and sustained assessment
developed with care; they should not be not merely
process (referred to in some Commission
anecdotal information nor collections of
documents as an “assessment plan”)
information that happen to be on hand. Strategies to
including:
assess student learning should include
Ø institutional guidelines, resources,
direct—clear, visible, and convincing—evidence,
coordination, and support for
rather than solely indirect evidence of student
assessment;
learning such as surveys and focus groups.
4. Planned assessment processes that are
purposefully linked to institutional goals promote
attention to those goals and plans and ensure that
disappointing outcomes are appropriately
addressed. Institutions often have a variety of plans,
such as a strategic plan, academic plan, financial
plan, enrollment plan, capital facilities master plan,
and technology plan. Just as such plans should be
interrelated to ensure that they work synergistically
to advance the institution, assessments should also
be interrelated. At many institutions, effective
institutional planning begins with academic
planning, which in turn drives the other plans.
If the academic plan calls for a new academic
program, for example, the technology plan should
ensure faculty and students in the new program
will be able to use appropriate instructional
technologies. Assessments of the technology plan
should evaluate not just whether instructional
technologies have been put in place but also how
effectively those technologies have helped students
to achieve the program’s key learning outcomes.
5. Organized, systematized, and sustained
assessment processes are ongoing, not once-anddone. There should be clear interrelationships
among institutional goals, program- and unit-level
goals, and course-level goals.
Ø assessment activities and initiatives that
are presently underway;
Ø plans to develop and implement future
assessment activities and initiatives;
q assessment results demonstrating that the
institution and its students are achieving key
institutional and program goals; and
q uses of assessment results to improve
student learning and advance the institution.
How Should This Information
Be Organized and Formatted for
Review by the Commission and
its Representatives?
Assessment documentation that is organized into a
coherent presentation of what the institution is
doing regarding assessment provides a roadmap
that facilitates the work of evaluation teams,
reviewers, and the Commission. Assessment
documentation is typically a living, fluid, organized
collection of documents and/or online resources,
often with references and/or links to further
documents and online resources, that are routinely
updated as the institution’s assessment processes
evolve. There is not, however, any prescribed
format or organization for these materials;
institutions have maximum flexibility in designing
and assembling assessment documentation that fits
best with the institution’s mission, organization,
56
q Assessment documentation incorporated
within the institutional (strategic) plan or
in separate documentation clearly linked to
the institutional plan.
and needs. A single, formal, polished document is
not required and, for many institutions, may not be
the most suitable format, because it may discourage
the continual modifications that are made in
effective assessment processes. The existence of an
effective process, clearly described to the
community and the Commission, is more important
than a formal plan.
q Separate assessment documentation for
each institutional division that is linked
together may be a feasible approach,
especially for large, complex institutions.
Institutions may choose to include an appropriate
combination of the following in their assessment
documentation
q More thorough information in an on-site
resource room and/or online enables
evaluation team members to review a
cross-section of program- and unit-level
assessment processes.
q An overview in a self-study, periodic
review report, or follow-up report gives
the Commission and its representatives a
useful introductory synopsis of the
institution’s assessment processes.
How Are the Documentation of
Institutional Assessment and Student
Learning Assessment Related?
q A chart or “roadmap” outlining
assessment documentation, provided
within a self-study or periodic review report
or as an appendix, can be especially useful
for large or complex institutions with a
broad array of goals and assessment
processes.
As noted earlier, because student learning is a
fundamental component of the mission of most
institutions of higher education, the assessment of
student learning is an essential component of the
assessment of institutional effectiveness. An
q A written or online assessment plan that
institution may therefore create institutional
documents an organized, sustained
effectiveness documentation that includes a
assessment process (including institutional
component on assessing student learning, or it may
guidelines, resources, coordination, and
create a bridge between two separate sets of
support for assessment, assessment activities documentation, one for the assessment of student
and initiatives that are presently underway, learning and one for other aspects of institutional
and plans to develop and implement future
effectiveness.
assessment activities and initiatives) can be
an excellent way to initiate, structure, and
What Might the Commission and
demonstrate compliance with Standards 7
and 14, although it is not required.
Its Representatives Look For in
Assessment plans can guide and support the Assessment Documentation?
institutional community in its efforts to
assess its mission and goals by:
Evaluation team members, reviewers, and
Ø helping to ensure that assessment is
efficient, effective, and purposeful,
rather than just a collection of available
information,
Ø providing information needed to carry
out assessment practices, and
Ø helping to ensure that assessment is
supported with appropriate resources
and that results are used appropriately.
Commissioners might look for information on the
following questions in an institution’s assessment
documentation:
1. Do institutional leaders support and value a
culture of assessment? Is there adequate, ongoing
guidance, resources, coordination, and support for
assessment? (This may include administrative
support, technical support, financial support,
professional development, policies and procedures,
and governance structures that ensure appropriate
57
collaboration and ownership.) Are assessment
efforts recognized and valued? Are efforts to
improve teaching recognized and valued?
2. Are goals, including learning outcomes,
clearly articulated at every level: institutional,
unit-level, program-level, and course-level? Do
they have appropriate interrelationships? Do the
undergraduate curriculum and requirements address
institutional learning outcomes and the
competencies listed in Middle States’ Standard 12
(General Education)? Are all learning outcomes of
sufficient rigor for a higher education institution?
Are learning outcomes for, say, master’s programs
more advanced than those for undergraduate
programs?
9. Where does the institution appear to be going
with assessment? Does it have sufficient
engagement and momentum to sustain its
assessment processes? Or does it appear that
momentum may slow? Are there any significant
gaps in assessment processes, such as key areas
where no assessment plans have been developed?
3. Have appropriate assessment processes been
implemented for an appropriate proportion of
goals? (Expectations for an “appropriate
proportion” are increasing.) Do they meet Middle
States expectations, as characterized above?
4. Where assessment processes have not yet been
implemented, have appropriate assessment
processes been planned? Are the plans feasible?
Are they simple, practical, and sufficiently detailed
to engender confidence that they will be
implemented as planned? Do they have clear
ownership? Are timelines appropriate, or are they
either overly ambitious or stretched out too far?
5. Do assessment results provide convincing
evidence that the institution is achieving its mission
and goals, including key learning outcomes?
6. Have assessment results been shared in useful
forms and discussed widely with appropriate
constituents?
7. Have results led to appropriate decisions and
improvements about curricula and pedagogy,
programs and services, resource allocation, and
institutional goals and plans?
8. Have assessment processes been reviewed
regularly? Have the reviews led to appropriate
decisions and improvements in assessment
processes and support for them?
58
5
Using Assessment Results
To Improve Teaching and Learning
A
commitment to the assessment of student
learning requires a parallel commitment to
ensuring its use. Perhaps the most difficult
part of assessing student learning is the process of
effecting change in teaching and learning as a result
of information gained through assessment practices.
It is pointless simply to “do assessment”; the results
of assessment activities should come full circle to
have a direct impact on teaching and learning and
on the institution’s strategic plan to fulfill its
mission.
Continuous improvement can occur in an upward
spiral if an institution’s structure is flexible, and if
members of the campus community are committed
to the assessment plan and are willing to integrate
the results of assessing student learning into their
collective vision of what the institution is doing
well and what it could do better.
The first section of this chapter discusses the ways
in which institutions can encourage the use of
assessment results, the second section presents
examples of specific types of change that might be
made as the result of information gained in
assessment, and the third section discusses the
interconnectedness of assessment, teaching, and
learning.
Institutional Support Strategies
Designed to Encourage the Use of
Assessment Results
An assessment plan will serve its purpose only if it
provides for the use of assessment results.
Regardless of the level at which assessment is
conducted, an articulated plan for translating
assessment results into changes in practice is
essential. For such a plan to be effective, it requires
an institutional commitment to the use of
assessment results, the sharing of results, a broad
campus discussion of and decision-making on those
results, individuals who are empowered to make
changes, the availability of resources for change,
and flexible procedures for implementing changes.
An Institutional Commitment
The institution should demonstrate a commitment
to developing a system for analyzing results,
identifying areas of strength and weakness, creating
a strategy for improving the learning experience,
and implementing that strategy. Such a
commitment will increase student learning as well
as increase faculty and staff commitment to
assessment. However, if the results of assessment
are not used to improve student learning,
assessment becomes at best a descriptive set of data
about students and, at worst, a useless exercise.
59
Consider a business department that collects data
regarding student or alumni performance on The
American Institute of Certified Public Accounting
Uniform CPA Examination and discovers that the
majority of its graduates are failing the exam.
This knowledge provides the opportunity to review
the relevant parts of the curriculum, implement
strategies for change, and gauge any improvement
in student learning as a result of the changes made.
In contrast, a tacit decision not to make curricular
changes after discovery of this information could
result in the demoralization of students and faculty,
diminished stature for the program, and reduced
selectivity in admissions.
levels of information literacy. Students may fail
to use analytical thinking when critiquing primary
source articles, may cite materials improperly, or
may conduct inadequate literature searches. This
information can help in revising social sciences
courses, but it also would be of great value to
library staff members who design and deliver
significant components of the information literacy
requirement.
The second problem—when faculty members show
little interest in assessment because they perceive it
as meaningless—can result when data are collected
at the institutional level to satisfy an outside
agency, such as a state board of education or an
Changes in programmatic curricula as a result of
accreditor, but are never shared with the faculty.
assessment data do not happen automatically, as
In such cases, the failure to share data may be the
many faculty and staff members can attest.
result of hectic and unplanned-for data collection
However, if the department plan outlines specific
rather than an intentional withholding of important
procedures for examining assessment results and
information from campus stakeholders. If there is
implementing curricular revision, those changes are no planned provision for collecting assessment
more likely to occur.
data—for instance, if they are collected ad hoc to
satisfy an external agency—there is unlikely to be a
Sharing Assessment Results
provision to share them regularly with the campus
Assessment data collected at the institutional and
community.
program levels should be made available to the
There are cases in which an institution decides not
relevant members of the campus community.
to share data because it fears that assessment results
Data at the course level should be shared when it is
indicating that students are not achieving desired
appropriate to do so, such as when faculty members
levels of learning or that students are not satisfied
are collaborating to develop or revise a course, or
will be shared with the general public and will
are team-teaching a course.6 When assessment data
impair the institution’s ability to attract students.
are collected but not shared with those who would
This is counter-productive for several reasons:
be responsible for implementing change, the data
silence by the institution about student performance
are useless for practical purposes. Similarly, a
is itself a red flag to the public, and poor
perceived lack of faculty interest in assessment
performance by the institution’s graduates will
could be caused by the belief that assessment
nevertheless be noticed by employers and the
initiatives yield little or no meaningful information.
public. Most importantly, failure to share
The first problem—when data are collected but not information with internal stakeholders precludes the
shared with those responsible for implementing
opportunity to improve and to produce the type of
change—can occur when one area or program
student learning that will attract students to the
collects data that are relevant to another area but
institution. Even if an institution chooses justifiably
fails to make the data available. For instance, social not to publicize certain results externally, it should
science faculty may assess their students’ research ensure that useful data are shared and used
performance via a required common paper,
internally.
presentation, or capstone course. Assessments
might reveal that students are not achieving desired
6 It is advisable to separate the assessment of student learning from the assessment of an individual faculty member’s
teaching, wherever possible, in order to encourage faculty members to engage in assessment activities, not to shun them.
60
For example, many institutions exercise their
prerogative not to disclose to the public their results
from the National Survey of Student Engagement
(NSSE). The NSSE is a student self-report survey
that measures what it describes as “student
participation in programs and activities that
institutions provide for their learning and personal
development. The results provide an estimate of
how undergraduates spend their time and what they
gain from attending college.”7 In the NSSE,
students report the number of hours they spent on
schoolwork outside of class, the number of written
papers completed, and the length of their papers,
in addition to many other aspects of how they
spend their time. Results such as these about
student engagement can be invaluable to faculty
members, librarians, and student affairs staff as
they revise curricula and programs. Therefore,
administrators and faculty members should give
priority to the best interests of students by devising
a system to share even sensitive assessment results
internally, regardless of the test or measure from
which they resulted.
Campus Discussion and Shared
Decision-making
Empowering Individuals to Effect Change
Clear and public charges should be made to those
who will be responsible for leading programmatic
and curricular change that occurs as a result of
assessment. At the course level, the individual
instructor or group of instructors who teach a
specific course would, of course, be responsible for
its revision. At the program level, someone such as
the department or program chair may be given the
responsibility to ensure that change occurs. This
person is often the same person who implemented
the assessments. At the institutional level, however,
several people from across the institution will be
responsible for assessing and for changing the
curriculum. For instance, the Office of Institutional
Research might collect the data, and other offices or
departments may be charged with effecting change.
It is important to articulate exactly who is
responsible for change so that the data do not
stagnate “on the shelf.” For example, even if the
office of career services is charged with conducting
an annual survey on student acceptance rates at
graduate and professional schools, it should be
made clear which faculty members are responsible
for implementing programs to improve education
and increase graduate school acceptance rates.
Assessment results are less likely to produce
meaningful improvement in learning if only a small
Resources for Change and
number of people or offices make all of the
Celebration of Achievements
decisions about modifications to the learning
After assessment data are collected and curriculum
experience.
and program revisions have been planned,
Students should be included in discussions about
resources must be available to implement the
assessment whenever possible, and they should be
changes. Unfortunately, funds often are not
encouraged to engage in conversations with their
available for every suggested change. Faculty
peers about the institution’s curricula and programs.
members and administrators should review the
Many campuses have specific courses or other
institution’s mission and strategic plan to determine
learning activities that become the nexus of student
funding priorities for new initiatives and to weigh
complaints. For example, some general education
the costs and benefits of proposed changes.
courses frequently become the focus of complaints
A clear process for determining budgetary priorities
about a lack of “real world meaning” and
should ensure commitment to the best interests of
connection to the major. Discussions about
all students, rather than giving priority to the
assessment results and curricular modification are
interests of a small group of faculty or students.
an ideal venue to channel students’ comments and
criticisms constructively.
7 See www.iub.edu/~nsse/html/facts.shtml .
61
Assessment successes need to be positively
reinforced in a way that makes the campus
community aware of the value of assessment.
Yearly celebrations can focus on effective
assessment strategies, positive change as a result of
assessment, or new assessment ideas. More
importantly, traditional reward systems related to
faculty evaluation, promotion, and tenure should
take into account the valuable work of assessment.
Flexibility
This handbook has stressed the importance of
well-communicated and clear procedures and plans
for developing and implementing assessment
programs. Procedures, however, need not be
arduous or cumbersome to result in positive
change. Inflexible and bureaucratic procedures
discourage faculty from embracing assessment and
adapting courses and programs in response to
assessment results. For instance, engineering
technology faculty members might decide that
students need mini-practicum experiences early in
their undergraduate careers because general student
performance on senior projects is inadequate.
Faculty are much more likely to modify the
program to include early practicum experiences if
their proposal is not stalled in committees.
Institutions should strive to develop facilitative
procedures that include relevant stakeholders and
do their best to avoid bureaucratic structures that
discourage change.
wants them to demonstrate. Alternatively,
institutions and committees may exhaust
themselves planning for assessment and become
“burned out” before results actually affect learning.
Even when faculty members are committed to using
assessment results to improve learning, the
institution may not commit the necessary resources.
It is common for accreditation teams to find
beautiful assessment plans that have yet to be
enacted, or “completed” assessment plans for which
the resultant data sit on the shelf because the
institution has not committed sufficient human or
economic resources to support change.
Using assessment results need not be an onerous
task, particularly for faculty who regularly adapt
and modify their courses for the sake of their
students. Using assessment results means changing
courses or programs on the basis of real data rather
than intuition. Even seasoned professors might be
surprised by assessment data. Perhaps students are
not reading a text because it is too elementary for
them or too factual, instead of providing the type of
analysis that might inspire their interest. Students
who are performing extremely well on
examinations nevertheless may not have been
sufficiently challenged by the course. Perhaps
students are performing poorly on one type of
examination (e.g., an essay) because the mode of
teaching was more conducive to performing well on
another type of examination (e.g., multiple choice).
The causes of ineffective learning experiences
cannot always be explained by intuitive hunches.
Translating Assessment Results into
Better Learning
Figures 15, 16, and 17 describe hypothetical and
diverse ways of using assessment results at the
institutional, course, and program levels. Although
The most important reason for assessment is to
they are presented in a condensed and
ensure that students are learning. Even when the
oversimplified form, they are not intended to imply
requirements of those to whom the institution is
that changing curricula, programs, or courses is
externally accountable—students, parents,
simple or requires little thought. Rather, they are
legislators, and accreditors—provide the impetus
presented in the spirit of encouraging discourse
for assessment, the fundamental expectation is that among all members of the campus community,
institutions of higher learning demonstrate that their especially among faculty members and students.
students are learning.
Each of the examples provides: (1) the initial
Unfortunately, there are many obstacles to change. learning goal at the institutional, course, and
Faculty often object to performing yet another task program level; (2) the measures or methods of
related to assessment, citing additional demands on assessing the goal; (3) the outcome evidenced by
the measures; (4) the possible reason for the
their time. They also might believe that the results
of some assessment activities are invalid, or that the outcome; and (5) the action or actions taken as a
results demonstrate merely what the administration result of the assessment.
62
Several noteworthy points are presented in these
examples. First, the reader will notice that not all of
the outcomes are “bad.” Some of the examples
indicate outcomes such as “students’ projects
demonstrate consistently high levels of quality.”
Examples of positive outcomes are used here to
stress the point that when outcomes are
“good”—that is, when students are meeting a
learning goal—faculty members should consider
whether the grade or the assessment reflect true
competence, or whether the assessment or the goal
is inappropriately easy and should be more
ambitious. However, most of the examples here
involve “bad” outcomes, because they provide an
opportunity to think about the causes of poor
performance and productive changes, not because
outcomes in the “real world” are usually “bad.”
Finally, the “possible reason” or hypothesis for the
outcome is presented as if it comes after the
“measure” in time. This is not always the case.
Often those who are focusing on the enhancement
of student learning have developed several
hypotheses before the assessments are
administered; the assessment measures are used
specifically to test the hypotheses. In these
examples, the assumption is made that the measures
listed are ongoing assessments that are part of a
plan. The possible reasons for the outcome or
hypotheses about what needs to be changed are
drawn either from data gathered from the measures
presented in the second column or from clearly
related program or course characteristics—not at
random or from intuitive suppositions.
The examples in Figures 14, 15, and 16 also
illustrate the use of multiple assessments, direct and
indirect measures, self-report and “objective”
measures, and qualitative and quantitative data.
Both direct and indirect measures are presented in
the second column for many of the examples to
illustrate the importance and interrelation of each.
For instance, one of the examples presented in
Figure 14 involves one direct measure
(standardized writing tests) and three indirect
measures (course registration statistics, transcript
analysis, and course content analysis).8 Other
variations in form of measurement are represented
in these examples as well. For instance, an
institution seeking to produce students who
compete well in the job market might use
qualitative data (e.g., an alumni survey with
open-ended questions) and quantitative data (an
alumni survey with numerically-scaled questions,
together with benchmarking statistics) to assess its
goals. Faculty members may use quantitative and
objective data (exam scores) and qualitative and
self-report data (teaching evaluation comments) to
assess the goal that students will master
discipline-specific material. Most of these examples
use multiple measures as well.
For instance, one example in Figure 14 depicts two
possible reasons for student dissatisfaction with a
first-year community service requirement: the first
is students’ failure to understand the relevance to
their chosen career or major, and the second is the
time and transportation hardships students perceive.
Both of these possible reasons for the outcome are
drawn from data gathered in the first-year
experience student satisfaction survey and focus
groups, presented as measures in the second
column. Another example, in Figure 15, describes
students as having low fluency in modern language
after having taken several courses. The hypothesis
that lack of sustained and regular practice
(i.e., infrequent class meetings) can be drawn from
external research about optimum pedagogical
practices in foreign language learning; it need not
be drawn from a locally administered survey or test.
Finally, the reader will notice that occasionally the
“action taken” is to collect more data.
Appendix 7 offers a companion brainstorming
exercise for which no likely causes or suggested
actions have been specified. This exercise can be
used with groups who are very new to assessment
concepts. It allows for participants to draw on their
own impressions and experiences in order to
suggest how the faculty members and staff at the
hypothetical institutions might proceed.
8 For an in-depth discussion of direct and indirect measures, their uses, and their advantages and disadvantages,
see Chapter 3.
63
Figure 14
Using Assessment Results at the Institutional Level:
Maintaining Mission and Achieving Vision
Learning
Goal:
Students will...
Appreciate the
importance of
civic
responsibility
Direct & Indirect
Measures
* First-year experience
student satisfaction
survey
Possible Reason
or Hypothesis
Outcome
Students express strong
dissatisfaction with a
first-year community
service requirement.
* Focus groups
* Alumni survey
Exhibit
competitive
career potential
* Data from benchmark
institutions
* Students do not
see relevance to
chosen career or
major.
* Students have
time or
transportation
constraints.
Graduates’ salaries five
years post graduation
are lower than those of
students with
comparable positions
who attended other
institutions.
* Students are not
trained in salary
negotiation.
Action Taken
* Introduce student/alumni-run
seminars about how community service
was relevant for them.
* Include references and examples
related to community service in general
education courses.
*Provide transportation, offer credit or
work-study funds.
* Offer opportunities for mock
interview and mock raise requests.
* Survey employers to determine cause.
* Students are
under-prepared
relative to other
institutions.
* Change curricula as a result of
employer survey.
* First-year retention
rates
Complete the
first-year
successfully
* General education
course grades
* First-yearexperience student
satisfaction survey
A first-year retention
problem is traced to
poor performance in
general education
courses.
Poor writing or
analytic skills of
students entering
the program
impede
performance.
Change course sequence to offer
writing, critical thinking course first.
* Exit interviews for
non-returning students
* Honors program
retention figures
Complete the
Honors
Program
successfully
* Analysis of course
availability and
registration statistics
* Not enough
honors courses
are available.
An honors program has
low internal retention.
*First-year-experience
survey
*Focus groups
Exhibit
independent
learning
* National Survey of
Student Engagement
(NSSE) results
* Student, faculty, and
student affairs staff
focus groups
National Survey of
Student Engagement
indicates that students
spend very little time on
academic work outside
the classroom.
64
* Courses are not
judged to be
significantly
different from
regular courses,
students do not
feel challenged.
* Students work
many hours per
week for pay.
* Students are not
being challenged
academically.
Allow first- and second-year honors
students to take upper-division honors
courses for credit.
* Allow paid internships in area
directly relevant to curriculum.
* Revamp course syllabi to require
more meaningful work outside of class.
Learning
Goal:
Students will...
Direct & Indirect
Measures
* Standardized writing
tests in first and third
years
Demontrate
high-level
writing skills
* Course registration
statistics
* Transcript analysis
Possible Reason
or Hypothesis
Outcome
Little difference
between performance of
first-year students and
juniors on a
standardized writing
test.
*Course content
analysis
* Descriptive data from
student affairs on
activity participation
Develop
leadership skills
* Student focus groups
Student participation in
campus governance is
low.
Demonstrate
career-skill
preparation and
preparedness
for job-seeking
* User activity analysis
of library resources
* Library usage survey
* Career Services
appointment logs and
attendance records for
Career Services
programs
Analyses of user
activity demonstrate
overuse of
non-academic databases
in lieu of scholarly
ones.
Very low numbers of
students make use of
career services
programs.
* Students work
many hours
off-campus for
pay.
65
* Consider course, internship, or
apprenticeship credit for student
government participation.
* Examine governance structure to
revise, if necessary, in order to
empower students.
*Students are not
being taught the
value of scholarly
sources relative to
popular sources.
* Develop a required information
literacy program that includes, among
other things, examples of erroneous
conclusions drawn from reviewing
inappropriate sources.
* Scholarly
journals and other
resources are not
widely available in
the library.
* Examine library budget with the goal
of making more appropriate resources
available.
* Students are not
aware of the
services or their
value.
* Require brief introduction to career
services at first-year orientation and
again in junior level general education
courses; include testimonials from
students who have used the services.
* Students who
have used career
services report low
satisfaction.
* Alumni surveys
*Require “writing intensive” courses
with multiple opportunities for
feedback on writing.
* Students have
infrequent
opportunities to
receive feedback
on their writing.
* Students believe
that the student
government
organization is
ineffectual.
*National Survey of
Student Engagement
(NSSE) results
Develop
proficiency in
academic and
scholarly
research skills
* Students avoid
elective courses
with a large
writing
component.
Action Taken
* Enter consortia that make scholarly
materials easily obtainable for students.
* Revamp career services program to
ensure better attainment of its goals.
Figure 15
Using Assessment Results at the Program Level:
Preparing Students for Future Success
Learning Goal:
Students will...
Be prepared for
graduate and
professional degree
programs
Communicate
competently in
the major
Integrate competently
knowledge and skills
acquired in the major
Direct and Indirect
Measures
* Departmental survey
of graduating seniors
and recent alumni
* Data from benchmark
institutions
* Scores on
faculty-developed
rubrics for final oral
exam and final report in
capstone course
* Departmental survey
of graduating seniors
* Grade distribution
analysis of senior
capstone course grades
Outcome
* Student admittance
rates to graduate and
professional programs
are low, compared to
similar institutions’
rates.
Possible Reason or
Hypothesis
* Students are not being
“coached” about the
graduate school
application process.
* Students have not been
exposed to experiences
(e.g., undergraduate
research) that enhance
their chances of graduate
school admissions.
* Student performance
in capstone courses is
poor, as measured by
rubrics for oral
presentations and
written reports.
* Students are not
receiving enough
experience in
communication in
prerequisite major
courses.
* Survey results reveal
that students think the
capstone course is an
“easy A.”
* Capstone course is
“watered-down” to
account for a variety of
previous course
experience.
* Grade distribution
reveals inflation.
66
* Capstone course does
not demand true
integration of previous
learning.
Action Taken
* Enlist junior faculty
members who have
recently finished
graduate school to
develop a coaching
program.
* Incorporate a research,
scholarship, or
practicum requirement
for students in a
graduate or professional
school “track.”
* Revamp departmental
curriculum to require
oral and written reports
in every course.
* Revamp syllabus of at
least one required course
to include multiple
communication
experiences.
* Change capstone
course from a
special-topics course to
a course that requires an
independent, integrative
project.
* Include a seminar
component that makes
students responsible for
some of the course
content.
Learning Goal:
Students will...
Direct and Indirect
Measures
Outcome
Possible Reason or
Hypothesis
Action Taken
* Schedule courses for
shorter periods, four or
five days a week.
Exhibit fluency in a
foreign language
* Scores on facultydeveloped rubrics for
oral presentation at end
of intermediate-level
courses
* Standardized oral
proficiency exam
Students in a modern
language program
exhibit low levels of
fluency in the language
when tested after having
taken several courses.
* Courses are offered
only two days a week.
* Students have few
opportunities for
sustained practice in
target language.
* Introduce dedicated
housing or separate
floors for language
students.
* Place a fluent
“graduate student in
residence” in student
housing.
* Require a study abroad
or immersion experience
for language majors.
Demonstrate applied
competency in the
major
Scores on facultydeveloped rubrics
Students’ applied
projects (e.g., design,
engineering, fine arts)
consistently reveal high
levels of quality as
evidenced by scores on
rubrics designed to
assess their efforts.
* Students are very well
prepared for applied
projects in lower-level
courses.
* Could students be
challenged even more?
* Require students to
submit their work to
external conferences or
scholarly undergraduate
journals.
* Engage upper-class
students as mentors for
lower-class students.
* A team of faculty
reassess course content.
Demonstrate
competence in
academic subject areas
in the major
Demonstrate practical
competence
in the major
Standardized
disciplinary test
* Rating forms
completed by practicum
supervisor
* Self-rating forms
completed by students
Standardized
disciplinary tests (e.g.,
the ETS subject area
tests, American
Chemical Society
Examination) reveal
“spotty” or irregular
command of the
discipline.
Students in nursing,
education, or other
programs requiring
practicum experience are
receiving lower than
desirable scores from
their supervisors.
67
* Curriculum
requirements uneven
(i.e., no required courses
in some target areas).
* Courses poorly
designed or delivered in
target area (i.e., no
permanent faculty
member in that
specialty).
* Lower-level courses
do not provide “minipracticum” experiences.
* Students have not been
made aware of the
importance of practica.
* Little guidance is
provided at practicum
site.
* Full-time faculty
create syllabi for and
mentor adjunct faculty.
* Evaluate need for
additional faculty time.
* Provide stipends for
excellent faculty to
“retool” to teach in those
content areas.
* Interact with on-site
practicum mentors to
brainstorm about reason
for the problem.
* Revise prerequisite
courses to include short
practicum assignments.
Figure 16
Using Assessment Results at the Course Level:
Ensuring Learning
[Note: Two of the goals are repeated on this chart to illustrate both “positive” and “negative” outcomes.]
Learning Goal:
Students will...
Master course
content
Direct and
Indirect
Measures
* Mid-term exam
scores
* Course evaluations
Possible Reason or
Hypothesis
Outcome
Subject mastery is
inadequate, as
demonstrated by
low in-class exam
scores.
Action Taken
* Students are not engaged
with the subject matter.
* Experiment with alternative
teaching formats (e.g.,
problem-based learning, rather
than lecture).
* Teaching format and
exam format may not be
compatible.
* Create “test-blueprints” to
acquaint students with
expectations.
* Analyze test format to determine
if it is appropriate (e.g., Is an
essay test being used when the
material was presented as a series
of facts requiring no analysis?)
and change test format if
warranted.
* Students are engaged
with material.
Master course
content
* Mid-term exam
scores
* Course evaluations
Subject mastery is
very high, as
demonstrated by
expecially high
in-class exam
scores.
* Teaching format and
exam format are
compatible.
* Students are aware of
what competencies are
necessary.
* Experiment with increasingly
more difficult exams to gauge
more accurately students’
potential.
* Provide additional and
especially challenging in-class or
out-of-class assignments and
assessment.
* But are students being
sufficiently challenged?
Exhibit
discipline-specific
writing skills
Exhibit
discipline-specific
writing skills
* Rubric-scored
writing assignments
* Course evaluations
* Rubric-scored
writing assignments
* Course evaluations
Average writing
performance is
expecially high, as
demonstrated by
rubric-scored
writing
assignments.
Average writing
performance is
unsatisfactory, as
demonstrated by
rubric-scored
writing
assignments.
68
* Writing was effectively
taught in prerequisite
classes and/or is taught
well in the target class.
* But can students be
challenged more, or can
the course focus on the
development of other
skills?
* Students were not
prepared well in
prerequisite classes.
* Students were not
receiving enough practice
and feedback in target
class.
* Engage students as mentors for
other students who need help with
writing.
* Encourage students to submit
their work to on- or off-campus
literary or scholarly outlets for
student work.
* Require writing assignments
with feedback in prerequisite
classes.
* Require writing assignments
with feedback in target class.
* Target specific aspects of
student writing and focus
assignments on them.
Learning Goal:
Students will...
Direct and
Indirect
Measures
* Commerciallydeveloped criticalthinking test
Think critically and
analytically
* Problem-solving
exercises
* Applications
problems
Demonstrate
discipline-specific
information literacy
* Rubric-scored term
paper
* Student survey or
course evaluations
Possible Reason or
Hypothesis
Outcome
Analytical skills/
critical thinking
are weak, as
demonstrated by
poor performance
on problemsolving exercises
or application
problems.
Research skills
are weak, as
demonstrated by
poor performance
on research term
paper.
* Students have received
little practice with these
skills.
* Faculty member has
limited expertise in
teaching these skills.
* Expectations for term
paper were not clearly
spelled out.
* Appropriate search
engines or sources were
not available in the library.
* Little practice with these
skills on smaller
assignments.
Exhibit Oral
Communication
Skills
Exhibit Quantitative
Skills
Exhibit High
Student Morale and
Satisfaction to
Support Learning
Rubric-scored oral
presentation
Faculty-developed
mathematics test
administered in an
accounting class
Teaching
evaluations
Oral presentation
skills are weak, as
demonstrated by
rubric-scored
presentation.
Quantitative
analysis skills are
weak.
There is
dissatisfaction
with course
format, teaching
style, and level of
course difficulty,
as demonstrated by
teaching
evaluations.
* Prerequisite courses
required little practice in
oral presentation.
* Students were not
directly informed about the
characteristics of a good
presentation and/or the
expectations for the
assignment.
No appropriate prerequiste
course is required.
* Over-dependence on
lecture format
* Little direct explanation
of the value of the course
or material.
* Few opportunities for
students to apply course
content.
* Faculty member
inaccessible outside
classroom.
69
Action Taken
* Examine prerequisite courses for
opportunities to engage in critical
thinking, and revise appropriately.
* Establish faculty-to-faculty
mentor pairs, with each faculty
member having expertise in an
area the other lacks.
* Create a “blueprint” for the
paper, clearly spelling out
expectations.
* Require an early non-graded
first draft and provide feedback.
* Examine library funding for
possible reallocation.
* Revamp required course to
include several practice
assignments drawing on scholarly
research skills in the discipline.
* Include an oral presentation
component in early required
courses with a comparable rubric
for providing feedback.
* Create teams of students to
critique each others’ presentations
before they are presented to the
whole class, and include
safeguards to make this a nonthreatening, critically constructive
situation.
Test students and assign them to a
prerequisite course, if necessary.
* Experiment in small ways with
non-lecture format (i.e., problembased learning, group projects,
“inter- teaching” in which
students are expected to explain
the material to each other).
* Include an interactive discussion
of the value of the material during
the first class period, and regularly
use applied examples to support
course material.
* Reconsider office hours
schedule, or offer students
opportunities to schedule
appointments outside of office
hours.
A Message to Faculty Members:
The Interconnectedness of Assessment, Teaching, and Learning
In the final section of this handbook we depart from the third person and move to the first and
second person in order to speak directly to faculty whose lives are so connected with students in
the classroom and in other learning situations on a daily basis.
Ø Students learn effectively when they have
opportunities to revise their work.
The staff of accrediting organizations or those who
support accreditation by volunteering their services
are very aware that an intense emphasis on student
outcomes carries with it the danger of shifting the
focus from teaching and learning to ill-considered
data-collection. Therefore, the Middle States
Commission on Higher Education advocates
sincere attention to what faculty want students to
learn; it does not advocate a “bean-counting”
approach to assessment and improvement.
If you are convinced, on a theoretical level, that
outcomes assessment is a well-intended and even a
good thing, you may be thinking that you cannot
possibly incorporate it into your classes, in your
laboratories, or in your other pedagogical
interactions with students. Yet, there are several
strategies that can be accomplished without
dramatic increases in faculty workload. Taken
together, they can transform the way students learn.
If you currently do not grade multiple drafts of
papers, projects, or lab reports, or provide
multiple critiques of artwork or performances,
consider taking at least one assignment and
building in two or more reviews. On the earlier
review or reviews, offer comments or grade with
a rubric, but give students an opportunity to
refine their work further before you assign a final
grade. If necessary, consider making the project
shorter or eliminating another graded assignment
in order to provide time for the extra grading
additional reviews will entail.
Another way to provide additional opportunities
for students to revise their work is to initiate a
system of peer review in which students share
work with each other and review it according to
pre-defined objective criteria.
The list presented in Figure 17 was developed from
a collection of recent research and wisdom on when
Ø Students learn effectively when they
and how students learn best. Nine of the eleven
understand course and program goals.
items involve practices or characteristics that
originate in individual classes. The two remaining
Human beings take in information and learn new
characteristics are also directly related to the
things much better when they have a framework
classroom. The list, for the most part, is
upon which to rest new ideas. For instance, as an
self-explanatory, and some of these characteristics
expert in your field, you can read a scholarly
can be fostered with little change in a professor’s
article much faster than can a novice. From prior
existing practices.
experience, you know why the material presented
in the article is important or not, whether it is
Here we elaborate on several of the items on the
controversial, whether it adds significantly to the
list, with an emphasis on how you can make small
current knowledge base in the area, and whether
changes by engaging in assessment.
it appears to be a reflection of solid theory in
your discipline. Without such background, the
contents of the article would be meaningless or at
least not as rich. If your syllabus has no goals or
objectives listed, you are providing your students
with no framework to help them understand
where the course fits in with other courses, how
the skills they will be acquiring translate to other
70
Figure 17
Strategies to Improve Student Learning
There is increasing evidence that students learn most effectively when:
• They understand course and program goals and the characteristics of excellent work.
• They are academically challenged and encouraged to focus on developing higher-order thinking
skills, such as critical thinking and problem solving, as well as discipline-specific knowledge.
• They spend more time actively involved in learning and less time listening to lectures.
• They engage in multidimensional “real world” tasks.
• Their learning styles are accommodated.
• They have positive interactions with faculty and work collaboratively with fellow students;
all learners—students and professors—respect and value others as learners.
• They participate in out-of-class activities, such as co-curricular activities and service learning
opportunities, that build on what they are learning in the classroom.
• Assignments and assessments are intertwined with learning activities and focus on the most
important course and program goals.
• They have opportunities to revise their work.
• They reflect on what and how they have learned.
• They have a culminating “capstone” experience, such as a seminar, internship, independent
study, research project, or thesis, that lets them synthesize what they have learned over the
course of their college experience.
Sources:
Angelo, T. A. (1993, April). A “teacher’s dozen”: Fourteen general, research-based principles for
improving higher learning in our classrooms. AAHE Bulletin, 3 (7), 13.
Barr, R. B., & Tagg, J. (1995, Nov/Dec). From teaching to learning: A new paradigm for undergraduate
education. Change, 27 (6), 12-25.
Chickering, A. W., & Gamson, Z. (1987, July). Seven principles for good practice in undergraduate
education. AAHE Bulletin, 39 (7), 5-10.
Kuh, G. (2001, May/June). Assessing what really matters to student learning: Inside the National Survey of
Student Engagement. Change, 33 (3), 10-17, 66.
Mentkowski, M. & Associates. (2000). Learning that lasts: Integrating learning, development, and
performance in college and beyond. San Francisco: Jossey-Bass.
Pascarella, E. T., & Terenzini, P. T. (1991). How college affects students: Findings and insights from
twenty years of research. San Francisco: Jossey-Bass.
What research says about improving undergraduate education. (1996, April). AAHE Bulletin, 48 (8), 5-8.
© 2002, Copyright by Linda Suskie. Reproduced with permission.
71
domains, or why they should be interested in the
topic.
However, if objectives are listed, and they are
made meaningful through a discussion of their
importance and of the students’ own objectives
for the course, you will be modeling the behavior
you use in your own scholarship. If you extend
the discussion further and ask students to reflect
on how class activities meet the objectives, you
will be offering them a framework to evaluate all
of their learning experiences, including those in
other classes and in their future educational
experiences.
Ø Students learn most effectively when they
(and you) reflect on what and how they have
learned.
If you have well-articulated goals for your course
and if you have engaged students in a
conversation about the importance of the course
and its relationship to other aspects of their
college program, the next step is to ensure that
the pedagogical content of the course leads to
achieving those goals. One technique that is not
time consuming is to spend five minutes or so
before each class listing the course goals or
objectives for the class. Keep the record of your
reflections for one semester, and at the end, look
at it in aggregate. Are the goals supported
relatively evenly? Is one goal over-represented
(e.g., Is more time devoted to content than to
process than seems appropriate given the goals)?
Is there evidence that a large portion of time is
spent on topics or activities that are not directly
related to a course goal? Use the data you have
collected to revise the course, perhaps only
slowly over the course of several semesters, in
order to help steer it toward the intended goals.
Ø Students learn most effectively when
assignments and assessments that are directly
relevant to course goals are intertwined with
learning activities and focus on the most
important course and program goals.
Consider an assignment that you give regularly
or an exam format that you use most often. Does
the assignment draw on or help teach the
72
particular capabilities that you are hoping to
nurture in your students? Perhaps you assign a
term paper each semester on a topic related to the
subject area of your course. Is the sole purpose of
the assignment to have students gain an in-depth
knowledge of a particular topic? Do you have
some tacit goals such as teaching students to
engage in scholarly research, develop writing
skills, or learn appropriate citation
techniques—or is your goal simply to evaluate
what they have been able to learn on their own?
If these goals are tacit, rather than explicit, it is
unlikely that your students will be aware of them,
and thus unlikely that they will organize their
work around them. If students are unable to
connect the assignment to specific goals, the term
paper is probably not an assessment of what they
have learned in your class but, rather, an
assessment of their “natural” competencies or
characteristics, or prior learning.
In the case of examinations, many teaching
faculty develop the habit of using the same
format for exams over and over, without thinking
of the congruence between what they want
students to learn and the way they are assessing
the learning. The format used for examinations
may be the one that they found most congenial
when they were students, the one that comes in a
test bank with the text, or the one that is quickest
to prepare or grade. However, every examination
format has positive and negative characteristics.
For example, multiple-choice exams can be more
psychometrically sound—more reliable and
valid—if they are developed properly, than
casually developed essay exams. Conversely,
essay exams can sometimes allow for the
assessment of a deeper, more synthetic (or
analytic) exposition and certainly provide a
venue for students to demonstrate their creativity
or unique perspectives. Yet, no examination is
valid if it is measuring something other than what
it is intended to measure. Students frequently
lament, “We didn’t know what you were going to
put on the exam” or “You didn’t teach us the
answers to the questions on the exam.” These
students may be more sophisticated and less lazy
than they appear. What students really might be
saying is that this particular class did not prepare
them to take this particular exam.
Some students, nevertheless, perform
exceedingly well. These students would be a
useful group to shed light on the connection
between the class and the exam. Do they feel
they could have answered some of the questions
without ever having taken the course? This can
happen in the case of essay questions that
involve the use of logic or synthesis more than
they do course content, or for which course
content could have been gleaned outside of class.
Do high- performing students feel that they could
have done well on a multiple-choice test simply
by memorizing the content of a text? An
instructor may deliver brilliant and engaging
lectures, but if he gives an exam on the content
of the text instead, he has no assessment of what
students learned during those great lectures.
absolute performance. Some professors probably
do grade generously hoping to receive good
student evaluations. Even when professors’
grades are meant to indicate mastery, we do not
know which student mastered the material as a
result of the course and which student had
already mastered it before her arrival. It is clearly
beyond the scope of a handbook on student
learning assessment to solve the complex
problem of incongruence among professors’
grading systems. However, regardless of your
perspective on grade inflation and within the
context of your own class, you can ensure that
grades are meaningful so that students recognize
the characteristics of excellent work. After all, it
is what the student learns, not the grade itself,
that is most important.
A brief look at your assignments and exams
relative to your goals and the content of your
class presentations can be accomplished in a very
short period of time. Greater insight can be
gained by conducting some 10- or 15-minute
“focus groups” with students about the
congruence between the course and the
assessments. Any changes that appear to be
indicated can be made one at a time or over the
course of a few semesters.
Test blueprints are one way to make grades
meaningful (see Chapter 3). The professor plots
out what it is that a student is expected to master,
and the professor might even create a hierarchy
of mastery tied to various grades. After a student
takes an examination, she can go back to the
blueprint and reflect on her grade relative to the
concepts she was expected to have mastered.
Ø Students learn most effectively when they
understand the characteristics of excellent
work.
The debate about grade inflation is complicated,
although most professors probably identify easily
with a specific perspective. The first common
perspective is that if we would all just use a
normal curve to grade, we wouldn’t have grade
inflation. The second perspective is that if we
didn’t tie teaching evaluations to faculty tenure
and promotion, we wouldn’t have grade
inflation. The third perspective is that grade
inflation is not necessarily a problem. If we are
teaching for mastery—i.e. achievement of
learning goals—then many students will earn
high grades. Without taking sides, it is easy to
make the case that grades mean very different
things to the outside world, depending upon who
assigns them. In a curve system, no one except
the professor knows anything about students’
73
Rubrics (see Chapter 3) make the grading
process more transparent, more accessible, and
when well formulated, they are diagnostic for
the student (Walvoord & Anderson, 1998).
In reviewing his rubric scores, a student can
pinpoint his areas of strength and weakness and
develop his own learning goals to strengthen
performance in specific areas.
Another accessible way to make grades more
meaningful, although less structured than a
rubric, is to write relevant and instructive
comments on student work, including
multiple-choice exams! Surprisingly, most
students find few comments on their work that
provide meaningful feedback, relegating tests
and assignments to the category of summative
assessment. It does take longer to write
comments, but if they are important and
strategically placed, there needn’t be many.
How could one write meaningful comments on a
multiple-choice exam? Questions on these
exams fall into several categories, including
memorization (Did the student acquire necessary
facts?); application (Given a scenario, can the
student classify it, or extrapolate from it?);
inference (If this happens, then what would
happen?); and analogy (A is to b, as c is to d?).
Students tend to err in patterns, frequently
missing questions of the same type. Some
guidance on which types of questions are causing
difficulty for each student will almost certainly
help him to focus his attention on the problem
area next time.
assimilating easily information in this format
may be learning less in the classroom than their
counterparts who “prefer” such a format. This is
an over-simplification of the learning styles
approach, but the concept is not complicated.
Different students learn best in different ways
and in different settings, and failure to take this
into account can impede learning in some
students.
For a variety of reasons, we are not advocating
that professors assess each student’s learning
profile and make associated individual
accommodations (see Appendix 8). However, any
instructor can diversify his or her presentation
and the types of experiences associated with a
specific class to increase the chances of engaging
students who may not thrive in a lecture-based
classroom. Knowledge about learning styles can
be used to modify other aspects of a course as
well. Hartman (1995) suggests that putting
students together in groups to work on a project
can have the best outcomes when students’
learning styles are complementary, rather than
similar. For instance, a group of students who all
prefer theory or abstract thinking to concrete
approaches may have difficulty organizing and
getting started on the project, whereas another
group of students, all of whom prefer detail and
information- gathering over integration of ideas,
may complete the project more efficiently, but
may produce a less synthesized or mature
product. Combining students with these styles
will mix the best of both approaches.
As with the other areas discussed above,
reflection on the meaning of grading—both the
“score” and the comments offered—needn’t take
a great deal of time, and modifications in one’s
practices can be made gradually.
Ø Students learn most effectively when their
learning styles are accommodated.
Recent pedagogical literature is replete with
information about the varieties of learning styles
that students can exhibit. Much of the work on
learning styles has its genesis in traditional
personality theory and its modern applied
counterparts, such as the Myers-Briggs Type
Indicator (Briggs-Myers, 1992), which classifies
personality along four dimensions and yields
sixteen types. Other heuristic impetus for work in
the area of student learning styles comes from the
theory of Kolb (1984) who defined four
constituent processes involved in learning,
Gardner (1983) who developed the now
well-known concept of “multiple intelligences,”
and Sternberg (1988) who posited three distinct
kinds of intelligence. Appendix 8 offers some
background information on learning styles in the
form of Frequently Asked Questions (FAQs) for
those who are interested in exploring this topic in
greater depth.
The focus of the learning styles approach as it
applies to teaching is that all students learn
differently, either because of their personalities
or their diverse patterns of cognitive ability, and
that teaching formats within a class should be
equally diverse to accommodate students’
learning needs. Because the most commonly
used teaching method is the lecture-discussion
format, some students with either personality
types or learning profiles that prevent them from
In parting...
The purpose of defining goals and assessing
learning is to improve learning through teaching.
Teaching lies primarily in the hands of faculty
members, and good learning cannot happen without
their commitment and dedication. Assessment, first
and foremost, is a tool for faculty members to use
as they do their very best to teach their students
well.
74
Appendix 1
Assessment Standards in
Characteristics of Excellence in Higher Education
Standard 7: Institutional Assessment
fundamental aspect of institutional effectiveness?”
Because student learning is a fundamental
The institution has developed and implemented component of the mission of most institutions of
an assessment process that evaluates its overall
higher education, the assessment of student learning
effectiveness in achieving its mission and goals
is an essential component of the assessment of
and its compliance with accreditation standards. institutional effectiveness and is the focus of
Standard 14 (Assessment of Student Learning).
Self-studies can thus document compliance with
Context
Standard 7 by summarizing the assessments of each
accreditation standard into conclusions about the
Assessment may be characterized as the third
institution’s overall achievement of its key goals.
element of a four-step planning-assessment cycle:
The fundamental question asked in the
1. Developing clearly articulated written
accreditation process is, “Is the institution fulfilling
statements, expressed in observable terms, of key
its mission and achieving its goals?” This is
institutional and unit-level goals that are based on
precisely the question that assessment is designed
the involvement of the institutional community, as
to answer, making assessment essential to the
discussed under Standard 1 (Mission and Goals);
accreditation process. Assessment processes help to
ensure the following:
2. Designing intentional objectives or strategies to
achieve those goals, as discussed under Standard 2
• Institutional and program-level goals are
(Planning, Resource Allocation, and Institutional
clear to the public, students, faculty, and
Renewal);
staff;
3. Assessing achievement of those key goals; and
• Institutional programs and resources are
organized and coordinated
4. Using the results of those assessments to improve
to achieve institutional and program-level
programs and services, as discussed under Standard
goals;
2 (Planning, Resource Allocation, and Institutional
Renewal), with appropriate links to the institution’s
• The institution is indeed achieving its
ongoing planning and resource allocation
mission and goals; and
processes.
• The institution is using assessment results to
The effectiveness of an institution rests upon the
improve student learning and otherwise
contribution that each of the institution’s programs
advance the institution.
and services makes toward achieving the goals of
the institution as a whole. This standard on
While the Commission expects institutions to assess
institutional assessment thus builds upon all other
institutional effectiveness, it does not prescribe a
accreditation standards, each of which includes
specific approach or methodology. The institution
periodic assessment of effectiveness as one of its
fundamental elements. This standard ties together is responsible for determining its expected goals
and the objectives or strategies for achieving them
those assessments into an integrated whole to
at each level (institutional and unit), assessment
answer the question, “As an institutional
approaches and methodologies, sequence, and time
community, how well are we collectively doing
frame. These may vary, based on the mission,
what we say we are doing?” and, in particular,
goals, organization, and resources of the institution.
“How do we support student learning, a
75
Whatever the approach, effective assessment processes are useful, cost-effective, reasonably accurate
and truthful, carefully planned, and organized, systematic, and sustained.
Useful assessment processes help faculty and staff make appropriate decisions about improving
programs and services, developing goals and plans, and making resource allocations. To assist with
interpretation and use of assessment results, assessment measures and indicators have defined
minimally acceptable performance targets. Because institutions, their students, and their
environments are continually evolving, effective assessments cannot be static; they must be
reviewed periodically and adapted in order to remain useful.
Cost-effective assessment processes yield dividends that justify the institution’s investment in them,
particularly in terms of faculty and staff time. To this end, institutions may begin by considering
assessment measures, indicators, “flags,” and “scorecards” already in place, such as retention,
graduation, transfer, and placement rates, financial ratios, and surveys. New or refined measures
may then be added for those goals and objectives for which evidence of achievement is not already
available, concentrating on the institution’s most important goals. Effective assessments are simple
rather than elaborate and may focus on just a few key goals in each program, unit, and curriculum.
Reasonably-accurate and truthful assessment processes yield results that can be used with
confidence to make appropriate decisions. Because there is no one perfectly accurate assessment tool
or strategy, institutions should use multiple kinds of measures to assess goal achievement.
Assessments may be quantitative and/or qualitative and developed locally or by an external
organization. All assessment tools and strategies should clearly relate to the goals they are assessing
and should be developed with care; they should not be merely anecdotal information nor collections
of information that happen to be on hand. Strategies to assess student learning should include
direct—clear, visible, and convincing—evidence, rather than solely indirect evidence of student
learning such as surveys and focus groups.
Planned assessment processes that purposefully correspond to institutional goals that they are
intended to assess promote attention to those goals and ensure that disappointing outcomes are
appropriately addressed. Institutions often have a variety of plans, such as a strategic plan, academic
plan, financial plan, enrollment plan, capital facilities master plan, and technology plan. Just as such
plans should be interrelated to ensure that they work synergistically to advance the institution,
assessments should also be interrelated. At many institutions, effective institutional planning begins
with academic planning, which in turn drives the other plans. If the academic plan calls for a new
academic program, for example, the technology plan should ensure faculty and students in the new
program will be able to use appropriate instructional technologies. Assessments of the technology
plan should evaluate not just whether instructional technologies have been put in place but also how
effectively those technologies have helped students to achieve the program’s key learning outcomes.
Organized, systematized, and sustained assessment processes are ongoing, not once-and-done.
There should be clear interrelationships among institutional goals, program- and unit-level goals,
and course-level goals. Assessments should relate clearly to important goals, and improvements
should clearly stem from assessment results.
As noted earlier, because student learning is a fundamental component of the mission of most
institutions of higher education, the assessment of student learning is an essential component of the
assessment of institutional effectiveness. An institution may therefore create institutional effectiveness
documentation that includes a component on assessing student learning (see Standard 14: Assessment
of Student Learning), or it may create a bridge between two separate sets of documentation, one for the
assessment of student learning and one for other aspects of institutional effectiveness.
76
w maximize the use of existing data and
information;
A commitment to the assessment of institutional
effectiveness requires a parallel commitment to
ensuring its use. Assessment information, derived
in a manner appropriate to the institution and to its
desired outcomes, should be available to and used
by those who develop institutional goals and carry
out strategies to achieve them. As discussed under
Standard 2 (Planning, Resource Allocation, and
Institutional Renewal), an accredited institution
uses the results of assessment for institutional
renewal: to maintain, support, and improve its
programs and services. Assessment information
should be used as a basis for assessing the
institution’s effectiveness in achieving its stated
goals, for monitoring and improving the
environment for student learning, and for enhancing
overall student success; to these ends, it should be
linked to the institution’s ongoing planning and
resource allocation processes.
w clearly and purposefully relate to the
goals they are assessing;
w are of sufficient quality that results can
be used with confidence to inform
decisions;
² support and collaboration of faculty and
administration;
² clear realistic guidelines and a timetable,
supported by appropriate investment of
institutional resources;
² sufficient simplicity, practicality, detail,
and ownership to be sustainable;
² periodic evaluation of the effectiveness
and comprehensiveness of the institution’s
assessment process;
Assessment results also should be used to evaluate
the assessment process itself, leading to
modifications that improve its relevance and
effectiveness.
Ø evidence that assessment results are shared
and discussed with appropriate constituents
and used in institutional planning, resource
allocation, and renewal (see Standard 2:
Planning, Resource Allocation, and
Institutional Renewal) to improve and gain
efficiencies in programs, services and
processes, including activities specific to the
institution’s mission (e.g., service, outreach,
research); and
Fundamental Elements of
Institutional Assessment
An accredited institution is expected to possess or
demonstrate the following attributes or activities:
Ø documented, organized, and sustained
assessment process to evaluate and improve
the total range of programs and services;
achievement of institutional mission, goals,
and plans; and compliance with accreditation
standards that meets the following criteria:
² a foundation in the institution’s mission
and clearly articulated institutional,
unit-level, and program-level goals that
encompass all programs, services, and
initiatives and are appropriately integrated
with one another (see Standards 1:
Mission and Goals and 2: Planning,
Resource Allocation, and Institutional
Renewal);
Ø written institutional (strategic) plan(s) that
reflect(s) consideration of assessment results.
Institutions and evaluators must consider the
totality that is created by the fundamental elements
and any other relevant institutional information or
analysis. Fundamental elements and contextual
statements should not be applied separately as
checklists. Where an institution does not possess or
demonstrate evidence of a particular Fundamental
Element, the institution may demonstrate through
alternative information and analysis that it meets
the standard.
² systematic, sustained, and thorough use of
multiple qualitative and/or quantitative
measures that:
77
recommendations for improvement, and
evidence of action based on
recommendations;
Optional Analysis and Evidence
In addition to the evidence inherent within or
necessary to document the fundamental elements
above, the following, although not required, may
facilitate the institution’s own analysis relative to
this accreditation standard:
Ø evidence that institutional assessment
findings are used to:
² improve student success;
² review and improve programs and
services;
Ø analysis of the institutional culture for
assessing institutional effectiveness,
including:
² plan, conduct, and support professional
development activities;
² the views of faculty and administrators on
assessment;
² assist in planning and budgeting for the
provision of programs and services;
² faculty and administrators’ understanding
of their roles in assessing institutional
effectiveness;
² support decisions about strategic goals,
plans, and resource allocation;
² campus-wide efforts to encourage,
recognize, and value efforts to assess
institutional effectiveness and to improve
programs and services;
² inform appropriate constituents about the
institution and its programs;
Ø evidence of renewal strategies, made in
response to assessment results [included also
under Standard 2 Optional Analyses]; or
Ø analysis of the quality and usefulness of
institutional support for assessment efforts,
including the quality and usefulness of:
Ø analysis of evidence that renewal strategies
made in response to assessment results have
had the desired effect in improving
programs, services, and initiatives.
² written statements of expectations for
assessment work;
² policies and governance structures to
support institutional assessment;
² administrative, technical, and financial
support for institutional assessment
activities;
² professional development opportunities
and resources for faculty and staff to learn
how to assess institutional effectiveness
and how to use the results;
Ø clear, appropriate criteria for determining
whether key institutional goals and
objectives have been achieved;
Ø analysis of whether the institution has
sufficient, convincing, written evidence that
it is achieving its mission and its key
institutional goals;
Ø analysis of results of surveys of students and
other relevant groups;
Ø review of evaluations of special, mission
driven programs or projects, with
78
Because student learning is at the heart of the
mission of most institutions of higher education,
the assessment of student learning is an essential
component of the assessment of institutional
effectiveness (see Standard 7: Institutional
Assessment), which additionally monitors the
environment provided for teaching and learning
and the achievement of other aspects of the
institution’s mission, vision, and strategic goals
and plans.
Standard 14: Assessment of
Student Learning
Assessment of student learning demonstrates
that, at graduation, or other appropriate points,
the institution’s students have knowledge, skills,
and competencies consistent with institutional
and appropriate higher education goals.
Context
The fundamental question asked in the
accreditation process is, “Is the institution fulfilling
its mission and achieving its goals?” This is
precisely the question that assessment is designed
to answer, making assessment essential to the
accreditation process. Assessment processes help to
ensure the following:
Assessment of student learning may be
characterized as the third element of a four-step
teaching-learning-assessment cycle:
1. Developing clearly articulated written
statements, expressed in observable terms, of key
learning outcomes: the knowledge, skills, and
competencies that students are expected to exhibit
upon successful completion of a course, academic
program, co-curricular program, general education
requirement, or other specific set of experiences, as
discussed under Standard 11 (Educational
Offerings);
• Institutional and program-level goals are
clear to the public, students, faculty, and
staff;
• Institutional programs and resources are
organized and coordinated to achieve
institutional and program-level goals;
• The institution is providing academic
opportunities of quality;
2. Designing courses, programs, and experiences
that provide intentional opportunities for students to
achieve those learning outcomes, again as discussed
under Standard 11;
• The institution is indeed achieving its
mission and goals; and
3. Assessing student achievement of those key
learning outcomes; and
4. Using the results of those assessments to improve
teaching and learning.
This standard on assessment of student learning
builds upon Standards 11 (Educational Offerings),
12 (General Education), and 13 (Related
Educational Offerings), each of which includes
assessment of student learning among its
fundamental elements. This standard ties together
those assessments into an integrated whole to
answer the question, “Are our students learning
what we want them to learn?” Self-studies can thus
document compliance with
Standard 14 by summarizing the assessments of
Standards 11 through 13 into conclusions about
overall achievement of the institution’s key student
learning outcomes.
• Assessment results help the institution to
improve student learning and otherwise
advance the institution.
Assessment is not an event but a process that is an
integral part of the life of the institution, and an
institution should be able to provide evidence that
the assessment of student learning outcomes and
use of results is an ongoing institutional activity.
While some of the impact of an institution on its
students may not be easily or immediately
measured—some institutions, for example, aim for
students to develop lifelong habits that may not be
fully developed for many years—the overall
assessment of student learning is expected whatever
the nature of the institution, its mission, the types of
programs it offers, or the manner in which its
educational programs are delivered and student
learning facilitated.
79
While the Commission expects institutions to assess
student learning, it does not prescribe a specific
approach or methodology. The institution is
responsible for determining its expected learning
outcomes and strategies for achieving them at each
level (institutional, program, and course),
assessment approaches and methodologies,
sequence, and time frame. These may vary, based
on the mission, goals, organization, and resources
of the institution. Whatever the approach, effective
assessment processes are useful, cost-effective,
reasonably accurate and truthful, carefully planned,
and organized, systematic, and sustained.
² Because there is no one perfectly accurate
assessment tool or strategy, institutions
should use multiple kinds of measures to
assess goal achievement. Assessments may
be quantitative and/or qualitative and
developed locally or by an external
organization.
² Assessment tools and strategies should be
developed with care; they should not be
not merely anecdotal information nor
collections of information that happen to
be on hand.
² Student learning assessment processes
should yield direct—clear, visible, and
convincing—evidence of student learning.
Tangible examples of student learning,
such as completed tests, assignments,
projects, portfolios, licensure
examinations, and field experience
evaluations, are direct evidence of student
learning. Indirect evidence, including
retention, graduation, and placement rates
and surveys of students and alumni, can be
vital to understanding the teachinglearning process and student success
(or lack thereof), but such information
alone is insufficient evidence of student
learning unless accompanied by direct
evidence. Grades alone are indirect
evidence, as a skeptic might claim that
high grades are solely the result of lax
standards. But the assignments and
evaluations that form the basis for grades
can be direct evidence if they are
accompanied by clear evaluation criteria
that have a demonstrable relationship to
key learning goals.
Useful assessment processes help faculty and
staff make appropriate decisions about
improving programs and services, developing
goals and plans, and making resource
allocations. To assist with interpretation and use
of assessment results, assessment measures and
indicators have defined minimally acceptable
performance targets. Because institutions, their
students, and their environments are continually
evolving, effective assessments cannot be static;
they must be reviewed periodically and adapted
in order to remain useful.
Cost-effective assessment processes are
designed so that their value is in proportion to
the time and resources devoted to them. To this
end, institutions can begin by considering
assessment measures already in place, including
direct evidence such as capstone projects, field
experience evaluations, and performance on
licensure examinations and indirect evidence
such as retention and graduation rates and
alumni surveys. New or refined measures can
then be added for those learning outcomes for
which direct evidence of student learning is not
already available, concentrating on the most
important institutional and program-level
learning outcomes. Effective assessments are
simple rather than elaborate and may focus on
just a few key goals in each program, unit, and
curriculum.
Planned assessment processes that clearly and
purposefully correspond to learning outcomes
that they are intended to assess promote
attention to those goals and ensure that
disappointing outcomes are appropriately
addressed.
Reasonably-accurate and truthful assessment
processes yield results that can be used with
confidence to make appropriate decisions. Such
assessment processes have the following
characteristics:
Organized, systematized, and sustained
assessment processes are ongoing, not
once-and-done. There should be clear
interrelationships among institutional goals,
program- and unit-level goals, and course-level
80
goals. Assessments should clearly relate to
important goals, and improvements should
clearly stem from assessment results.
all programs that aim to foster student
learning and development, that are:
² appropriately integrated with one another;
As noted earlier, because student learning is a
fundamental component of the mission of most
institutions of higher education, the assessment of
student learning is an essential component of the
assessment of institutional effectiveness. An
institution may therefore create institutional
effectiveness documentation that includes a
component on assessing student learning (see
Standard 14: Assessment of Student Learning), or it
may create a bridge between two separate sets of
documentation, one for the assessment of student
learning and one for other aspects of institutional
effectiveness.
The improvement of overall educational quality and
the enhancement of effective teaching and learning
is most likely to occur when faculty and
administrators work together to implement a sound,
institution-wide program of assessment. Because
the faculty guide decisions about curriculum and
pedagogy, the effective assessment of student
learning is similarly guided by the faculty and
supported by the administration.
A commitment to assessment of student learning
requires a parallel commitment to ensuring its use.
Assessment information, derived in a manner
appropriate to the institution and its desired
academic outcomes, should be available to and used
by those who develop and carry out strategies that
will improve teaching and learning.
² consonant with the institution’s mission;
and
² consonant with the standards of higher
education and of the relevant disciplines;
Ø a documented, organized, and sustained
assessment process to evaluate and improve
student learning that meets the following
criteria:
² systematic, sustained, and thorough use of
multiple qualitative and/or quantitative
measures that:
w maximize the use of existing data and
information;
w clearly and purposefully relate to the
goals they are assessing;
w are of sufficient quality that results can
be used with confidence to inform
decisions; and
w include direct evidence of student
learning;
² support and collaboration of faculty and
administration;
² clear, realistic guidelines and timetable,
supported by appropriate investment of
institutional resources;
² sufficient simplicity, practicality, detail,
and ownership to be sustainable; and
Assessment results should also be used to evaluate
the assessment process itself, leading to
modifications that improve its relevance and
effectiveness.
² periodic evaluation of the effectiveness
and comprehensiveness of the institution’s
student learning assessment processes;
Fundamental Elements of
Assessment of Student Learning
Ø assessment results that provide sufficient,
convincing evidence that students are
achieving key institutional and program
learning outcomes;
An accredited institution is expected to possess or
demonstrate the following attributes or activities.
Ø evidence that student learning assessment
information is shared and discussed with
appropriate constituents and is used to
improve teaching and learning; and
Ø clearly articulated statements of expected
student learning outcomes (see Standard 11:
Educational Offerings), at all levels
(institution, degree/program, course) and for
81
Ø documented use of student learning
assessment information as part of institutional
assessment.
Ø analysis of the use of student learning
assessment findings to:
Institutions and evaluators must consider the
totality that is created by the fundamental elements
and any other relevant institutional information or
analysis. Fundamental elements and contextual
statements should not be applied separately as
checklists. Where an institution does not possess or
demonstrate evidence of a particular Fundamental
Element, the institution may demonstrate through
alternative information and analysis that it meets
the standard.
² improve pedagogies, curricula and
instructional activities;
Optional Analysis and Evidence
² support other institutional assessment
efforts (see Standard 7: Institutional
Assessment) and decisions about strategic
goals, plans, and resource allocation; and
² assist students in improving their learning;
² review and revise academic programs and
support services;
² plan, conduct, and support professional
development activities;
² assist in planning and budgeting for the
provision of academic programs and
services;
In addition to the evidence inherent within or
necessary to document the fundamental elements
above, the following, although not required, may
facilitate the institution’s own analysis relative to
this accreditation standard:
² inform appropriate constituents about the
institution and its programs;
² written statements of expectations for
student learning assessment work;
Ø analysis of evidence that improvements in
teaching, curricula, and support made in
response to assessment results have had the
desired effect in improving teaching, learning,
and the success of other activities;
² policies and governance structures to
support student learning assessment;
Ø analysis of the institutional culture for
assessing student learning, including:
Ø analysis of institutional support for student
learning assessment efforts, including:
² administrative, technical, and financial
support for student learning assessment
activities and for implementing changes
resulting from assessment; and
² the views of faculty and institutional
leaders on assessment;
² professional development opportunities
and resources for faculty to learn how to
assess student learning, how to improve
their curricula, and how to improve their
teaching;
² the quality and usefulness of institutional
support for student learning assessment
efforts;
² faculty members’ understanding of their
roles in assessing student learning;
² campus-wide efforts to encourage,
recognize, and value efforts to assess
student learning and to improve curricula
and teaching;
Ø analysis of the clarity and appropriateness of
standards for determining whether key
learning outcomes have been achieved;
² evidence of collaboration in the
development of statements of expected
student learning and assessment strategies;
Ø evidence of workable, regularized,
collaborative institutional processes and
protocols for ensuring the dissemination,
analysis, discussion, and use of assessment
results among all relevant constituents within
a reasonable schedule;
Ø evidence that information appropriate to the
review of student retention, persistence, and
attrition, is used to reflect whether these are
consistent with student and institutional
82
expectations [also included in Standard 8
Optional Analyses];
Ø evidence of the utilization of attrition
information to ascertain characteristics of
students who withdraw prior to attaining their
educational objectives and, as appropriate,
implementation of strategies to improve
retention [also included under Optional
Analyses in Standard 8];
Ø analysis of teaching evaluations, including
identification of good practices; or
Ø analysis of course, department or school
reports on classroom assessment practices and
their outcomes, including grading approaches
and consistency.
83
Appendix 2
Enhancing the Campus Climate for Assessment:
Questions for Academic Leaders
What is your personal commitment to assessment?
q Are you sufficiently familiar with current thinking about the principles and practice of assessment?
q Are you comfortable with the concept of assessment? Have you worked through any reservations
you have about assessment?
q Do you understand why assessment is important?
q Are you personally committed to sharing leadership of assessment with the faculty?
How do you stimulate interest in assessment?
q Do you promote assessment when you talk formally and informally with faculty, students, and staff?
q Do you sponsor consultants, speakers, and forums on assessment? Do you support these programs
with your active presence?
q Do you explain to faculty, students, and staff how assessment findings affect major decisions that
you and your colleagues make?
q Do you have communication channels with your campus assessment committee(s)? Do you actively
use them to promote assessment?
How do you help provide the people who will help the campus focus on assessment?
q Do you see faculty vacancies as an opportunity to move substantively toward promoting a
learning-centered environment?
q Do you give hiring preference to faculty applicants who have documented success in creating a
learning-centered environment for their students and in using assessment to strengthen teaching and
learning?
q Do you ask faculty and staff applicants to demonstrate their skills in promoting active learning and
their skills in assessment?
How do you give faculty incentives to focus on assessment?
q Do you offer ample incentives for faculty and staff (e.g., promotion/tenure/merit considerations,
reassigned time, budget supplements) to refocus their work in ways that promote a learning-centered
environment and/or strengthen assessment?
q Are you promoting a learning-centered environment and strengthening assessment major goals for
your institution?
q Are you promoting a learning-centered environment and strengthening assessment major goals for
you personally?
q Do you require proposals for new programs to include plans for assessing student learning?
Continued on next page ã
84
How do you provide the training to enable faculty to strengthen assessment?
q Do you encourage your faculty and staff to participate in campus and off-campus professional
development programs on assessment?
q Do you alert your campus’s teaching/learning center and/or assessment officer to faculty and staff
needs for professional development on assessment?
q Do you fund faculty and staff travel to assessment conferences, institutes, and workshops?
How do you provide the resources to enable faculty to strengthen assessment?
q Do you give any special funding to programs that make the most progress in strengthening and using
assessment?
q Do you provide both “seed money” and sustained or other special funding for initiatives that
significantly strengthen assessment?
q Do you encourage your institution to give priority to fundraising for programs and activities that
make assessment a priority?
q Do you encourage and honor faculty who seek grants for resources that will promote a
learning-centered environment and strengthen assessment?
How do you help the faculty focus their time on assessment?
q Do you make assessment a focus of program reviews?
q Do you encourage departments to set department goals that contribute substantively toward
promoting a learning-centered environment and strengthening assessment?
q Do you encourage and reward scholarship of teaching as a scholarly activity?
q Do you help faculty find the time for assessment initiatives by helping to minimize paperwork and by
relieving them of less-critical responsibilities?
How do you encourage measurable outcomes of assessment endeavors?
q Do you track the number of programs that make major progress in strengthening assessment?
q Do you track the percent of courses/sections that use the most appropriate assessment tools and
strategies?
q Do you track the percent of students who participate in the embedded assessments of higher-order
thinking skills?
q Do you track resource utilization to see how well it supports assessment?
q Do you develop other key performance indicators for assessment?
How do you celebrate and reward assessment achievements?
q Do you announce noteworthy student and faculty accomplishments to internal and external
constituents?
q Do you create celebrations of assessment achievements, consistent with campus culture?
q Do you provide special resources (e.g., revenue sharing) to those making extraordinary contributions
to assessment?
© 2002, Copyright by Linda Suskie. Reproduced and modified with permission.
85
Appendix 3
Assessment Practices Quiz
Mark a T next to those statements that accurately describe the Middle States Commission’s views on
assessment of student learning. Mark an F next to those statements that do not accurately describe the
Commission’s views on assessment of student learning.
1.
_____
Published tests are always preferable to locally developed assessment measures.
2.
_____
Class assignments can be used to assess the learning goals of academic programs.
3.
_____
Tests with numeric scores are preferable to qualitative measures such as focus groups.
4.
_____
Surveys of student satisfaction with a program are insufficient evidence of what students
have learned.
5.
_____
Every learning outcome of every course and program must be assessed.
6.
_____
All students should be assessed to demonstrate the effectiveness of a course or program;
a sample of students is inadequate.
7.
_____
Goals should not be changed after they are selected.
8.
_____
The same assessment measures should be used in every assessment cycle.
9.
_____
Grades alone are not direct evidence of student learning.
10. _____
The primary purposes of assessment are to maintain accreditation and satisfy external
stakeholders; therefore, it is appropriate to schedule assessment cycles so that they
coincide with accreditation self-studies.
11. _____
The most effective way to create an assessment plan is to adopt the assessment plan
of another institution.
12. _____
Assessment efforts should focus on what is learned in academic courses and programs;
assessing what is learned in out-of-class activities is not important.
13. _____
Self-report measures can yield useful information about student learning.
14. _____
The assessment of educational effectiveness and the assessment of institutional effectiveness
are not related.
86
Appendix 4
Key to “Assessment Practices Quiz”
1.
False Published tests are always preferable to locally developed assessment measures. [Both published and
locally developed instruments have pros and cons, and both may have a place in an assessment program.]
2.
True
Class assignments can be used to assess the learning goals of academic programs. [Class assignments,
especially in senior capstone courses, can be valuable sources of “embedded” information on how well
students are achieving the major goals of a program.]
3.
False Tests with numeric scores are preferable to qualitative measures such as focus groups. [Qualitative and
quantitative measures offer different perspectives to an assessment program, and both can be valuable.]
4.
True
Surveys of student satisfaction with a program are insufficient evidence of what students have learned.
[Student satisfaction surveys are not direct measures of student learning. They can, however, be informative
indirect measures of learning.]
5.
False Every learning outcome of every course and program must be assessed. [Only the key learning
outcomes of courses and programs need be assessed on a regular basis.]
6.
False All students should be assessed to demonstrate the effectiveness of a course or program; a sample of
students is inadequate. [Samples can be cost-effective sources of information, provided that the samples are
representative of all students, and the samples are sufficiently large so that the results can be generalized.]
7.
False Goals should not be changed after they are selected. [Goals should be modified whenever it
becomes clear that revising them would improve the student learning experience.]
8.
False The same assessment measures should be used in every assessment cycle. [Assessment
strategies can be implemented on a staggered basis and can be modified whenever it is clear that a new or
revised strategy would be more useful.]
9.
True
Grades alone are not direct evidence of student learning. [Grades alone do not tell us exactly what
a student has and has not learned. The information upon which grades are based—tests, student papers and
projects, and the like—are direct evidence of student learning.]
10. False The primary purposes of assessment are to maintain accreditation and satisfy external stakeholders;
therefore it is appropriate to schedule assessment cycles so that they coincide with accreditation self-studies.
[The primary purpose of assessment is to improve student learning. Assessment should be systematic,
continuous, and ongoing.]
11. False The most effective way to create an assessment plan quickly is to adopt the assessment plan
of another institution. [Although an institution may choose to adapt some of the features of another institution’s
assessment plan, each institution should develop an assessment plan that is tailored to its own culture,
mission, and needs.]
12. False Assessment efforts should focus on what is learned in academic courses and programs; assessing what
is learned in out-of-class activities is not important. [Both in-class and out-of-class activities include valuable
learning opportunities that should be assessed.]
13. True
Self-report measures can yield useful information about student learning. [Asking students to reflect
upon their learning experiences can yield valuable insights into what they have and have not learned,
especially
their attitudes and values.]
14. False The assessment of educational effectiveness and the assessment of institutional effectiveness are
not related. [Because teaching and learning are fundamental missions of every institution of higher
education, the assessment of educational effectiveness is a major component of the assessment of institutional
effectiveness.]
87
Appendix 5
Department/Program Student Outcomes Survey
1.
Does your department require a Capstone/Senior Culminating Experience? q Yes; q No
If yes, what form does this experience take?
q A choice among several options (Check all options available to students):
q Senior Thesis
q Honors Thesis
q Research Seminar
q Topical Seminar
q Service Learning Course
q Internship
q Independent Study
q Other
q A single course or requirement that must be completed by all students
(Check which form this requirement takes):
q Senior Thesis
q Honors Thesis
q Research Seminar
2.
q Topical Seminar
q Service Learning Course
q Internship
q Independent Study
q Student Teaching
q Other
Is this a Capstone/Culminating Experience required by
disciplinary accreditation or for professional certification?
q Yes; q No
3.
Please give examples of how your department adapts its curricula as a result of student performance
in capstone experiences.
4.
Does your department administer its own surveys or questionnaires
to current students?
q Yes; q No
If yes, when are these questionnaires or surveys administered? (Check all that apply.)
q First Year
q Sophomore Year
q Junior Year
q Senior Year
Which of the following kinds of information is gathered on these surveys? (Check all that apply.)
q
q
q
q
q
5.
Graduate school applications
Graduate school acceptances
Graduate school chosen
Career plans
Jobs applied for
q
q
q
q
q
Jobs offered
Jobs accepted
Expected salary
Salary for accepted job
Community service
activities
q
q
q
q
q
q
Leadership activities
Satisfaction with the institution
Satisfaction with department
Satisfaction with major
Satisfaction with teaching
Perceptions of how their
education could have been
improved
q Other
Please give examples of how your department adapts its curricula as a result of student surveys
or questionnaires.
88
6.
Does your department administer alumni surveys or questionnaires to graduates? q Yes; q No
If yes, when are these questionnaires or surveys administered? (Check all that apply.)
q At graduation
q One year after graduation
q Two years after graduation
q Repeatedly on a regular cycle
q Other
Which of the following kinds of information is gathered on alumni surveys or questionnaires?
q
q
q
q
q
q
q
q
q
Alumni contact information
Graduate school applications
Graduate school acceptances
Graduate school chosen
Career plans
Jobs applied for
Jobs offered
Expected salary
Current salary
q
q
q
q
q
q
q
Community service activities
Professional leadership activities
Retrospective satisfaction with the institution
Retrospective satisfaction with department
Retrospective satisfaction with major
Retrospective satisfaction with teaching
Retrospective perceptions of how alumni’s
education could have been improved
q Other
7.
Please give examples of how your department adapts its curricula as a result of alumni surveys
or questionnaires.
8.
Outside the context of a senior or alumni survey, does your department keep
an ongoing database or written record of any of the following student data?
q Yes; q No
If yes, check all that apply.
q Graduate school applications
q Graduate school acceptances
q Career plans
9.
q Jobs applied for
q Community service activities
q Leadership activities
q Salary for accepted job
q Other
If your department requires its students to take a comprehensive or exist examination (either created
in-house or obtained from an external source), please list the name of this examination and its source.
10. Please give examples of how your department adapts its curricula as a result of comprehensive or
exit exams.
11. Students may take professional licensure examinations, professional certification examinations,
or graduate entrance examinations (e.g., GRE, MCAT, LSAT, GMAT). Please list the exams of this type
that your students are likely to take.
12. a.
b.
Does your department keep data on the number of students
taking these professional licensure examinations, professional
certification examinations, or GREs?
q Yes; q No
Does your department have access to student scores?
q Yes; q No
89
13. Can you give an examples of how information about scores on professional licensure examinations,
professional certification examinations, or GREs are used to adapt your curriculum?
Please add an addendum if necessary.
q Yes; q No
14. Does your department collect student portfolios?
If yes, are they collected from:
q All students q A random sample
If your department uses portfolios, briefly describe the types of material included in them.
Please add an addendum if necessary.
15. Can you give examples of how your department adapts its curricula as a result of information from
student portfolios?
16. Does your department keep an ongoing record of student accomplishments
(e.g., student presentations at conferences, student gallery showings of artwork,
student publications, student productions, etc.)?
q Yes; q No
17. Does your department keep an ongoing record of student/faculty
collaborative research and scholarship (e.g., presentations, publications, etc.)?
q Yes; q No
18. Does your department require students to present their work to an audience of
their peers and/or faculty?
q Yes; q No
If yes, list the course(s) for which such a presentation is required.
19. Does your department keep a record of competitive scholarships, fellowships,
internships, or grants awarded to or won by your students?
q Yes; q No
20. Does your department assess achievement of the general education goals
within your program?
q Yes; q No
If yes, list the goals you assess, and briefly describe how they are assessed.
Please add an addendum if necessary.
21. Can you identify other information-gathering techniques, not listed in this survey, that you use to assess
student learning at the department or program level? Please add an addendum if necessary.
22. Which techniques for assessing student learning at the program or departmental level do your faculty
find most useful and informative? Please add an addendum if necessary.
Source: The College of New Jersey, Ewing, NJ. Reproduced and adapted with permission.
90
Appendix 6
Learning Goals and Assessment Techniques
This worksheet can help faculty and staff begin to define important learning goals and to determine
appropriate assessment techniques.
A
Important
Goals:
What
students can
do after
completing
the program
(course,
activity)
B
How do
students
learn to do
this?
C
D
What
information
or evidence is
there that
students are
learning
this?
1.
2.
3.
91
How has this
information
been used to
help students
learn?
E
What
additional
evidence is
needed to
understand
how well
students are
learning
this?
F
What
possible new
or improved
assessment
tools or
techniques
might be
used?
Appendix 7
From Effect to Cause: A Brainstorming Exercise
The scenarios presented here can be used as the focus of a brainstorming exercise to help faculty and staff
members get a sense of the kinds of changes in curriculum and practice that may result from assessment.
Scenario 1. Faculty in the Biophysics Department
agree that student majors should be able to make
effective oral presentations of their research
findings, but they are not satisfied with the quality
of the oral presentations made by their seniors.
Unfortunately, they can’t find a place in the
curriculum for students to practice preparing and
making oral presentations. All the faculty agree that
they have so much content to cover in their courses
that they don’t have time to teach students how to
make effective oral presentations and then listen to
them. How might the faculty address this?
Scenario 4. The faculty members of the European
Studies Department agree that their student majors
should be able to summarize the principles or
teachings of the major ancient Greek philosophers.
Unfortunately, a review of a sample of student
papers shows that the students are generally poor at
doing this. To make matters worse, there is only
one course in the department that covers ancient
Greek philosophy, taught by a senior faculty
member who adamantly refuses to consider
modifying what or how he teaches. What might the
department do?
Scenario 2. Senior Biology majors at Roselyn
College scored poorly on the botany section of the
XYZ National Biology Test. Some faculty believe
that this is not a concern, because virtually all
Roselyn Biology graduates go on to careers in the
health and medical fields. Others believe that a
grounding in botany is essential to being a
well-rounded biologist. How might the faculty
resolve this?
Scenario 5. One of the goals of the Organizational
Leadership program is that students are able to
“write clearly and effectively.” Although
Organizational Leadership majors are asked to
write term papers in at least six department courses,
their writing quality is nonetheless inadequate by
the time they become seniors. Faculty are quick to
point to the woefully poor writing skills of entering
freshmen and equally quick to blame the English
Department for not bringing students’ writing skills
up to par in freshman composition classes. What, if
anything, might be done to improve students’
writing skills before they graduate?
Scenario 3. In blind reviews, 85% of Cultural
Anthropology senior theses were scored
“outstanding” in terms of clarity, organization, the
comprehensiveness of their review of scholarly
literature, and the soundness of their analysis and
conclusions. Five percent were scored “very good,”
5% “adequate,” and 5% “inadequate.” How might
the faculty use this information?
© 2002, Copyright by Linda Suskie.
Reproduced and modified with permission.
92
Appendix 8
Student Learning Styles: Frequently Asked Questions
What is “learning style”?
There is no one universally-accepted definition of
the term learning style, but the most frequently
cited definition appears to be cognitive, affective,
and physiological factors that affect how learners
perceive, interact with, and respond to the learning
environment (Keefe, 1979).
What is the predominant conception
of learning style?
sensing-intuition scale, and the social interaction
models are similar to the Jungian introversionextroversion scale.
Some instruments (Index of Learning Styles,
Productivity Environmental Preference Survey, and
some of the instruments listed above) draw on
multiple models.
Is the concept of learning style valid
and useful?
Vincent and Ross (2001) note that most
professional educators “agree that learning styles
exist and acknowledge the significant effect that
learning styles have on the learning process.” The
concept of learning styles makes sense intuitively.
It is apparent, for example, that some people prefer
reading books rather than listening to them on tape
and vice versa, and that some people prefer
working alone rather than working with others and
vice versa (Curry, 1987). Indeed, some learning
Field dependence/field independence
preferences (e.g., a preference for a quiet
(Group Embedded Figures Test)
background) seem so self-evident that it may not be
Jungian models (Myers-Briggs Type Indicator, necessary to have a validated instrument to assess
those preferences. As Nagy (1995) notes, “Little
Gregorc Style Delineator, Keirsey
can be said about preference questions that ask, for
Temperament Sorter II, Kolb Learning
example, what time of day a student prefers to
Style Inventory)
study, except to wonder if such information
Sensory (visual-auditory-kinesthetic) models
requires the expense of a standardized test.”
(several inventories)
Learning style advocates point to a number of
Social interaction models (Grasha-Reichmann validating studies. Swanson (1995), for example,
Student Learning Style Scales and Learning
cites numerous studies identifying cultural
Preference Scales)
differences in learning styles, and the discussions of
individual instruments that follow include other
Howard Gardner’s multiple intelligences
examples of validating studies.
model (several inventories)
There are many models and instruments for
categorizing learning styles, but they have not yet
been integrated into an overall learning style theory
(Bonham, 1988a; Rayner & Riding, 1997).
As Vincent and Ross (2001) note, “Professional
educators…are unable to form a consensus
regarding the establishment of a single set of
accepted principles.” Instruments have been
developed from at least six models:
1.
2.
3.
4.
5.
6. John Biggs’ approaches to learning model
(Study Process Questionnaire)
These models are not mutually exclusive or
necessarily complementary (Vincent & Ross,
2001). The field dependence/field independence
model, for example, is similar to the Jungian
Critics, however, point out that for a learning style
theory to be valid and useful, it must be shown that
students learn more effectively when their learning
styles are accommodated, and only a limited
number of studies have shown this. Some therefore
feel that the usefulness or validity of learning style
models and instruments has not been definitively
93
established (Bonham, 1988a; Bonham, 1988b;
Kavale & Forness, 1987; Rayner & Riding, 1997).
While unvalidated instruments should not be used
to make potentially harmful decisions about
students, pedagogy, curriculum, etc., they may be
used to help students gain self-awareness, provided
that students have the opportunity to complete
several instruments, so they do not take the results
of any one instrument too seriously.
try to get to know their students, help their students
get to know them, and provide plenty of feedback.
Most good teachers, of course, already do this
instinctively.
Claxton and Murrell (1987) further recommend that
faculty members:
1. Participate in workshops and other
professional development opportunities that
help them better understand the importance of
learning style and its role in improving
students’ learning.
A particular concern (Grasha, 1990; Stellwagen,
2001) is that most learning style theories label or
pigeonhole students into a few discrete,
quantitative, often dichotomous categories, rather
than recognizing that individuals develop and
practice a qualitative mixture of learning styles that
evolve as they learn and grow and that vary by
discipline (Silver, Strong, & Perini, 1997). In some
instances, pigeonholing begets a risk of
stereotyping cultural groups.
How can faculty members use
information on learning styles
to help students learn?
2. Engage in classroom research that investigates
how information on learning styles can
improve their teaching.
3. Create curricular experiences that help
students learn how to learn by raising their
awareness of their own preferences and
strengths and developing strategies for
succeeding in courses taught by faculty
members whose styles differ from their own.
Should faculty members aim
to accommodate each student’s
learning style?
Faculty members first should understand that
students use a variety of approaches to learning that
may not match their own. Schroeder (1993) reports Faculty members should not try to accommodate
individual learning styles for several reasons.
that over 75% of faculty members prefer the
Myers-Briggs intuitive style and learn best through
1. Models of teaching and learning styles are not
abstract concepts, ideas, and theories, compared to
yet sufficiently validated to be able to
just 40% of entering students and 25% of the
determine definitively how each student learns
general population. Most students, in contrast,
best and customize instruction accordingly
prefer the sensing style and learn best through
(Zarghani, 1988).
concrete, practical, structured, and sequential
2. As Gregorc (cited in Wilson, 1998) notes,
experiences.
“attempting to teach to all students’ styles can
Anderson and Adams (1992) and Wilson (1998)
quickly cause a teacher to burn out.” Gardner
urge faculty members to use a flexible variety of
(1996) states that “there is no point in
approaches to help students learn; Montgomery and
assuming that every topic can be effectively
Groat (2002), and Vincent and Ross (2001) offer
approached in at least seven ways, and it is a
specific suggestions. In addition to offering the
waste of effort and time to attempt to do this.”
usual lectures and readings, faculty can engage
3. While students should use their “strong”
their students’ senses and give them an idea of
learning styles to best advantage, it’s just as
structure by providing visual aids such as bulleted
appropriate for them to develop their abilities
lists, charts, and diagrams; written outlines or study
to use other learning styles (Grasha, 1990;
guides of key points; structured opportunities for
Montgomery & Groat, 2002) and to work with
group interaction; practical “real world” examples;
faculty whose styles differ from their own.
and a variety of assignment formats. They can also
94
How might students learn about their
own learning styles?
Bonham, L. A. (1988a). Learning style use: In need of
perspective. Lifelong Learning: An Omnibus of
Practice and Research, 11(5), 14-17, 19.
Bonham, L. A. (1988b). Learning style instruments: Let
Because any one instrument is an incomplete,
the buyer beware. Lifelong Learning: An Omnibus
imperfect assessment of learning style, students
of Practice and Research, 11 (6), 12-16.
should not let any one instrument dictate their
Claxton, D. S., & Murrell, P. (1987). Learning styles:
learning styles (Bonham, 1988a). Instead, they
Implications for improving educational practices
should be encouraged to develop their own sense of
(Report No. 4). Washington: Association for the
their learning styles, using multiple learning style
Study of Higher Education.
inventories as clues rather than as definitive
Cornwell, J. M., & Manfredo, P. A. (1994, Summer).
determinations. Crowe (2000) and Grasha (1990)
Kolb’s learning style theory revisited. Educational
suggest that students could be asked to:
& Psychological Measurement, 54 (2), 317-328.
q Write a paragraph or two explaining how
they learn best.
Crowe, R. (2000). Know your student’s learning style:
The missing link in the lecture v. active learning
issue. Paper presented at the NISOD Conference,
q Complete at least two learning style
Austin, TX.
inventories (perhaps chosen from among the
Keirsey Temperament Sorter, the VARK
Curry, L. (1987). Integrating concepts of cognitive or
Questionnaire, Owens and Barnes’ Learning
learning style: A review with attention to
Preference Scales, the Multiple Intelligences
psychometric standards. Ottawa: Canadian College
Developmental Assessment Scales, and the
of Health Service Executives.
Study Process Questionnaire) and compare Dunn, R., & Griggs, S. (1995). A meta-analytic
the results with their self-description.
validation of the Dunn and Dunn model of
learning-style preferences. Journal of Educational
q After completing a course project, write a
Research, 88 (6), 353-362.
reflection on what and how they learned
from the project.
Gardner, H. (1996, Apr.). Multiple intelligences: Myths
and messages. International Schools Journal, 15 (2),
q Use all of the above to develop a refined
8-22.
statement of how they learn best, along with
a list of study/learning strategies they could Grasha, T. (1990). The naturalistic approach to learning
styles. College Teaching, 38 (3), 106-114.
use to take best advantage of their own
particular learning style and to help them
Hayes, J. & Allinson, C. W. (1997, May-June). Learning
styles and training and development in work.
learn in situations where they must use
Educational Psychology, 17 (1/2), 185-194.
approaches that do not correspond with
their style.
Kavale, K. & Forness, S. (1987). Style over substance:
Assessing the effectiveness of modality testing and
teaching. Exceptional Children, 54, 228-239.
References
Anderson, J. A. (2001). Developing a learning/teaching
style assessment model for diverse populations. In
Linda Suskie (Ed.), Assessment to promote deep
learning: Insight from AAHE’s 2000 and 1999
Assessment Conferences. Washington: American
Association for Higher Education.
Anderson, J. A., & Adams, M. (1992). Acknowledging
the learning styles of diverse student populations:
Implications for instructional design. In L. L. B.
Border & N. V. Chism (Eds.), Teaching for
diversity. San Francisco: Jossey-Bass.
Kavale, K. A., Hirshoren, A., & Forness, S. R. (1998,
Spring). Meta-analytic validation of the Dunn and
Dunn model of learning-style preferences:
A critique of what was Dunn. Learning Disabilities
Research and Practice, 13 (2), 75-80.
Keefe, J. W. (Ed.) (1979). Student learning styles:
Diagnosing and prescribing programs. Reston, VA:
National Association of Secondary School
Principals.
Klein, P. D. (1997, Fall). Multiplying the problems of
intelligence by eight: A critique of Gardner’s theory.
Canadian Journal of Education, 22 (4), 377-394.
95
Lewthwaite, B., & Dunham, P. H. (1999). Enriching
teaching scholarship through learning styles. Paper
presented at the Annual Meeting of the American
Association of Colleges for Teacher Education,
Washington.
Silver, H., Strong, R., & Perini, M. (1997, Sept.)
Integrating learning styles and multiple
intelligences. Educational Leadership, 55 (1),
22-27.
Stellwagen, J. B. (2001, May/June). A challenge to the
learning style advocates. Clearing House, 74 (5),
265-269.
Montgomery, S. M., & Groat, L. N. (2002). Student
learning styles and their implications for teaching.
Ann Arbor: Center for Research on Learning and
Teaching, University of Michigan.
Swanson, L. J. (1995, July). Learning styles: A review of
the literature. The Claremont Graduate School.
ERIC Document Reproduction Service No. ED
387067.
Nagy, P. (1995). Review of the Learning Style Profile.
In J. C. Conoley & J. C. Impara (Eds.), The Twelfth
Mental Measurements Yearbook. Lincoln, NE:
Buros Institute of Mental Measurements.
Thompson, P., et al. (1979, April). Interrelationships
among five cognitive style tests, student
characteristics, and achievement. Paper presented at
the Annual Meeting of the American Educational
Research Association, San Francisco. ERIC
Document Reproduction Service No. ED 174678.
O’Neil, J. (1990). Making sense of style. Educational
Leadership, 48 (2), 4-9.
Presland, J. (1994). Learning styles and continuous
professional development. Educational Psychology
in Practice, 10, 179-184.
Traub, J. (1998, October 26). Multiple intelligence
disorder. The New Republic.
Rayner, S. & Riding, R. (1997). Towards a
categorization of cognitive styles and learning
styles. Educational Psychology, 17 (1/2), 5-29.
Vincent, A., & Ross, D. (2001, Summer). Learning style
awareness: A basis for developing teaching and
learning strategies. Journal of Research on
Technology in Education, 33 (5).
Ruble, T. L., & Stout, D. E. A critical assessment of
Kolb’s learning style inventory. ERIC Document
Reproduction Service No. ED 377221.
Wilson, V. A. (1998, Jan. 1). Learning how they learn:
A review of the literature on learning styles.
Muskingum College. ERIC Document Reproduction
Service No. ED 427017.
Rule, D. L., & Grippin, P. C. (1988). A critical
comparison of learning style instruments frequently
used with adult learners. Paper presented at the
Annual Conference of the Eastern Educational
Research Association, Miami Beach.
Zarghani, G. H. Z. (1988). Identification of learning
style strategies which enable college students with
differing personality temperament to cope with
learning blocks. Unpublished doctoral dissertation,
University of Nebraska, Lincoln.
Schroeder, C. C. (1993, Sept./Oct.) New students – New
learning styles. Change.
Seidel, L. E., & England, E. M. (1997). Gregorc’s
cognitive styles: Preference for instructional and
assessment techniques in college students. Poster
presented at the Annual Convention of the American
Psychological Society, Washington.
© 2002, Copyright by Linda Suskie. Reproduced with
permission.
Sewall, T. J. (1986). The measurement of learning style:
A critique of four assessment tools. University of
Wisconsin at Green Bay, Assessment Center. ERIC
Document Reproduction Service No. ED 267247.
Shearer, B. (n.d.). Assessment of the multiple
intelligences: Theoretical issues. Multiple
Intelligences Special Interest Group of the American
Educational Research Association. Available online:
http://www.geocities.com/Athens/Column/7568/asse
ssment.htm.
96
References
Anastasi, A. & Urbina, S. (1996). Psychological Testing, Seventh Edition. Upper Saddle River, NJ :Prentice-Hall.
Angelo, T. A. & Cross, P. K. (1993). Classroom assessment techniques: A handbook for college teachers (2nd ed.).
San Francisco: Jossey-Bass.
Angelo, T. A., Ewell, P.T., & Lopez, C. (2001). Assessment at the millennium: Now what? In L. Suskie (Ed.),
Assessment to Promote Deep Learning (pp. 59-63). Washington, DC: American Association for Higher
Education.
Association of College & Research Libraries. (2000). Information literacy competency standards for higher
education. Chicago: Author.
Astin, A. (1991). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher
education. Portland: Oryx Press and the American Council on Education.
Banta, T. W., Lund, J. P., Black, K. E., & Oblander, F. W. (1996). Assessment in practice: Putting principles to
work on college campuses. San Francisco: Jossey-Bass.
Briggs-Myers, I. & McCaulley, M. (1992). Manual and guide to the use of the Myers-Briggs Type Indicator.
Consulting Psychologists Press.
Chickering, A. & Gamson, Z. F. (1991). Applying the seven principles for good practice in higher education.
New Directions for Teaching and Learning, No. 47. San Francisco: Jossey-Bass.
Gall, J. P., Borg, W. R. & Gall, M. D. (1998). Applying educational research: A practical guide (4th ed.). Boston:
Allyn & Bacon.
Gardner, H. (1983). Frames of mind: The theory of multiple intelligences. New York: Basic Books.
Haladyna, T. M. (1999). Developing and validation multiple choice test items. Mahwah, NJ: Erlbaum.
Hartman, V. F. (1995). Teaching and learning style preferences: Transitions through technology. VCCA Journal, 9,
2, 18-20.
Huba, M. E. & Freed, J. E. (2000). Learner-centered assessment on college campuses. Needham Heights: Allyn &
Bacon.
Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Hillsdale, NJ:
Prentice Hall.
Krueger, R. A. & Casey, M. A. (2000). Focus groups: A practical guide for applied research. London: Sage.
Maxwell, J. A. A. (1996). Qualitative research design: An interactive approach. London: Sage.
Middle States Commission on Higher Education. (2006). Characteristics of excellence in higher education:
Eligibility requirements and standards for accreditation. Philadelphia: Author.
_______________________________________. (2003). Developing research and communication skills:
Guidelines for information literacy in the curriculum. Philadelphia: Author.
Morgan, D. L. (1997). Focus Groups As Qualitative Research, Second Edition. London: Sage.
Palomba, C. A. & Banta, T. W. (1999). Assessment essentials: Planning, implementing, improving. San Francisco:
Jossey-Bass.
Pascarella, E. T. & Terenzini, P. T. (1991). How college affects students: Findings and insights from twenty years of
research. San Francisco: Jossey-Bass.
Schein, E. (1996). Organizational culture and leadership. San Francisco: Jossey-Bass.
Silverman, D. J. (2001). Interpreting qualitative data: Methods for analyzing talk, text and interaction. (2nd ed.).
London: Sage
97
Sternberg, R. J. (1988). The triarchic mind: A new theory of human intelligence. New York: Viking Press.
Stufflebeam, D. L. (2001). Evaluation models. New Directions for Evaluation, No. 89. San Francisco: Jossey-Bass.
Suskie, L. (1996). Questionnaire survey research: What works? (2nd ed.). Tallahassee, Florida: Association for
Institutional Research.
Walvoord, B. E. & Anderson, V. J. (1998). A tool for learning and assessment. San Francisco: Jossey-Bass.
98