Handbook Page | 1
University of Texas at El Paso Program Learning Outcomes Assessment Handbook Provost’s Office 915­747­5725 provost@utep.edu http://www.utep.edu/provost Acknowledgements:
Parts of this handbook have been adapted or adopted from the sources listed in the Bibliography at the end of this
document, specific sections are referenced.
Handbook Page | 2
Table of Contents
Introduction
What is assessment
Misconceptions about assessment
Student learning outcomes
Rationale for Assessment
The role of the provost office
The Role of the College, Department Assessment Coordinator,
and the Faculty in Program Learning Outcome Assessment
A Recommendation from UTEP-College of Business Administration.
Starting the Work
Step 1. Revisit the Mission and General Goals for your Program.
Program Mission Statements of Various Qualities
Examples of Mission Statements at UTEP
Worksheet for Step 1
Step 2. Identify the 4-6 most important student learning outcomes
Examples of Program Learning Outcomes at UTEP
Worksheet for Step 2
Step 3. Identify where in the curriculum your learning outcomes and
objectives are covered: Curriculum and syllabus analyses
Worksheet for Step 3
Syllabi Analysis
Step 4. Identify assessment methods and techniques.
Step 5. Collect, tabulate, analyze the data, and report the results.
Step 6. Using the Evaluation Results: Closing the Loop!
Evaluation, Reflection, and Taking Action
Monitoring the Impact of the Changes and comparing them to Past Data
Communicate Conclusions and the Actions of Improvement to be implemented
Step 7. Determine who will implement the Improvement Actions
Appendix A
Appendix B
Bibliography
3
4
4
6
6
6
7
8
8
9
9
9
12
13
14
17
20
21
22
23
31
33
34
34
34
35
36
37
38
Handbook Page | 3
Introduction
Academic assessment at UTEP consists of three major components. First, each academic
department and program conducts annual Program Learning Outcomes Assessments (PLOA).
This is a continuous assessment of student performance on the program learning outcomes for
each undergraduate program as required by Southern Association of College and Schools
(SACS) and included in the commitments UTEP made in its mandatory Quality Enhancement
Plan (QEP)1,2,3. The UTEP-QEP states, “We will measure student learning outcomes using a
distributed assessment procedure, but under centralized reporting and oversight.”
The second component of program assessment is the Periodic Program Unit Review (PPUR)
which is a “rolling” review that will occur in cycles of 5 yrs for all academic units. Each year a
select group of academic units will complete a comprehensive PPUR of all their undergraduate
and graduate programs and unit activities as promised in the UTEP-QEP4. This handbook does
not cover PPUR.
Finally, UTEP has also committed to a regular review of the university Core Curriculum5. The
responsibility to lead this effort lies with the UTEP Faculty Senate’s Curriculum Review
Committee6, but will also involve data reporting and review activities by the departments and
programs that offer courses in the core curriculum. This handbook does not cover core
curriculum review.
To assist the departments and programs in completing these tasks, the Provost’s Office is
developing online tools and a website that should be functional by spring 2009. The website and
its tools will help departments report data, store relevant documentation, and have these easily
accessible and continuously available for updates and improvements. The Provost’s Office will
work with Deans and Chairs to standardize reporting formats as much as possible. This
handbook will be part of the website.
SACS requirements include the following standards that address the need to assess student
learning outcomes7 (note that only relevant items are presented here):
•
•
•
•
3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these
outcomes, and provides evidence of improvement based on analysis of the results in each of the
following areas:
3.3.1.1 Educational programs, to include student learning outcomes
3.4.10 The institution places primary responsibility for the content, quality, and effectiveness of the
curriculum with its faculty.
3.4.11 For each major in a degree program, the institution assigns responsibility for program
coordination, as well as for curriculum development and review, to persons academically qualified in
the field. In those degree programs for which the institution does not identify a major, this
requirement applies to a curricular area or concentration.
1
UTEP Quality Enhancement Plan, revised edition, September 2006, paragraph 5.2.2, #3. UTEP Quality Enhancement Plan, revised edition, September 2006, paragraph 9.4, items 5‐7. 3
SACS Commission on Colleges (2008). The Principles of Accreditation: Foundations for Quality Enhancement (3rd Ed.) p. 7. Available at http://www.sacscoc.org/pdf/2008PrinciplesofAccreditation.pdf. 4
UTEP Quality Enhancement Plan, revised edition, September 2006, paragraph 5.2.2. #7. 5
UTEP Quality Enhancement Plan, revised edition, September 2006, paragraph 5.2.2. #4. 6
See UTEP Quality Enhancement Plan, revised edition, September 2006, Timeline Table in Section 8. 7
SACS Commission on Colleges (2008). The Principles of Accreditation: Foundations for Quality Enhancement (3rd Ed.) p. 27. 2
Handbook Page | 4
•
3.5.1 The institution identifies college-level general education competencies and the extent to which
graduates have attained them.
The purpose of this handbook is to help academic programs understand PLOA requirements and
develop an effective and efficient assessment plan that is simple to implement and maintain
without imposing a large burden on chairs and faculty. The objective of every PLOA assessment
plan should be to improve the academic programs. This handbook briefly introduces the various
steps in the process of developing a plan and provides recommendations, example worksheets,
and matrices that departments and programs can use to complete each of the steps. Appendix A
and B contain rubrics developed at California Polytechnic State University and shared by Dr.
Morrobel-Sosa that can be used to evaluate the quality of program learning outcomes and the
progress of the PLOA development process.
What is Assessment?
According to Palomba (1999) to assess is “to examine carefully” and assessment can be defined
as “the systematic collection, review, and use of information about educational programs
undertaken for the purpose of improving student learning and development.” The American
Association for Higher Education8 developed 9 principles of good practice for assessing student
learning. These are:
1) Assessment of student learning begins with educational values.
2) Assessment of learning is most effective when learning is understood as multidimensional,
integrated, and revealed in performance improvements over time.
3) Assessment works best when the programs it seeks to improve have clear, explicitly stated
purposes.
4) Assessment requires attention to outcomes but also an equally important attention to the
experiences that lead to those outcomes.
5) Assessment works best when it is ongoing and not episodic.
6) Assessment fosters wider improvement when representatives from across the educational
community are involved.
7) Assessment makes a difference when it begins with issues of use and illuminates questions
that people really care about.
8) Assessment is most likely to lead to improvement when it is part of a larger set of conditions
that promote changes.
9) Through assessment educators meet responsibilities to students and the public.
Misconceptions about Assessment
While assessment defined as such is a laudable effort, numerous misconceptions persist among
faculty and administrators in higher education. The University of Central Florida’s Academic
Program Assessment Handbook lists several misconceptions about program assessment many
hold. These misconceptions also exist at UTEP and are listed here.
1. The results of assessment will be used to evaluate faculty performance on merit or tenure
and promotion evaluations. This is not the case. Program assessment solely serves to provide
data about the quality of academic programs that will help faculty improve them where
necessary.
2. Our program is working well, the students are learning, and therefore we don’t need to
bother with assessment. While individual faculty may know that students are learning in
their classes, most programs have no system in place to determine how effective their entire
8
American Association for Higher Education; hhtp://www.aahe.org/ Handbook Page | 5
3.
4.
5.
6.
7.
curriculum is in terms of student learning. Even if faculty believe that the quality of their
program is good, often that opinion is based on anecdotal evidence or “gut feeling” rather
than valid and reliable assessments. Most likely there is room for improvement. Continuous
assessment to determine how to best improve the educational experience of our students must
be an integral part of departmental activities. Rather than trusting unsubstantiated claims by
programs that they do what they say they do, external stakeholders such as SACS and the
Texas Higher Education Coordinating Board (TXHECB) now require data that provide
evidence for those claims. To retain our institutional accreditation, carefully assessing all our
programs is the only option.
We will just assign a single faculty member to plan and conduct the assessment. It is a
good idea to have one or two faculty members take responsibility and lead the assessment
process for the department, but it is important that everyone is involved at all stages of the
process. Each person in the department contributes different perspectives and ideas for
improving the academic program and combining that wealth of ideas creates a much stronger
end product.
Administration will use the results to eliminate departments and programs. This is a
formative assessment process that will provide substantive feedback to help improve
programs through assessment by their own faculty, not the administration. Program
assessment is not a summative evaluation aimed at eliminating programs; at UTEP we aim to
grow our programs, not to eliminate them.
Assessment is a waste of our time and does not benefit the program or the students.
Nothing can be further from the truth. All programs established learning outcomes for our
last SACS reaffirmation of accreditation process in 2006, but many programs have no data to
show how their students have performed on these learning outcomes over the past 2 years,
and thus have no clear information about potential improvements. We can no longer simply
claim we do certain things and have no evidence to show that we actually do them.
We will just do assessment in the year SACS comes back. In the past this may have
worked, but in the present it no longer does. SACS demands of its accredited programs that
they engage in ongoing and continuous assessment, and requires updates every 5 yrs.
Program assessment sounds like a good idea, but it is time consuming and complex. The
most time consuming part is formulating relevant and clear learning outcomes that are the
essential goals of your program. These will drive the measurement process and data
collection methods. Once that process has been developed, the collection and evaluation of
the data should take little time beyond the everyday activities of faculty.
“To be effective, assessment should be a comprehensive, systematic and continuous process that
is used as a means for self-improvement, based on multiple measures and sources that are
meaningful to the program. Programs should use assessment as a management tool, coordinated
by one person but reviewed by a committee or the entire faculty. Program assessment should
involve the participation and input of all faculty, staff and students”9.
As a result of such an approach, all programs at UTEP should be able to present evidence to their
students that they are actually accomplishing what they claim to accomplish. Programs
accredited by discipline specific organizations such as engineering and nursing, have been doing
this for a long time to assure society that they graduate competent engineers and nurses. Only
through well-designed and executed assessment can we know that we are achieving our desired
9
University of Central Florida, Program Assessment Handbook, 2008 edition. Handbook Page | 6
outcomes. It is also the only way by which we can improve our programs and increase student
learning.
Student Learning Outcomes
Student learning outcomes are defined as the accumulated knowledge, skills, and attitudes
students develop during a course of study. As the experts in their discipline, the faculty in the
departments and programs at UTEP are asked to publicly state these program learning outcomes,
because they have the specialized knowledge needed to write relevant, clear, and precise
discipline-specific learning outcomes and know how to assess student achievement. Faculty from
each program must critically evaluate the results of their analyses and determine if these results
suggest that improvements in the curriculum or pedagogy are necessary. If so, the program must
determine which actions can be taken to improve student performance on the program learning
outcomes. It must then implement these changes, collect further data and determine whether the
changes made a difference in student achievement. Besides being required by SACS, such
careful assessment informs each department and program about the quality of their “product”
and is of great intrinsic value. We should be able to provide clear and compelling evidence of the
educational value we provide our UTEP students.
Rationale for Assessment
Part of the rationale for continuous assessment of program learning outcomes is grounded in the
belief that systematically collecting relevant data improves awareness of how well students
obtain knowledge, skills, and attitudes as a result of the learning experiences in a relevant
curriculum aligned with dynamic professions and changing disciplines.
Continuous assessment of the academic program will enable us to
1.
2.
3.
4.
5.
6.
7.
Devote time to activities that we value most;
Help us decide how to improve instruction, strengthen curricula, and create effective
academic policies;
Strengthen our ability to show that our graduates are well-prepared to succeed in their
chosen professions;
Always have recent data on hand that will satisfy the requirements of accrediting and funding
agencies, without having to engage in a mad scramble to meet deadlines;
Develop policies to effectively allocate funding and/or resources to areas that are producing
valued outcomes;
Increase the effectiveness and truthfulness of our communications about the value of a UTEP
education to the community and other stakeholders;
Provide the faculty with feedback it needs to strengthen and grow its program.
The Role of the Provost’s Office
The Provost’s Office has the primary responsibility for coordinating academic assessment at
UTEP. The Provost collects all assessment plans developed by all academic programs at UTEP
and facilitates the assessment process by providing various electronic tools. These tools will
allow Chairs, Deans, and the Provost’s Office to respond rapidly to inquiries and communicate
the results of assessments and reviews on a regular basis to the President, SACS, the TXHECB,
and the public at large.
This assessment handbook is an effort by the Provost’s Office to provide a step by step approach
to developing an assessment plan and making annual program learning outcome assessment an
Handbook Page | 7
integral part of departmental activities without substantially increasing the burden on faculty and
administrators. The various handbooks and guides listed in the Bibliography have additional
worksheets and information that may be helpful in clarifying the process.
This 7-step approach to developing and implementing a useful and feasible plan can serve as a
review of already completed work and your progress in that process or as a tool to help you get
started.
1. Revisit the mission and general goals for your program.
2. Identify the most important student learning outcomes and objectives of the program.
3. Identify where in the curriculum your learning outcomes and objectives are covered and
whether the learning activities are integrated with the learning outcomes to ensure that your
students have the opportunity to achieve the outcomes you desire.
4. Select relevant, valid, reliable, and objective assessment methods, techniques or tools you
will use for each of the learning outcomes. You should carefully check that learning
outcomes, assessments, and learning experiences are tightly integrated.
5. Develop a process to tabulate, analyze, and report the results for review and evaluation.
6. Develop a process for evaluation of the results. After evaluation of the analysis results, the
faculty should decide on appropriate corrective actions to improve students’ learning and
determine how to best document the impact of the changes.
7. Determine who will implement these actions. What are the possible implications and
consequences for individual faculty members? How and to whom is the responsibility for
implementation delegated?
The Role of the College, Department Assessment Coordinator, and the
Faculty in Program Learning Outcome Assessment
The greatest effort and responsibility resides in the academic departments and programs. The
departments and programs are strongly encouraged to communicate and coordinate with their
Dean for continual input on all aspects of the PLOA plan development, receive feedback, and
ensure that timelines are met. To ensure successful implementation, faculty should identify one
of their members as their Assessment Coordinator. This should not be the Department Chair or
the Program Director. It is important that someone in addition to the Chair or Director can
represent the department on the issues related to PLOA. The Assessment Coordinator ensures
that assessments are conducted at the appropriate time and in the appropriate manner, that data is
compiled, analyzed, and presented for evaluation by the faculty. The Assessment Coordinator
also monitors the implementation of improvements and ensures that their effects are measured.
The Assessment Coordinator and Department Chair ensure that the data are entered in the
appropriate reporting tool and the necessary reports are submitted when necessary.
Upon completion of the assessment plan, the Dean reviews it, requests necessary revisions, and
submits the final version of the plans of all departments and programs in the College to the
Provost’s office. The Dean is responsible for the quality of all plans in his or her College. The
Provost’s Office will not review or critique the assessment plans. The Deans also are responsible
for the implementation of all plans in their Colleges.
Appendix A provides a rubric that can be used in the development and evaluation of the program
learning outcomes. Appendix B provides a rubric departments and Deans can use to determine
the progress and the quality of their assessment plans. Another example of such a rubric can be
found at the TX A&M website at
http://assessment.tamu.edu/asmt_help/asmt_feedback_rubric.pdf.
Handbook Page | 8
Departments, programs and the Deans are encouraged to use these tools to determine where
they are in the process at this moment, because much was accomplished in 2005 and 2006 that
should be retrieved and carefully examined before starting anew.
A Recommendation from UTEP-College of Business Administration.
All programs in the College of Business Administration (COBA) at UTEP are accredited by
AACSB, a discipline specific accreditation agency. In light of accreditation, COBA recently
revised its program assessment procedures and created a “Centralized Assessment Process” that
integrated 1) curriculum coverage of educational material related to learning objectives; 2)
measureable activities in the curriculum and coursework related to learning objectives; and 3)
collection and reporting of the effectiveness of the curriculum in achieving learning objectives.
In the Centralized Assessment Process all graduating seniors participate in a business simulation
and related exam in their capstone class. Their individual performance on specific exam
questions provides excellent feedback about their performance on associated learning
objectives. One instructor is charged with supervising and collecting the assessment activities in
the multiple sections of the capstone course. The Associate Dean consolidates the materials in a
report to the College Curriculum Committee for consideration and evaluation.
The capstone course or project creates a separate, independent and objective educational activity
and measurement process, especially if faculty not involved in the activity or outside reviewers
conduct the assessments. If such objectivity is not present, it’s difficult to identify problems
associated with program learning outcomes. Second, when assessments of learning outcomes are
distributed among courses, “the noisier the process and feedback,” which is then more difficult to
interpret. As the system becomes more complex it is also more difficult to maintain. This
suggests that a program should try to capture learning outcomes in a single capstone experience
that is graded by independent faculty and/or other skilled professionals on criteria and standards
related to all learning outcomes. This is a recommendation that deserves serious consideration.
Somewhat akin to a capstone experience, internships or service learning activities at a work site
may offer even more authentic assessments short of actual employment or professional
experience, and should be part of the assessment plan.
The 7-step process presented in this handbook can easily be applied to the development of a
capstone course or learning experience that a department or program might want to use as their
point of assessment.
Starting the Work
As you embark on this collective journey, it may be helpful to keep these basic questions in
mind:
1. What are we really trying to accomplish with our program?
2. How well are we actually accomplishing those goals?
3. Using the answers to the first two questions, how can we improve what we are doing as a
program?
4. What and how does our program contribute to the development and growth of our students?
5. How can we improve student learning in our program?
Handbook Page | 9
Step 1. Revisit the Mission and General Goals for your Program.
A mission is a broad statement of what your program does, and for whom it does it. It should
provide a clear description of the purpose of the program and reflect how the program
contributes to the education and careers of students graduating from the program. The mission of
your department or program should be aligned with the College and University missions, but be
specific to your program’s unique identity.
It is important that everyone in your department or program, including your students, is very
clear about what you are collectively trying to achieve. The mission should state what you do
and for whom you do it. It should stir excitement and garner commitment from everyone, and
guide you in the development of your program learning outcomes. Revision of your mission may
be necessary because the discipline has changed over time, you have new faculty, new resources,
a revised strategic plan, etc.
Program Mission Statements of Various Qualities 10
Poor: The mission of the Paso del Norte Physical Activity Education Program is to provide a
broad education on the benefits of physical activity.
The statement is very vague and does not distinguish this particular program from other physical
activity programs. It lacks information about the primary functions of the program and does not
identify the stakeholders. Additionally, there is no indication that the program’s mission is
aligned with UTEP’s mission.
Better: The mission of Paso del Norte Physical Activity Education Program is to educate
students from diverse backgrounds in the principles of physical activity education that will
prepare them for both current and future professional challenges in physical activity education.
This statement is better because it identifies the stakeholders as well as a primary function of the
program. However, it still is not a distinctive statement that sets the program apart from others.
Best: The mission of Paso del Norte Physical Activity Education Program bachelor’s degree
program is to educate students from diverse backgrounds in the fundamental skills, knowledge,
and practice of Physical Activity Education through carefully designed courses and internships
in order to prepare them for (1) Physical Activity Education positions in service organizations,
schools, and private industries and (2) graduate programs in Physical Activity Education or
related disciplines. The program promotes scholarship, service and a spirit of innovation and
entrepreneurship in an environment that is inclusive, diverse, and collaborative.
This statement includes a purpose, the primary functions of the program, the primary
stakeholders, it is distinct, and supports the UTEP mission. It is not brief and would be difficult
to memorize, but a slogan (or brand) that captures the essence of this mission may cover that.
Examples of Mission Statements at UTEP
The following missions are examples from various academic departments and programs at UTEP
and other universities. Some are very detailed and others are not. Since the true value of a
mission statement is its unique application by the department or program, these statements are
10
Adapted from University of Central Florida Academic Program Assessment Handbook, June 2008 Edition Handbook Page | 10
only samples to be examined and generate ideas. They are not an endorsement. The UTEP
mission is listed to help faculty review their departmental and program missions, as they should
be aligned with the university’s mission.
The University of Texas at El Paso Mission
The University of Texas at El Paso is dedicated to the advancement of the El Paso region
through education, creative and artistic production, and the generation, interpretation, application
and dissemination of knowledge. UTEP embraces its role as an intellectual, cultural and
socioeconomic asset to the region, offering programs to meet human resource needs and
contribute to the quality of life. As a public university, UTEP is committed to providing access
and opportunity to the people of the El Paso region and the State of Texas. UTEP’s mission of
ensuring access is coupled with a commitment to excellence reflected in rigorous programs,
which prepare students to make significant contributions to their professions, their communities
and the world. As a research/doctoral institution, UTEP fosters a climate of scholarly inquiry,
with a special focus on applying innovative interdisciplinary approaches to explore and address
major issues that confront the multicultural, U.S.-Mexico border region.
Department of Electrical and Computer Engineering
The Department of Electrical and Computer Engineering will:
• Dedicate itself to provide its students with the skills, knowledge and attitudes that will allow
its graduates to succeed as engineers and leaders.
• Maintain a vital, state-of-the art research enterprise, which provides its students and faculty
with opportunities to create, interpret, apply, and disseminate knowledge.
• Prepare its graduates for life-long learning to meet intellectual, ethical, and career challenges.
• Recognize and act upon the special mandate to make high quality engineering education
available to the residents of El Paso and the surrounding region.
Department of Geological Science
The Department of Geological Science seeks to instill core knowledge in the geosciences in nonmajor undergraduate students, including those seeking a degree that leads to teaching and the
primary and secondary levels. We also seek to equip geoscience major undergraduate students
with the knowledge, attitudes, and skills necessary for them to become successful professionals
and leaders in the local to national arena. We also seek to develop graduate students into
professionals that can be employed in industry, academia, and government and have outstanding
careers. Central to this mission is an active research program that involves students at all levels
and serves as a focal point for geoscience, environmental, and materials expertise for our region,
the state of Texas, and beyond to national and international venues.
Department of Information and Decision Sciences
The Department of Information and Decision Sciences Mission is to provide the highest quality
of education to the citizens of El Paso and the West Texas region and is in line with the mission
of the College of Business Administration. In support of The University of Texas efforts, The
Department provides a wide access to quality higher education which allows our graduates to be
competitive in the job market in the fields of Production and Operations Management and
Computer Information Systems.
Department of Kinesiology
The Department creates, interprets, applies, and disseminates knowledge in order to facilitate
students in becoming responsible professionals who can meet life's intellectual and ethical
Handbook Page | 11
challenges, value diversity, be life-long learners, and establish successful careers as teachers,
scholars, leaders, and promoters of physically active, healthy lifestyles in a multicultural society.
Department of Psychology
The mission of the Department of Psychology is to engage in the discovery of knowledge about
behavior and mental processes, and the biological, social, and psychological processes that
underlie them, and to share that knowledge and the processes for obtaining it with our students
and the scholarly community. The Department integrates graduate and undergraduate education,
teaching, research, and service activities to further the overall mission of the university.
Department of Teacher Education
We seek to prepare citizens with the skills needed for lifelong learning and personal growth and
to participate in positive societal change. Therefore, we prepare teachers who:
• Demonstrate the pedagogical and content knowledge, skills, and dispositions that help all
students learn;
• Design learning environments that are infused with educational analysis and critique in the
context of culture, community, power and knowledge;
• Understand the political and historical implications of diverse educational models; and
• Lead to improved educational programs in the communities they serve.
To inform the field of educational research about the best practices in teaching and learning
in diverse, and particularly, multilingual communities such as ours, we conduct and make
public quality research focusing on educational experiences on both sides of the U.S. Mexico border. We align with the university vision to create and maintain multicultural,
inter-American educational research collaborations.
Handbook Page | 12
Worksheet for Step 1 All departments and programs at UTEP should already have a mission. This worksheet can help you determine if your mission statement is effective and clearly in line with the current activities of your department or program. Stakeholders (consider also asking students, alumni, employers) complete this exercise individually first, after which you can identify the degree of agreement or disagreement on each item. You can then compare notes, discuss the differences, and work on a possible revision. Be aware that revising the mission can take quite some time and probably should be done in a dedicated workshop. Examples of departmental mission statements at UTEP are provided on the following pages. Write (or copy and paste) your mission statement here and then complete the checklist: Complete this checklist by placing a checkmark in the “yes” or “no” column. Criteria Yes No Does your mission clearly state the purpose of your program? Does it indicate the primary functions or activities of your program? Does it indicate for whom you do it? Is your mission statement brief and memorable? Is your mission statement distinctive, unique to your program? Does it clearly support the College and UTEP missions? List reasons for your choices here and discuss them with other stakeholders: Handbook Page | 13
Step 2. Identify the 4­6 most important student learning outcomes As a department or program you stand for something. When students decide to enter your major
they have certain expectations about your program and the promises it holds for them. By
publishing clearly written learning outcomes (a.k.a. learning goals), you present the students with
your promise, a promise about what they can expect to achieve when they complete your program
successfully. As a faculty, this is your commitment to your students. The assessment process
presents evidence to them that you honor that commitment.
During Step 2 you should again invite input from all stakeholders in your program, including
staff, students, alumni and professionals in the community who will employ your graduates to
create the most relevant learning outcomes possible.
In the past many faculty thought that SACS wanted to know every learning objective covered in
every course in the curriculum. That is no longer the case. SACS does not want to see how every
student performed on every learning objective for every course. It wants to see evidence of how
your students performed on the program learning outcomes, and what you did to improve the
situation. As a unit you need to agree on 4-6 key program learning outcomes that answer the
question: “What will students who graduate from our program be able to DO intellectually,
physically, and emotionally?” What minimal skill set should these “new players” on our team
have acquired by the time they leave our undergraduate training program? While SACS is mostly
interested in undergraduate students at this time, you should strive to develop the undergraduate
curriculum such that the transition to graduate studies is virtually seamless for the students.
Drafting student learning outcomes can be difficult. It is challenging to reach the level of
specificity required in a relevant, measurable student learning outcome. Therefore, faculty
members might want to plan a mini-retreat or set aside specific meetings that center solely on
formulating the learning outcomes so that undivided attention can be focused on this task for a
period of time. To begin drafting the program’s student learning outcomes, it may be helpful to
start with a very general phrasing of the outcome, and then get more specific with each revision. It
may take several iterations to progress from lofty, idealistic student learning goals to one or more
specific, measurable outcomes.
Your learning outcomes should distinguish the program’s graduates from other University
students. You can achieve this by clearly answering the question: “What knowledge, skills, or
attitudes distinguish the graduates of our program from other students on campus?” To arrive at
answers to this question, it is recommended that the faculty identify what constitutes the “ideal
student.” The worksheet on the next page can be used as a basis for discussions and work
sessions on the various questions about skills, knowledge, values and attitudes that you believe
this student has acquired or strengthened as a result of your program. Individual participants
should be given some time to think through these items first before continuing the discussion in
either small groups or the entire body present. Having small groups discuss the items first and
then report them on large easel pads usually leads to richer results.
Here are some examples of learning outcomes from A Program Guide for Outcomes Assessment
at Geneva College (April 2000) as presented in the UCF Handbook11:
11
University of Central Florida Academic Program Assessment Handbook, June 2008 Edition Handbook Page | 14
“Poor Learning Outcome Statement: Students should know the historically important systems of
psychology.
This is poor because it says neither what systems nor what information about each system
students should know. Are they supposed to know everything about them or just names? Should
students be able recognize the names, recite the central ideas, or criticize the assumptions?”
“Better Learning Outcome Statement: Students should understand the psychoanalytic, Gestalt,
behaviorist, humanistic, and cognitive approaches to psychology.
This is better because it says what theories students should know, but it still does not detail
exactly what they should know about each theory, or how deeply they should understand
whatever it is they should understand.”
“Best Learning Outcome Statement: Students should be able to recognize and articulate the
foundational assumptions, central ideas, and dominant criticisms of the psychoanalytic, Gestalt,
behaviorist, humanistic, and cognitive approaches to psychology.
This is the clearest and most specific statement of the three examples. It provides even beginning
students an understandable and very specific target to aim for. It provides faculty with a
reasonable standard against which they can compare actual student performance.”
A simple guide to writing learning outcomes can be found at the TX A&M University assessment
website at http://assessment.tamu.edu/asmt_help/writing_learning_outcomes.pdf
Examples of Program Learning Outcomes at UTEP
Department of Civil Engineering
1. Graduates will be educated in the fundamental concepts of engineering and science to create
intellectual curiosity in order to provide for a successful career and life-long learning.
2. Graduates will be able to design effective civil engineering systems.
3. Graduates will have the ability to function on multidisciplinary teams.
4. Graduates will serve as productive members of society and the profession by recognizing the
social, ethical, environmental, and political implications of engineering decisions.
5. Graduates will be able to communicate effectively to technical and non-technical audiences.
6. Graduates will have exposure to real-life problems including hands-on experience
Department of Computer Sciences (includes instruments and process of assessment)
The overarching educational objective of the B.S. in Computer Science program at UTEP is to
produce graduates who will be in a profession or in a graduate program that utilizes their
technical expertise, the foundation of which was obtained at UTEP. Specifically, graduates of
UTEP's undergraduate program in Computer Science will be able to:
1. Use the theoretical and technical computer science knowledge to specify requirements,
develop a design, and implement and verify a solution for computing systems of different
levels of complexity. (Objective 1)
a. Apply mathematical foundations, algorithmic principles, and computer science theory in
the modeling and design of computer-based systems.
b. Estimate the feasibility and effort required to build a particular computing system.
c. Identify and specify requirements for computing systems by selecting appropriate
modeling techniques and tools.
Handbook Page | 15
2.
3.
4.
5.
d. Design, implement, and verify computing systems of varying complexity by using
appropriate techniques and tools and by selecting appropriate design patterns,
architectures, languages, and testing approaches.
e. Evaluate a system with respect to criteria such as performance, complexity, correctness,
and usability.
f. Determine the impact of an architecture or platform on software design and
implementation alternatives.
g. Apply problem-solving techniques to solve real-world problems.
Convey technical information in both oral and written formats.
a. Present technical information orally.
b. Write a professional technical report.
c. Formulate and pose incisive, technical questions.
Work in teams.
a. Participate as a productive member of a team.
b. Solve common problems in team dynamics.
Apply a professional code of ethics in the daily practice of their profession.
a. Project the potential impacts of technical decisions on the individuals, organizations and
external constituencies involved, and identify ethical and legal implications.
b. Apply the insights embodied in professional codes of ethics.
Stay current in their profession.
a. Describe the importance of and options available for continuing education.
b. Describe the role of professional societies
c. Articulate the benefits of graduate studies
Assessment Instruments used in Computer Science
The instruments used to assess the program include: Graduating Senior Survey; Alumni Survey;
College of Engineering Employer’s Survey; Senior Exit Interviews; Departmental Advisory
Board; Teaching Evaluations; Course Assessment; Industry Feedback; Advising Survey. The long
version of this document includes description, frequency and timing of instrument, data that is
collected, means of data collection, sources of the data, and use of each instrument.
Process used in Computer Science
The Department’s process for improvement and evaluation of attainment of Program Educational
Objectives (PEOs) and Program Outcomes (POs) is a cyclical, continuous-improvement model
consisting of three phases. Fig. 1 shows the information flow into and out of each phase. The
open rectangles represent information stores, the sources of information are denoted by a square,
and the rounded boxes represent the processes associated with a phase.
Phase 1: Short-term Program and Curriculum Review
Each semester, faculty review their course assessments and teaching evaluations. There are four
main sub-committees (each representing a grouping of courses) that review whether course
outcomes are met. The courses associated with each of the subcommittees are scheduled for
formal assessment every other year, or as deemed necessary. Results from the review are
discussed at a faculty meeting.
Phase 2: Long-term Program and Curriculum Review
Approximately every two to three years, the Department reviews the scheduled recommendations
from Phase 1 and assessment results. Changes made at this level are significant and have broad
impact on the program and, thus, require deeper analysis and discussion with the Department
faculty at large.
Handbook Page | 16
Phase 3: Educational Objectives and Outcomes Review
Once every five years, the Department meets to evaluate the POs for relevance. During this
evaluation, recommendations, assessment results, and national trends are reviewed and discussed
by the faculty. If warranted, the POs and the PEOs are revised to meet the current needs of our
constituencies.
There are many resources available on websites of other universities. It is recommended that you
visit the sites of programs in your discipline to avoid duplicating work. Some professional
organization such as the American Psychological Association have published learning outcomes
on their websites for various programs (see http://www.apa.org/ed/pcue/taskforcereport2.pdf) .
Each department or program should look for and examine these resources carefully in light of the
department or program’s unique nature and the mission and vision of UTEP. Clearly, simply
copying the learning outcome statements from a program at another university would be a
mistake.
Handbook Page | 17
Worksheet for Step 2 1.
Describe “the perfect student” who just graduated from your program in terms of his or her knowledge, skills, values, and attitudes. What would you like this “golden nugget” to look like as a result of your curriculum and pedagogy? Try to be as specific and comprehensive as possible when you identify • What does this “ideal student” know and understand? • What can this “ideal student” do physically or mentally? • What does this “ideal student” value? 2.
Organize knowledge, skill, and attitude (values) characteristics in logical groupings. Handbook Page | 18
3.
Now think about which of the characteristics you identified for your “ideal student” can be directly attributed to the experiences currently presented to them in your program. If you cannot identify experiences to support certain desired characteristics write that down too (expand or shrink the table as needed). Characteristic of our “Ideal Student” Attributable Learning Experiences in Our Curriculum Characteristics not Addressed by Current Learning Experiences: Handbook Page | 19
4.
Use the work from Items 2. and 3. to either evaluate and revise, or formulate your key learning outcomes. Try to follow the guidelines in the Table12 below. Learning Outcome Characteristics Yes No
Our learning outcomes (LO) are aligned with our mission statements and goals Our LO are relevant to our discipline Our LO clearly indicate the level and type of competence that is required of graduates of our program Our LO are written clearly, precisely (just the right level of detail) and unambiguous using action verbs Our LO are measurable Our LO can be measured by more than one assessment method
We have the resources to conduct the necessary measurements of out LO
Our LO are for our program not a specific course
Our LO can be understood by our undergraduate students; they are simple, focused statements not a bundle of different things Our LO describe intended learning outcomes, not actual outcomes.
Our LO describe learning results and not the learning process
We have the resources and capabilities in our program to successfully pursue our learning outcomes Thoughts on Revisions of Learning Outcomes: 12
Adapted from University of Central Florida Academic Program Assessment Handbook, June 2008 Edition
Handbook Page | 20
Step 3. Identify where in the curriculum your learning outcomes and objectives are covered: Curriculum and syllabus analyses Once faculty members have decided upon the essential program learning outcomes, they need to
identify where in the curriculum and coursework the students receive the opportunities to learn
the knowledge, practice the skills, and develop the attitudes and values incorporated in the
learning outcomes. It is the faculty’s responsibility to ensure that the curriculum and courses are
designed to offer students sufficient opportunities to practice the skills and gain the knowledge
and attitudes necessary for success. In other words, learning outcomes, assessments, and learning
activities within the curriculum need to be tightly integrated.
To start the analysis of the curriculum, create a simple overview of curricular coverage of the
learning outcomes by completing a Curriculum Map of Learning Outcomes. The map helps you
chart which courses address and assess the program student learning outcomes you developed.
As your assessment plan changes, the Assessment Coordinator should update the Map.
Worksheet for Step 3 Curriculum Map of Learning Outcomes: Steps to Complete Your Curriculum Map List and number your learning outcomes (expand or shrink as needed): o Learning Outcome 1 (LO1): ƒ Objective 1.1: ƒ Objective 1.2: o Learning Outcome 2 (LO2): o Learning Outcome 3 (LO3): o Learning Outcome 4 (LO4): o Learning Outcome 5 (LO5): o Learning Outcome 6 (LO6): 2. Completing the map. You can do this with the entire faculty on a large flip chart or on the computer projection screen. Alternatively, you can have the faculty complete the map individually and have the Assessment Coordinator compile all the information. Make sure all learning outcomes are covered at minimum in one course or significant learning activity at the “very important” level or at the “moderately important” level in several courses (see below). 1.
Handbook Page | 21
To complete the map, faculty members place a marker under the program learning outcome that is covered in their class and for which they have an appropriate assessment tool or method. Faculty should indicate whether this learning outcome is not important (NI), somewhat important (SI), moderately important (MI), or very important (VI) in each course. Curriculum Map of Learning Outcomes (expand or shrink as needed) Program:______________________ Date:___________ LO 1 LO 2 LO 3 LO 4 LO 5 Course Important Thoughts about LO Coverage: LO 6 Handbook Page | 22
Redundancies and Gaps. After collectively reviewing the Curriculum Map and the integration of learning outcomes and their assessments, faculty should look for redundancies and gaps in the coverage of the learning outcomes. Determine whether each and every program learning outcome receives sufficient attention in various courses to ensure that the students experience enough practice opportunities to successfully attain the program learning outcomes. Not every course needs to address multiple learning outcomes; covering all learning outcomes and how this occurs is a function of the entire curriculum. 4. Collecting Data across the Curriculum and Time. It may be necessary to collect data on several measures across time or in a sequence of key courses or learning experiences to truly document that students achieved your learning outcomes. This type of measurement can be an excellent demonstration of “value added” to the student’s knowledge, skills, and attitudes as a result of your curriculum. Student portfolios are probably the best tool to show progression of competencies, and Alverno College's Diagnostic Digital Portfolio13 is a great example of an assessment system based on mastery of knowledge and skills using student portfolios. 3.
Syllabi Analysis By closely analyzing the course syllabi, determine the alignment between courses on the program learning outcomes and the assessment tools and methods used, especially for courses offered in a preplanned sequence. First, the faculty need to determine whether program learning outcomes are or should be covered in a specific course. Then they should look for a strong integration of learning outcomes, assessments, and learning activities. Check that there is a logical connection and that the assessment tools used in each class allow the faculty to measure performance on the learning outcomes and possibly track performance improvement over time. If integration is lacking the course needs to be revised. At minimum the faculty should examine the links between the stated learning outcomes (by some referred to as objectives or goals) and the assessments used in the course. Often, learning outcomes are stated but not assessed or assessments are included in the course that is not linked to any learning outcomes. Additional information about constructing a syllabus and a checklist can be found on the Center for Effective Teaching and Learning (CETaL) website at http://academics.utep.edu/Default.aspx?tabid=36893. Important Thoughts on Syllabi Alignment with Program Learning Outcomes: 13
A demonstration of this approach can be found at http://ddp.alverno.edu/. Retrieved September 15, 2008. Handbook Page | 23
Step 4. Identify assessment methods and techniques. The unit of analysis for assessing student learning outcomes is the program, not individual
courses, faculty, or students. The better the integration of the assessments into existing
student work (e.g. courses, capstone projects, service learning, etc), the greater the likelihood
that your assessment plans will succeed. The next step is to determine how you can
efficiently integrate program learning outcomes assessment into the existing curriculum. It is
very likely that you only need to make small modifications to courses or the curriculum to be
able to effectively assess student performance on program learning outcomes. Examples of
commonly used assessment tools are provided at the end of this section.
Caution! Assessment of student learning outcomes can be conducted using many different
quantitative and qualitative instruments and methods, and therein lies the danger of trying to do
too much. Select or develop instruments and methods that are simple to use, require little extra
time or effort, and still provide the necessary data for a specific learning outcome, not more and
not less.
Course Grades are (usually) not enough
It is often asked, "Aren't course grades a satisfactory measure of student performance?" In
general, grades have significant short-comings when used as a single measure of assessing
student learning.
• An exam grade is usually the result of how well individual students may have learned the
prescribed information being assessed on that particular exam. Unless the exam is designed
to be a relevant, valid, and reliable assessment of a program learning outcome, it may not
provide useful information.
• A course grade often is a composite of many activities that have little relevance to program
learning outcomes (for example, when attendance or general educational outcomes are part
of the grade) and thus cannot be used in identifying student performance on program learning
outcomes.
• Grading is often approached differently by individual faculty members even when teaching
different sections of the same class, because of individual variations in how tools are used
and standards are interpreted.
• Program learning outcomes sometimes span multiple courses, and individual course syllabi
often do not align well with the program’s learning outcomes.
• If the course addresses multiple learning outcomes and contains assessments for each of
those, the course grade is a composite and lacks the detail necessary to evaluate performance
on the individual learning outcomes. In this case, results of the individual assessments for
each learning outcome should be used.
Handbook Page | 24
Worksheet for Step 4 This exercise expands upon the previous one in Step 3 and asks you to look in greater detail at the assessment tools used to determine student performance on each program learning outcome in the various courses and experiences. To help identify what kind of assessments of your learning outcomes are currently being conducted in, complete the following table. List the course number or the learning experience and their embedded assessments such as course exams, student presentations, and non‐embedded assessments such as portfolios, institutional writing exam, and standardized tests in the matrix. This allows you to determine whether the assessments are appropriate and sufficient. Some advocate that each program learning outcome should be assessed using at least three different measures. Assessment Key (change as needed): P=Paper, E=Exam, PO=Portfolio, L=Lab; S=Standardized Test; O=Oral Presentation; I=Internship Curriculum Map of Learning Outcomes (expand or shrink as needed) Program:______________________ Date:___________ Course LO 1 LO 2 LO 3 LO 4 LO 5 LO 6
Math 1320 S E Handbook Page | 25
When deciding how to analyze your data consider basing your data collection techniques and analyses on questions directly tied to your learning outcomes. These questions will guide the selection or design of your data collection tools. Questions that may be worth asking are for example: • To what degree have students become proficient in quantitative scientific skills and written and oral presentation skills? • Have the students fallen short of, met or exceeded criteria and standards we set? • Are there subgroups of students that differ significantly from each other in what they have learned? • Have students’ performances on all learning outcomes increased over time? Is this improvement similar or different for all learning outcomes? • Did performance stall or plateau prior to senior capstone projects? Why did this happen? • Are the assessment tools (rubrics, tests) still valid and reliable or do they need to be re‐examined in light of other significant changes? • What are the strengths and weaknesses in our curriculum based on the data analysis? • How does the analysis inform the improvements we need to make in the learning experiences and the teaching techniques we employ? Once you have determined what measures, instruments, and methods of data collection can answer your questions, here are some items to consider when developing your process of assessment. 1. Create a detailed timeline for the assessment cycle and collect data each semester. Make this a habit and normal part of the department’s business. 2. Have designated faculty members be responsible for collecting the appropriate data and report them to the Assessment Coordinator 3. The Assessment Coordinator compiles the data from the various sources, enters the data in the software tool and stores it, analyzes the data, and formats the results in a clear manner. 4. Develop a process for the faculty to interpret the results of the analysis and evaluate the program. 5. Develop a process for the faculty to decide on improvement measures that need to be taken. 6. Decide who will be responsible for implementing these improvements and the kind of support these persons will receive? 7. How will you determine that the changes were successful? Is the same assessment plan the right tool to show positive change? Further Investigation of your Assessment Tools. Integration of learning outcomes, assessments, and learning activities that prepare the students for the assessments is critical to an effective course and curriculum. Now that the types of assessments and where they are used in the curriculum have been identified, it is important to complete one more step. Determine how tightly the learning outcomes, their assessments, and the learning activities are connected. This is especially important when you have multiple sections of the same course, taught by different instructors. The discussion should focus on relevance, validity, reliability, and objectivity of the assessment tool or method and the learning activities in relation to the learning outcomes. The focus of this exercise is to create the best possible integration to support student performance using constructive comments and collaboration. In the table on the next page enter the course, the learning outcome(s) covered in that course, the assessment tool(s) used to measure student performance, and whether the tool(s) has sufficient relevance, validity, reliability, and objectivity. Completion of this table helps you identify “broken links” and whether or where you lack quality assessment tools. Handbook Page | 26
Integration of Learning Outcomes and Assessment Tools (expand as needed) Course & section Program:______________________ Date:___________ Learning Outcome LO‐Related Relevance, LO‐Related covered Assessment Validity, Learning Activities
Reliability, Objectivity Important Thoughts about Integration of Key Components: Handbook Page | 27
A Few More Thoughts about Assessment
Data that You can Use. Select assessment methods that are under your control and provide
information you can do something with. An assessment method that is influenced by external
factors beyond the control of the program will give you data that will not help you make a
change in that aspect of the student learning process.
For Qualitative Assessments Create a Scoring Rubric. Rubrics clarify criteria and standards
and are an important tool for evaluating student work, especially if it is difficult to create
objective tests. Rubrics exist for many purposes and there are many types. Most likely someone
has already developed a rubric that you can use or adapt to your needs. Search the internet or
contact your network of colleagues so you do not have to “spin your wheels” creating your own.
When multiple sections of one course are taught simultaneously, faculty should make sure that
the same rubric is used in all sections, and that all faculty are trained to use them similarly. When
rubrics are used to grade a capstone project that serves as the assessment of the learning
outcomes, at least two faculty members should score the student work to ensure reliability. If
these two faculty members disagree significantly, a third member should score the work.
Rubrics are Helpful for Assessing Learning Outcomes.14 Many faculty members consider
using rubrics when looking for ways to grade, give feedback, and assess learning outcomes.
Many resources exist on how to construct a valid, reliable, and objective rubric that can be used
to assign a grade to student work. Below is a brief explanation of rubrics. Keep in mind that
rubrics can be designed such that they track a student’s performance across several courses or
learning experiences and show improvements in performance over time rather than in a single
instance. In this case the rubric should be part of a portfolio of work.
For most educators, a rubric is a printed set of scoring guidelines (criteria) and standards for
evaluating work (a performance or a product) and for giving feedback. A rubric helps address
several important issues:
1. It identifies the key elements (criteria) of the work that will be judged.
2. It indicates the differences between good and poor work (standards) on each criteria.
3. It is a tool to ensure that judgments (or scores) of work or performance are valid and reliable.
4. It helps both performers and judges be more clear about what is expected for excellence.
Why should we use rubrics? The process of designing criteria and standards must be
primarily centered on teaching and learning. In other words, faculty must first identify the key
elements (criteria) of a work or performance, and then develop the standards that discriminate
between poor and excellent accomplishments on those key elements. Subsequently, these ratings
can be converted into scores or grades. A carefully designed rubric:
1. Focuses instruction by identifying the key elements and minimum standards of a skill,
knowledge, or attitude;
2. Helps the instructor provide feedback that is focused and meaningful;
3. Characterize the desired results in a relatively objective manner by clearly describing
expectations;
4. Operationalizes performance standards such that students know in advance what is expected
of them;
5. Rubrics, when given in advance and used constantly, develop self-assessment competence in
students;
14
Adapted from the University of Texas at Dallas, Assessment Workbook, March 2006
Handbook Page | 28
6. Can be developed with the involvement of students, helping them understand the issue in
greater depth.
For more information on rubric construction you can visit http://rubistar.4teachers.org/index.php
or a multitude of other websites.
Use Course-Embedded Direct Methods of Assessments. Plan to use direct methods of
assessment as much as possible, selecting student work that they produce as part of the
curriculum. Many direct assessment methods are already built into and embedded in the courses
and the curriculum. Identify and critically examine the work products your undergraduates
already produce as part of the program’s curriculum, and determine which of these are relevant,
valid, and reliable assessments of your program learning outcomes. Students often take these
assignments more seriously and perform better than they might on a standardized test given
outside of class.
1. Written work:
a. Demonstrates knowledge of important content on an exam or in a paper.
b. Shows analysis, application, synthesis, and evaluation capabilities
c. Displays writing skills
d. Produces reflections on what, how, when, and why they learned
2. Portfolios of student work:
a. Are assessed systematically using a rubric.
i. May be evaluated to determine student learning over time, or may be comprised
of the student’s best work.
ii. Encourage student self-reflection
b. Videotape of oral presentations or performances with self, peer, and or instructor
evaluations using a rubric; include recordings of subsequent performances to document
improvements
c. Senior thesis or “capstone” course experiences
d. Field or service learning projects
e. Performance on in-class tests, assuming they are valid, reliable and objective
Please note that many of these course-embedded assessments need to be evaluated using
a rubric.
In some cases independent measures such as student performance on state or national
certification exams may show that your students achieved the program’s learning outcomes. For
example, the TExES, the state-wide teacher certification exam in Texas, provides a criterionbased, valid, reliable, and independent assessment of knowledge of prospective teachers across
the state. It can be regarded as a clear indication of how well a program prepared students for the
knowledge component of the teaching profession. In many cases, employer ratings of recent
graduates on specific items of interest may also provide valuable information.
Indirect Methods of Assessment can supplement the Evidence. Indirect methods include
student, alumni, or employer surveys, student exit interviews, curriculum and syllabus analysis,
course evaluations, or use of external reviewers. Surveys are particularly good for revealing
students’ attitudes and opinions about what they learned. Surveys are also useful to evaluate
outcomes that are realized in students’ post-college careers. Some of your program’s learning
outcomes may be better assessed through the use of indirect methods which can provide
feedback that is useful in interpreting the results of direct assessments. If several of these tools
are used, for example a graduation survey and an alumni survey, the faculty should ensure that
each uses a similar protocol or prompts so there is consistency in the questions that respondents
Handbook Page | 29
are asked to address in relation to the program learning outcomes. Indirect methods alone are not
sufficient to assess student learning outcomes; they supplement direct methods.
A Mix of Qualitative and Quantitative Methods Can Work to Your Advantage. Also
referred to as triangulation, it is recommended that you use multiple measures to obtain richer
data that enables you to put greater trust in your final conclusions. Consider using both
qualitative and quantitative assessment methods. Quantitative methods assign numerical scores
to student work, but qualitative methods often focus on the quality of work without assigning a
numerical value to them. Using well-developed rubrics allows one to give numerical scores for
qualitative assessments which make the reporting, analysis, and evaluation components much
easier. Adding indirect measures described above allow you to paint a richer picture of your
accomplishments.
Creating Your Own Tools. If you decide to design your own measurement tool keep these
things in mind:
1. Locally developed measurement tools tend to have tremendous content validity15. Work
together to develop your own tool, for example a rubric that can be used in multiple classes
to track performance improvement over time. You could use it for formative and/or
summative assessment of student performance.
2. Simple, course embedded tools are best because they contribute to lasting and more
meaningful learning.
3. Collect information that informs you about student performance on program learning
outcomes. Only collecting relevant data leads to an effective and efficient system and wellinformed decision-making.
4. To increase efficiency you can randomly sample students rather than test everyone, but make
sure that you obtain enough data to trust your outcomes and conclusions.
5. If you do collect data on random samples, obtain enough data to show stakeholders that you
are doing what you say you do with confidence. Collect data across the full range of students
and don’t throw out the data you don’t like. If you found that more than 50% of your students
could not perform a skill listed in your learning outcomes, document it; present the
improvement you instituted, and the results of these actions. Reviewers do not expect
perfection; they want to see that you did something about the problems you identified.
The UT-Dallas Assessment Workbook16 (March, 2006) presents several charts on pages 25-31
that should help clarify and provide ideas to help faculty identify effective assessment methods
and tools they could use (can be retrieved from http://sacs.utdallas.edu/hotdox/assessmentworkbook-2006-03-01.pdf.
15
Content Validity: Degree to which a test (usually in educational settings) adequately samples what was covered in a course. Thomas, J., Nelson, J., & Silverman. (2005). Research methods in physical activity. Champaign, IL: Human Kinetics. 16
Can be retrieved from http://sacs.utdallas.edu/hotdox/assessment‐workbook‐2006‐03‐01.pdf Handbook Page | 30
Some Important Considerations to Make Your Plan Work
• Clearly answer the question what skills, knowledge, or attitudes should your students possess
at the end of their course of study.
• Clearly formulate the minimum performance criteria and standards the program would like
students to meet for each learning outcome.
• Identify which courses and learning experiences (e.g., papers, exams, presentations) are best
suited for the assessment.
• Choose assessments that provide data that are easy to understand and interpret.
• Consider whether the assessments you select allow you to detect the effects of improvements.
• Protect the confidentiality of students whose learning will be assessed.
• Keep the students informed of what you’re trying to accomplish.
• Ensure that the learning experiences in class prepare them to succeed rather than fail and that
their preparatory work flows seamlessly into the assessment.
• Try to make assessments part of the regular course work and learning activities in the
curriculum rather than an additional task for students and faculty outside normal coursework.
• Keep it simple: develop a process that requires little extra time or effort.
Handbook Page | 31
Step 5. Collect, tabulate, analyze the data, and report the results. Analyzing the Data
How data is analyzed and results are reported depends on the type of data collected (qualitative
vs. quantitative, cross sectional vs. longitudinal, categorical vs. scale) and the audience. For the
majority of data analyses MS Excel will suffice; it is widely available and easy to use. Data
entered in Excel can be imported into SPSS for more sophisticated statistical analyses if
necessary. Regardless of how you analyze the data, the results should be reported in a clear, easy
to understand manner, so that it facilitates optimal evaluation. Patterns, problems, and questions
should become apparent while summarizing and evaluating the data.
Organizing Your Results
The simplest method is most likely a matrix that displays your program learning outcomes, how
they were assessed, what the minimum performance criteria were, whether improvements were
deemed necessary, and what types of improvements were planned. The matrix on the next page
is for illustration purposes only, but may provide ideas on how to organize your results, and
report the evaluation and improvement actions. Note that in electronic documents links to other
supporting documents and evidence can be inserted. This allows one to create a concise
“executive summary” that links to more detailed information. This approach works well if the
links are bookmarks to other sections in the same document or to documents stored in an
accessible location.
The Provost’s Office is at this time (September 2008) pursuing an online reporting tool called
Digital Measures that may simplify data entry and reporting of the results. Digital Measures
converts entered data into MS-Word reports that can then be edited and commented upon.
Below are two examples of how one could conceptualize and organize the results of the analyses
relative to the program learning outcomes. Please note that when Digital Measures is fully
functional the approach illustrated below may no longer be necessary or the accepted way of
reporting the results of your assessment process.
The Provost’s Office aims to provide instructions on how to use Digital Measures and examples
of data entries and reports before the conclusion of the fall 2008 semester and prior to the start of
data collection in the spring of 2009.
Handbook Page | 32
Example 1: Program Learning Outcomes Assessment and Review Matrix (based on Eder, 2004) Expectation for Satisfactory Performance Summary of Skills
Percent Met Standards
Satisfactory performance: 85% of enrolled students achieved > 70%: Specific criteria for learning outcomes in KIN courses. Pass rate on TExES > 90% Skill / competencies: PETE Courses’ Satisfactory learning outcome performance: • Teaching assessments 85% of enrolled preparation and Performance during students achieved > execution skills internships 70% on criteria: • Self‐regulated Professional portfolio Test performance learning Rubric for SRLC competencies Rubric for teamwork • Cooperative and social skills Learning, Rubric for teamwork and technological fluency
social skills • Effective use of instructional technology Values and Satisfaction Criteria:
Assessment Data: Attitudes: >70% of majors • KIN Club & member of KIN club; TAHPERD • Engaged professionally membership as % of >100% of PETE majors member of registered majors • Engaged in TAHPERD physically active, • Portfolios and >75% of students healthy lifestyle reflections received satisfactory • Contributes to • Exit survey and/or rubric scores on community interview • Positive work • Student satisfaction portfolio >50% satisfied or habits survey better on exit and • Clinical site other surveys assessments and surveys PETE Courses’ learning outcome assessments Annual TExES Certification Exam Outcome of Analysis 100
80
Teamw ork
60
SRLC
40
Tech fluency
20
0
Percent
Learning Outcom es
Summary of Skills
Percent Met Standards
Academic Knowledge and Reasoning at BS level: Application Analysis Synthesis Evaluation Where, When, and How Monitored; Assessment Tools 100
80
Teamw ork
60
SRLC
40
Tech fluency
20
0
Percent
Learning Outcom es
Summary of Values and Attitudes
Percent Met Standards
Program Goals 90
80
70
60
50
40
30
20
10
0
Healthy Lifestyle
Citizenship
Integrity
Workhabits
Percent
Learning Outcom es
Program Follow‐up
____None required
_X__ Follow‐up required (Link to page that outlines the actions to be taken with timelines) ____None required _X__ Follow‐up required (Link to page that outlines the actions to be taken with timelines) _X__ None required ____Follow‐up required Handbook Page | 33
Step 6. Using the Evaluation Results: Closing the Loop! Evaluation and a determination of how well the program learning outcomes were achieved by the
students can only occur when you are able to compare the actual data to a predetermined target.
At this stage you put a value on the results of your analysis to determine how well the program
achieved its goals. If the results suggest you missed the targets set, and students performed below
expectations on one of the program learning outcomes, you should be able to “drill down” into
the measurement process and tools used for that learning outcome, the learning experiences
students were offered, the pedagogy used, and other variables to determine where improvements
should be introduced.
Evaluation could lead to changes in many aspects of a program such as the learning outcomes,
pedagogical practices, the measurement tools and methods used to document student
performance, and the information collected. Consequently, it is important that the tools and
methods used provide the capability to accurately identify the practices that need improvement.
Evaluation, Reflection, and Taking Action
To gain a rich perspective on the data, assessment results should be disseminated widely and a
variety of stakeholders should be engaged in the evaluation to ensure the obvious is not
overlooked. Don’t forget to ask the students, they experienced the assessment plan and can give
the “inside out” point of view faculty will never be able to see. Use a thorough evaluation as a
basis for targeted improvements in the program. Once the various perspectives have been
compiled consider the following actions.
• Return to the program’s mission, goals, and outcomes. How do the results line up with
previous expectations? Did student do better, worse? Did you make incorrect assumptions or
use incorrect information during the previous planning stage? What are the outside
stakeholders telling you?
• Review the performance levels set earlier in the process. Were those expectations met? Are
the established standards adequate, or did you set the bar at the wrong height? What level of
performance is good enough for undergraduates, graduates?
• Evaluate the assessment instruments, rubrics, and methods. Are they optimally efficient and
effective in light of the questions you seek answers to? Are they the correct tools to use?
• Determine the kinds of corrective actions that will have the greatest potential to improve
student learning. In other words, try to identify the “low hanging fruit,” improvements that
do not require large amounts of resources but lead to significant increases in the quality of
student learning. Again, don’t forget to include the students’ voice in the deliberations.
• Clearly articulate what is to be done, by whom, by when and how data will be collected to
assess the impact. Make sure these actions are aimed directly at improving student learning.
• Determine the implications and consequences of the plan on department policies, curriculum,
resources allocations, faculty effort, the students’ experience of the program, etc. and
prioritize improvement actions based on high impact, low cost.
As the faculty decide on actions to improve student learning, it should set specific targets
wherever possible so it has goals to work towards. For example, if your discipline has a national
database of student performance measures (related to a certification exam for example), you can
compare your student performance with the national means. The faculty should also state what
proportion of the students should achieve a specific performance level. If previously measured
performance data on a learning outcome are available, it can be used as the baseline for setting
new targets for the next cycle.
Handbook Page | 34
Monitoring the Impact of the Changes and comparing them to Past Data
Upon implementation of changes the entire assessment cycle starts again. A carefully developed
assessment plan can likely be used to measure the impact of the changes. However, for a number
of different reasons, such as a change in learning outcomes, it may be necessary to make
modifications in the plan.
A well-designed assessment plan and careful evaluation of the results enables the faculty to make
targeted improvements or affirm current practices efficiently and effectively. This is the main
reason assessments are conducted.
Communicate Conclusions and the Actions of Improvement to be implemented
Honest communication to stakeholders, especially students, is critical in creating trust and
confidence that the program has their best interest in mind. Besides the students, the Dean will
want to know and so will the Provost, the accrediting bodies, future employers of your students,
high school seniors and other prospective students, as well as the community at large. The final
program report needs to tell them what happened, why it happened that way, what the faculty
learned from it, and what it intends to different to improve student learning.
The Department Chairs and Assessment Coordinator have the final responsibility for ensuring
that everyone on the teaching staff has had an opportunity to participate in the development of
the assessment plan, received the final copy, and understood the implications for their teaching
practice. Consider the following when communicating the results:
• Celebrate and publicize your successes. At UTEP we tend to forget to let people know what
we do well. Promote the program vigorously, but use accurate data and evidence to do so.
• Identify the shortcomings and don’t try to hide or minimize them, but present the actions you
will take to improve these weaknesses and explain what you expect of these improvements.
• Consider whether the results should be presented differently to different audiences such as
prospective students, the Dean and other administrators, the rest of the UTEP community,
and beyond.
• Avoid “data dumps,” especially to the lay people. Ask for assistance to format your outcomes
in an effective manner, especially if you are considering publishing a newspaper article and
placing the final report on your program’s website.
Handbook Page | 35
Step 7. Determine who will implement the Improvement Actions Given that these are the program learning outcomes the entire faculty bears responsibility for
helping students achieve them. However, the work to help students achieve the program learning
outcomes mostly occurs in the courses. The Curriculum Map helped you identify in which
courses students experience learning activities and tests related to the program learning outcomes.
Most likely, the program learning outcomes are covered in several courses so that responsibility
for helping students achieve acceptable performance falls on several faculty members. The
question who does what, how, where, and when to improve the situation is an important one.
Clearly, unless one course is responsible for one program learning outcome, several faculty
members will have to shoulder the task of developing learning experiences and material pertinent
to the learning outcomes.
It is advisable for several faculty members who each teach a course or lead designated learning
activities related to a specific learning outcome discuss what precisely each one does to help the
students learn. This discussion will help identify gaps and redundancies that can be eliminated to
better align the efforts in different course sections and maximize student learning. Faculty can
also ask colleagues at other institutions about their most effective strategies and best practices. In
addition, numerous disciplines now have extensive sources and sometimes specific journals
dealing with teaching and learning. These sources should be examined by a committee of faculty
especially those who teach a sequence of courses focused on a select set of program learning
outcomes. A one-semester reading group of faculty and possibly students focused on identifying
best teaching practices in the discipline may be able to identify highly effective improvements
that could have major impacts on student learning.
Handbook Page | 36
Appendix A The following rubric, borrowed from California Polytechnic State University, can be used to determine
the quality of a department or program’s assessment plan. Note that this is a draft, should not be
referenced, nor will it be used by the Provost’s Office to judge assessment plans. The evaluation of the
quality of assessment plans is performed by the Dean of each college and school.
Handbook Page | 37
Appendix B The following rubric, also borrowed from California Polytechnic State University, can be used to
determine a how well a department or program is progressing in the assessment plan development
process.
Handbook Page | 38
Bibliography
California Polytechnic State University: http://www.academics.calpoly.edu/accountability/. Documents
contributed by Dr. Anny Morrobel-Sosa.
Palomba, C. (1999). Assessment essentials: planning, implementing, and improving assessment in higher
education. San Francisco: Jossey-Bass.
Texas A&M University, Guidelines for Academic Program Review, August 2007. Retrieved August 25,
2008 http://ogs.tamu.edu/forms/documents/AcadProgReviewGuidelines_2007.pdf
University of Central Florida, Program Assessment Handbook, June 2008. Retrieved August 25, 2008
from http://www.assessment.ucf.edu/
University of Delaware, Student Learning Outcomes Assessment Manual. Retrieved August 25, 2008
from http://www.assessment.udel.edu/The%20Assessment%20Office/manual.html
University of Texas at Dallas; Assessment Workbook, March 2006. Retrieved August 25, 2008 from
http://www.utdallas.edu/oee/assessment/index.html
University of Virginia; Assessment Guide, Retrieved August 30, 2008 from
http://www.web.virginia.edu/IAAS/assessment/assessment.shtm
The following annotated list of websites is a selection of a larger list developed by the University
of Texas at Dallas, SACS Leadership Team for their UTD Assessment Workbook: A Resource for
Departments and Programs, March 2006. URLs have been checked for functionality September
16, 2008.
The SACS site and several others of universities accredited by SACS
• SACS: Principles of Accreditation — These are the principles accredited universities must
meet for reaffirmation: http://www.sacscoc.org/principles.asp
• Auburn University: http://www.auburn.edu/administration/specialreports/sacsdocuments.html
• Georgia Tech: http://www.assessment.gatech.edu/SACS/index.php
• UNC Chapel Hill: http://www.unc.edu/inst_res/SACS/sacs.html
• Texas A&M International University: http://www.tamiu.edu/sacs/
Helpful guidance and information on assessment
• http://www.bridgew.edu/AssessmentGuidebook/ This is an excellent guide on assessment
developed by the faculty at Bridgewater State University. Particularly helpful are Chapter 4:
Establishing Learning Outcomes, Chapter 5: Assessment Tools (good explanations of various
indirect and direct assessment methods with clear examples), and examples of Rubrics.
• http://www.skidmore.edu/administration/assessment/faq.htm#academicassessment In this
area of Skidmore College's website, you will find brief answers to questions about the
rationale of assessment. The jargon is explained clearly and concisely. Two questions and
answers might be of interest and helpful: The question about primary trait analysis (also
related to developing rubrics) and the question about the connection between assessment of
majors and assessment of general education.
• http://www.k-state.edu/assessment/manual/index.htm Kansas State's Office of Assessment
has one of the best and most user-friendly faculty manuals available. A brief, one-page tips
sheet is found under the "Assessment Tips" link.
• http://www2.acs.ncsu.edu/UPA/assmt/resource.htm This is a meta-site for assessment
resources.
Handbook Page | 39
Sites for Bloom’s Taxonomy for use in writing learning outcomes:
There is a new version of Bloom’s taxonomy. Departments and programs should use it instead of
the original version.
• http://www.coe.uga.edu/epltt/bloom.htm#end This is the best thorough and up-to-date
explanation of Bloom's Taxonomy on the internet. There is an animation that shows verbs
along with types of assessment for each level of Bloom's classification system. In addition,
readers can click on a PowerPoint "test" to practice using this classification system. It shows
a graphic of the old and new versions of the taxonomy.
• http://eprentice.sdsu.edu/J03OJ/miles/Bloomtaxonomy(revised)1.htm The chart at the end of
this site is very helpful. It includes activities and products that students could do based on
Bloom's levels.
• http://www.kurwongbss.qld.edu.au/thinking/Bloom/blooms.htm This site is designed for
public school teachers but it contains many links to explanations of Bloom's revised levels as
well as excellent science examples in different areas.
Help with writing learning objectives
• http://www.utsouthwestern.edu/vgn/images/portal/cit_56417/53/40/416402Creating_Learnin
g_Objectives.pdf This PowerPoint presentation from UT Southwestern provides a helpful
introduction to learning objectives.
• http://edweb.sdsu.edu/Courses/EDTEC540/objectives/ObjectivesHome.html This site
contains a helpful tutorial on writing learning objectives.
• http://www.missioncollege.org/workforce/work_experience/learning.html Although the
Mission College website is oriented toward writing objectives for workforce purposes, it
provides helpful and easy-to-follow steps that could easily be used for classroom learning.
• http://tlt.its.psu.edu/suggestions/research/Write_Objectives.shtml This website from Penn
State gives details about common problems faculty have when writing learning objectives.
• http://captain.park.edu/facultydevelopment/writing_learning_objectives.htm The resource
links and references at the end of this website are helpful. Many other resources related to
teaching and learning.