CALIFORNIA STATE UNIVERSITY SAN MARCOS
PROGRAM-LEVEL ASSESSMENT WORKSHOP
November 9, 2012
Michelle A. Saint-Germain, Professor
Graduate Center for Public Policy and Administration
California State University, Long Beach
(562) 985-8883
Michelle.Saint-Germain@csulb.edu
OVERVIEW
I. PROGRAM STUDENT LEARNING OUTCOMES
Where do they come from? How are they best stated?
II. CURRICULUM MAPPING
Which outcomes are covered in what courses?
III. EVIDENCE OF STUDENT LEARNING
Teaching and Learning Strategies
Direct and Indirect Evidence
Gathering Student Work
IV. JUDGE/ANALYZE EVIDENCE
Did students meet faculty expectations for learning?
V. PROGRAM IMPROVEMENT
How can we increase student success?
VI. SAMPLE ASSESSMENT PLAN
Manageable and Meaningful Assessment
1
I. PROGRAM-LEVEL STUDENT LEARNING OUTCOMES (PSLOs)
An academic program is a program of study for majors at either the undergraduate or
graduate level. Programs are administered by Colleges, Departments, or other units.
Complete the following process for each academic program separately.
Name of degree program: _________________________________________________
Program-level Student Learning Outcomes (PSLOs) are specific types of knowledge
and skills or competencies (and perhaps perspectives, values, or dispositions) that
students are expected to acquire in the program and to demonstrate upon completion.
Where do PSLOs come from? The program mission can point to the program level student
learning outcomes. If this academic program has a mission statement, type it into the
following box:
Mission:
Program level SLOs may also come from university-level (institutional) SLOs, disciplinary
associations, professional accrediting bodies, academic literature, consensus among peers
in the field, alumni, employers, or doctoral degree granting institutions.
Program-level SLOs are expressed in a way that they can be observed in student work.
PSLOs often ask students to demonstrate their learning on different levels, from the most
basic levels of knowledge and comprehension, through application and analysis, to the
more advanced levels of synthesis and evaluation.
 Knowledge refers to rote memorization, recognition, or recall of facts.
 Comprehension is the demonstration of what the facts mean.
 Application is the correct use of facts, rules, or ideas.
 Analysis is breaking down information into component parts.
 Synthesis is the combination of facts, ideas or information to make a new whole.
 Evaluation is judging or forming an opinion about a situation or information.
2
Use active verbs for writing PSLOs such as those in the following table:
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Choose, define, find, identify, label, list, match, name, recall,
show, spell, state, tell, select
Classify, compare, contrast, demonstrate, describe, differentiate,
explain, extend, outline, paraphrase, rephrase, summarize, show,
translate
Apply, build, calculate, choose, construct, develop, experiment
with, illustrate, interpret, interview, manipulate, model, modify,
organize, plan, relate, select, solve, utilize
Analyze, categorize, conclude, deduce, discover, discuss, dissect,
distinguish, divide, examine, infer, inspect, simplify, state
assumptions, state relationships, survey
Adapt, change, combine, compile, construct, create, design,
develop, estimate, formulate, hypothesize, imagine, improve,
invent, justify, originate, predict, report, solve, support, theorize
Appraise, assess, criticize, defend, determine, disprove, estimate,
evaluate, judge, rate, recommend, rule on, select, test
State the major PSLOs for this academic degree program in the following box:
3
1.
2.
3.
4.
5.
4
II. CURRICULUM MAPPING
Curriculum mapping shows where in the required program courses (and other
required elements) the students acquire knowledge and skills needed to meet the
program-level student learning outcomes (PSLOs). In the curriculum map below,
write the names of the required courses (and any other requirements) for the
academic program across the top. Down the left hand side of the table, write the
PSLOs for this academic program. Then identify where the student acquires the
knowledge and skills pertinent to each PSLO by placing an X in the corresponding
box. For example, PSLO #1 may be pertinent in Required Course #1.
Major Requirements
Upper-Division Core
Programmatic
Student
Learning
Outcomes
(PSLOs)
PSLO #1:
Required
Course #1
Required
Course #2
Required
Course
#3
Required
Course
#4
Required
Course #5
PSLO #2:
PSLO #3:
PSLO #4:
PSLO #5:
Please note that this table is an abbreviated version of the official matrix that has
been provided for you in the form of an excel spreadsheet. It does not show the
elective or other major requirements. There may be other required activities
outside of courses that also help students attain the outcomes, such as participation
in colloquia, community service, performances, student or honorary societies,
intercollegiate competitions, internships, etc. You can include these in additional
boxes for Major Requirements.
Note: Instead of an X, you may want to indicate where the PSLOs are first
introduced (I), then reinforced (R), and finally applied at an advanced level (A).
A curriculum map can be a very useful tool. For example, you may find that some
PSLOs are not covered by any courses. Another thing to look for is a course that
does not contribute to student learning on any of the PSLOs. Two sequential
courses may not be reinforcing the same PSLOs. There may be unnecessary
overlap where nearly all courses cover the same PSLO in the same way. In that
5
case, some courses can be redesigned to cover other PSLOs that are less well
addressed.
The curriculum map is also quite useful in setting up an assessment plan for the
PSLOs (discussed later). Be selective in your choice of which courses contribute to
which PSLOs. Only mark an X in the box if that PSLO is a major component of that
course. That is, the course provides instruction concerning the PSLO (whether it is
communication skills, critical thinking, or knowledge of disciplinary concepts) and
the student produces work as a required part of the course that demonstrates his
or her learning on the PSLO and that forms part of the student’s course grade. You
can revisit the curriculum map to indicate in each box the assignment that the
student completes as evidence of their learning on that PSLO.
III. EVIDENCE OF STUDENT LEARNING
Teaching and learning strategies are the types of instruction (e.g., lecture, small
group discussion, laboratory, case study, online discussion boards, peer reviews of
student work, team projects, etc.) adopted in courses and other parts of the
curriculum (e.g., internships, colloquia, field research, service learning) to facilitate
student attainment of the learning goals. Teaching and learning also includes the
types of assignments that students complete in the course, such as tests, term
papers, journals, lab reports, homework, oral presentations, team projects, etc.
Evidence of Student Learning is generally either direct or indirect. Direct evidence
consists of examples of student work. Indirect evidence consists of surveys,
opinions, exit interviews, etc. Some examples are provided below.
Type of
Evidence
Direct
Evidence
Indirect
Evidence
Examples
Annotated Bibliography
Assessment Center/Mock Interviews
Blogging
Case Study
Comprehensive Exam
Community Public Service Project
Embedded exam questions
Individual/Team Presentation
Internship
Journaling
Language proficiency exam
Oral presentation
Portfolio
Policy Analysis Project
Role-Playing Simulation
Standardized Test
Alumni survey
Course evaluation by students
Employer survey
Exit interview
Student self-evaluation
Student survey
Team Project
Technology Project
Term/Research Paper
Thesis/Project
(Individual)
Videotaped presentation
Written paper
Wiki
Gathering Student Work to Use as Evidence of Student Learning can be easily
accomplished by using “embedded” assessment. This means collecting work that
6
students already complete as a required part of a course and that counts toward
the course grade. The curriculum map indicates where such student work can be
found (in which courses).
The student work (or a copy of it) that will be used for assessment of student
learning can be gathered during the course of the semester. Students can turn in
two copies of the assignment, of which one is graded and returned to the student
and the other is retained for assessment.
When developing an assessment plan, it is important to remember that it is not
generally necessary to collect evidence on every student learning outcome, from
every course, from every student, every semester. Rather, select one or two
outcomes to focus on each year and collect a sample of student work from one or
two courses on that outcome. A different outcome can be assessed each year until
all outcomes have been assessed at least once in any five year period.
The assessment plan should include a detailed plan for the systematic collection of
direct evidence of student proficiency in the learning outcomes. Evidence may be
collected only once, or it may be collected multiple times over the life of the
student’s tenure in the program (e.g., at the beginning, middle, and end). Different
learning outcomes may be assessed with different forms of direct evidence.
Describe the direct evidence of student learning that is already available from
courses that focus on each learning outcome:
PSLO #1: __________________________________________________________
Evidence 1:
Evidence 2:
PSLO #2: __________________________________________________________
Evidence 1
Evidence 2:
PSLO #3: __________________________________________________________
Evidence 1:
Evidence 2:
PSLO #4: __________________________________________________________
Evidence 1:
Evidence 2:
PSLO #5: __________________________________________________________
Evidence 1
Evidence 2:
7
IV. Analysis or Judgment of Evidence
Analysis or Judgment of Evidence is the application of faculty expertise to the
evidence of student learning. How well have students demonstrated their learning?
Have students met faculty expectations for learning? This usually involves
application of a rubric or scoring system to the evidence of student work.
What are the scoring criteria or rubric that will be used for examining student
learning for each of the learning outcomes, for each type of evidence that was
collected? For example, a (very simple) rubric for oral presentations might look like
this:
Oral Presentation Rubric
Criteria
Visual aids
Speech,
grammar,
voice
Geared to the
audience
Meets time
limit
1: Below
expectations
None; or not used; or
poor quality;
Too soft or too loud;
poor grammar;
inappropriate words
Not appropriate for
the audience
Presentation is too
short or too long
2. Meets
expectations
Uses visual aids; satisfactory
quality
Speaking voice, word choice,
grammar, are all satisfactory
3: Exceeds
expectations
Visual aids of superior quality;
support audience learning
Excellent speaking voice;
appropriate speech; no errors
Appropriate to the expected
audience
Presentation meets specified
time limit
Easily adjusts to unexpected
developments
Presentation makes excellent
use of time
The choice of criteria is up to the faculty as is the definition of the expected levels
of performance (e.g., what goes into each of the boxes defined as below
expectations, meets expectations, exceeds expectations).
In addition to creating a rubric or scoring guide, the faculty may also wish to set a
benchmark or level of expectation for student learning. For example, you expect
that the average student score on a final exam will be higher than 75%. Or it may
be that at least 85% of students will receive a score of meets expectations or better
on an outcome such as oral presentations. Or that more than half of all students
will show improvement on a PSLO over time.
Let’s say that all students in courses that require oral presentations were
videotaped making their presentations in class. After the semester is over, a small
group of faculty watch a random sample of the videotapes and score them using
the rubric above. The scoring sheets are collected and the results are analyzed.
Student performance on this PSLO can be summarized as shown below. For
example, on PSLO #1, you may find that 5% of student work was scored as below
expectations, 75% scored as meets expectations, and 20% scored as exceeds
expectations. In the comments section, you could note the important conclusions
of the faculty scoring group.
8
PSLO #1
Assessment of
Evidence
Below expectations
Meets expectations
Exceeds expectations
5%
75%
20%
Comments: Most students did well on the four criteria for oral presentations but
those that did not were unable to meet the time limit.
V. FEEDBACK FOR PROGRAM IMPROVEMENT
A feedback mechanism must be adopted to communicate the results of assessment.
How will the results of assessment be communicated to all program faculty? It
could be through an annual retreat, a special department meeting, or as regular
items on a faculty meeting agenda. How will your program communicate the
results of assessment?
Program improvement is the careful change of aspects of the program, based on
the analysis of evidence of student learning, in order to increase the level of
student attainment of the PSLOs. Program improvement is a good way to “close
the loop” on assessment, by incorporating evidence about student learning into
program change. What are some examples of program change that have been
made based on evidence from assessment of student learning?












No Changes – all students met faculty expectations
Course based changes, e.g., content, teaching & learning methods
Created new requirements, e.g., portfolio
Changed course offerings, scheduling, sequencing
Revised internship, capstone, exit requirements (e.g., thesis)
Added/Deleted courses
Created student orientation, handbook, organization
Created capstone
Eliminated a concentration
Petitioned for impacted status, changed entry requirements
Adopted pre-requisites
Changed advising system for majors/grad students
Changes can result in better student attainment of PSLOs, more students meeting
faculty expectations, more students graduating in a timely manner, fewer students
dropping out, more students earning recognition in their field, better student
placement upon graduation, and so forth.
9
VI. SAMPLE ASSESSMENT PLAN
Sample Timetable for Assessment of PSLOs
TIME
STEP
RESPONSIBLE PARTY
Fall
2012
Review PSLOs; align PSLOs with
requirements for the major and with the
LEAP goals
Spring
2013
Complete all matrices; Establish 5-year
assessment plan; identify evidence to be
collected and analyzed; adopt rubrics
and/or benchmarks for student
performance
Fall 2013
Implement assessment cycle for Year 1;
collect evidence of student learning, and
analyze;
Spring
2014
Report findings of assessment and
proposed program changes; provide
details on Year 2 assessment plans
Fall 2014
Implement assessment cycle for Year 2;
collect evidence of student learning, and
analyze;
Spring
2015
Report findings of assessment and
proposed program changes; provide
details on Year 3 assessment plans
For your academic program, you can decide who will carry out each of the tasks
associated with assessment. Ideally it should be a group of faculty who are familiar
with each PSLO to be assessed. Instructors will need to identify the type of
evidence to be collected and be responsible for collecting the evidence. Instructors
will need to assess the evidence collected, analyze the results, and prepare a brief
report. Program faculty will need to consider the evidence of student learning and
decide whether program changes are warranted and, if so, which ones. Finally
responsibility will need to be assigned for completing the annual assessment
reports. See a modified version of the required program assessment schedule on
the next page. For more information, please contact the resources listed on your
handout.
10
Division of Academic Affairs
Annual Program Assessment Schedule
For Undergraduate Degree Programs
2012-2019
Program Information
Name of Program:
College:
Prepared by:
(Department
Chair/Program
Coordinator/Program
Assessment
Coordinator))
Email Address:
Date:
Telephone Extension:
Instructions: Each Program Assessment Schedule must indicate the plan for assessing all PSLOs during the program
planning cycle. Departments may assess any combinations of PSLOs in a given year, but they must assess all PSLOs
in a program review cycle (5 years of data collection; 6th year spent summarizing data for Program Review).
Program Student Learning
Outcomes (PSLOs):
How will this PSLO be assessed? (e.g., How will the data be used?
embedded questions on exam;
pre/post test; assignment scored by
rubric)
PSLO 1: (EXAMPLE) Conduct
original psychological
research and report results in
writing and orally to scientific
audiences.
Lab courses: Students complete
original scientific experiments, write
them up in APA style scientific format,
and present the findings orally to the
class in a poster presentation. Scored
by rubrics.
PSLO 2:
PSLO 3:
PSLO 4:
PSLO 5:
(Add additional rows if/as needed.)
When will
this PSLO
be
assessed*?
Instructors share students’
AY: 13-14
scores on various criteria on
the rubrics (e.g., statement of
hypothesis) with the
department. Department
provides feedback to
instructors to highlight
successes and identify areas in
need of additional support
and/or resources.
AY:
AY:
AY:
AY:
* Do not use the Program Review year for measuring any of the PSLOs; programs summarize data they have
collected up to this point and prepare the program review report during this year. See the Annual Assessment
2012-2013 Instructions document for an example of how assessment activities can be scheduled so as not to
overwhelm faculty (e.g., for each PSLO, refining the assessment plan in one year, measuring the PSLO in the next
year, and implementing changes based on assessment findings the third year). Programs wishing to do follow-up
assessment to see the effect of these changes may apply for additional assessment funding beyond the usual
amount intended for the assessment activity already planned.
11