Program Assessment
Professional Development Days
August 25th and 26th 2009
Workshop Learning Outcomes
•
•
•
•
Describe the purpose of program assessment
Outline the steps in program assessment
Write measurable program learning outcomes
Choose program assessment delivery points
Your worksheets will be collected at the end and
returned to you next week.
What is the Purpose?
Program “assessment is a framework for
focusing faculty attention on student learning
and for providing meaningful discussions of
program objectives, curricular organization,
pedagogy, and student development.”
Mary Allen, Assessing Academic Programs in Higher Education
The Goal is Improvement
Program assessment concentrates on the group
of students in a program.
To improve:
Courses
Programs
Services
Planning
Budgeting
Last but not least - Accreditation
By 2012, the ACCJC expects us to:
• Have SLOs for all courses, programs and the
institution.
• Have assessed SLOs at all levels
• Have reported and discussed the results and
used them to guide decision-making and
improvements.
Assessment Levels
Institutional
Program
Course
• Institutional outcomes
should be reflected in
the programs.
• Program outcomes
should be reflected in
the courses.
What is a program?
It’s up to the college to decide how to group activities for
assessment purposes. Some common ones:
• Sequence of courses leading to Degree or Certificate
• General Education
• Basic Skills
• Library, Learning Assistance Center, etc.
• Special sequences for groups of students like PACE,
Honors, Extension
• Student services (not addressed here)
By our normal way of defining
“program”, there are over 100!
How can we get it all done?
One step at a time!
1. Identify assessable programs
2. Develop program learning outcomes
3. Develop an assessment plan
4. Collect assessment data, analyze and report
5. Use results to improve the program
This workshop concentrates on 1, 2 and 3
Identify Assessable Programs
S
T
E
P
1
For efficiency, create groupings of programs
with similar goals and courses:
– CSU General Education + Graduation Plan A +
Graduation Plan B + IGETC + Liberal Studies
Degree + ?
– Business + Real Estate + Accounting
– Industrial Engineering + Electronic
Engineering + Engineering Technology + ?
– Is Political Science a program?
Identify Programs
S
T
E
P
1
Set up a focus group to work on the next
steps. It could include:
• Division chair
• Faculty point person
• Full time and adjunct faculty
• Students
Worksheet and Discussion
S
T
E
P
1
Take a few minutes to answer the Step 1
questions on your worksheet.
2. Write Program SLOs
S
T
E
P
2
A. Review LAHC institutional SLOs
B. Look at course SLOs in your program
C. Search the Web for outcomes to
“borrow”
D. Describe the ideal graduate
E. Check draft SLOs for measurability
F. Build course to program matrix
Institutional SLOs (ISLO)
S
T
E
P
2
A
5 ISLOs (handouts and campus web site).
Modify them to be specific to your
program. They don’t ALL have to fit but
most will.
This makes it easier for us to conduct
institutional assessment!
ISLO #1 Communication
S
T
E
P
2
A
Use language and non-verbal modes of
expression appropriate to the audience
and purpose.
Example SLO for a Child Development
program:
Teach and care for young children using
effective communication and relationship
skills.
ISLO #2 Cognition
S
T
E
P
2
A
Use critical thinking skills to analyze,
synthesize and evaluate ideas and
information.
Example SLO for a Math program:
Estimate and check answers to
mathematical problems in order to
determine reasonableness, identify
alternatives, and select optimal results.
ISLO #3 Information Competency
S
T
E
P
2
A
Utilize research skills necessary to achieve
educational, and professional objectives.
Example SLO for a History program:
Use primary and secondary sources to
construct sophisticated, persuasive, and
logical interpretations of historical
problems and events
ISLO #4 Social Responsibility
S
T
E
P
2
A
Demonstrate sensitivity to and respect for
others and participate actively in group
decision making.
Example SLO for a Computer Science
program:
Analyze the global impact of computing on
individuals, organizations, and society.
ISLO #5 Personal Development
S
T
E
P
2
A
Demonstrate self-management, maturity, and
growth through practices that promote
physical, mental and emotional well-being.
Example SLO for a Human Services program:
Identify their personal values and articulate
how their value systems inform and
influence their professional practices.
Look at the course SLOs
S
T
E
P
2
B
• Find repeating
– Words
– Activities
– Ideas
• Course assessment forms are on the
campus web site (Faculty/Staff, Student
Learning Outcomes/Assessment, by
department)
Course SLO Word Cloud
History program word cloud created at wordle.net
Search the Web
S
T
E
P
2
C
• Other institutions
Use www.google.com or www.bing.com to search.
Search: history program learning outcomes
Example: University of Maryland
http://www.ugst.umd.edu/core/LearningOutcome.
htm
• Professional Organizations
– Go to the site and search for learning outcomes
ACM (Computer Science)
ABET (Engineering)
MAA (Math)
Warning: you will find bad SLOs and good ones.
The Ideal Graduate
S
T
E
P
2
D
• Look at program review
– Mission and goals
• Advisory committee members
• Transfer institution contacts
– Call or email and ask what the students
transferring in need to have
– What do they find lacking?
The Ideal Graduate
S
T
E
P
2
D
Search the Internet or ask Bob Richards for help locating data.
For students studying computers:
According to Kelly IT Resources, there are a number of technical
skills currently in demand, including Java 2 Platform, Enterprise
Edition (J2EE), Visual Java++, WebSphere, ERP, and CRM.
However, the IT worker who wants to get ahead needs more than
just technical savvy. In a survey conducted by the Journal of
Computer Information Systems, results indicated that attributes
including, “interpersonal and personal skills/traits, such as team
skills, communication skills, critical/creative thinking skills, and
personal motivation” are often valued higher than the
traditional technical requirements of a job.
From “IN-DEMAND IT SKILL SETS: WHAT EMPLOYERS WANT, WHAT WORKERS NEED” Published
by Kelly IT Resources at http://www.kellyit.com.pr/res/content/us/kit/en/docs/kitr_indemand_skillsets.pdf
Check: Measurability
S
T
E
P
2
E
• Use active and specific terms. See
Bloom’s taxonomy (in handouts)
• Don’t use words like “understand” – go
for higher level skills
• The meaning of a word can be defined
more clearly in the scoring rubric. For
example, “sophisticated” from the
history SLO.
Check: Simplicity
S
T
E
P
2
E
• Keep the number down to around 5
• Have more if your outcomes are
determined by an outside accrediting
body like Nursing.
• Concentrate on the most important skills
or knowledge the students will come
away from the program with and what
they will produce to show you that they
have mastered those skills.
Worksheet and Discussion
S
T
E
P
2
E
Take a few minutes to answer the Step 2
questions on your worksheet.
Check Alignment with Courses
S
T
E
P
2
F
Curriculum Alignment Matrix
Course
PSLO
#1
100
I
120
200
PSLO
#2
PSLO
#3
PSLO
#4
PSLO
#5
I
I
P
PSLO
#6
P
P
P
204
P
300
P
329
D
400
P
P
P
D
480
490
D
D
D
D
A matrix ensures
that student
have been
exposed to
the outcome
in the
program.
Can point out
problems!
From Assessing Academic Programs in Higher Education by Mary J. Allen p. 43
Program Assessment Form
S
T
E
P
2
– Blank form is a document on the campus
Assessment Web site
– Download and type in your program SLOs
with Word
– Send document to Division Chair who will
forward to Dean and Assessment
Coordinator for Web publication.
– Program Learning Outcomes are also to be
published in the college catalog and all
other program materials.
Develop Assessment Plan
S
T
E
P
3
A.
B.
C.
D.
Select delivery points for assessment
Select tool(s)
Set timeline
Create assessment and rubric
Delivery Point
S
T
E
P
3
A
Is there a course or set of courses all
majors would take before graduating?
If not, is there a course many majors take?
Give a survey to identify who is a
major.
See Accounting program example on
handouts.
Delivery Point
S
T
E
P
3
A
If there isn’t a terminal course,
• Can you reach students as they are
graduating to give out an assessment?
– One campus had a free lunch for students
who filed petitions
• Can you have students build a portfolio
as they progress through the program?
Worksheet and Discussion
S
T
E
P
3
A
Take a few minutes to answer the Step 3
questions on your worksheet.
Choose Assessment Tool(s)
S
T
E
P
3
B
• Direct Assessment
• Indirect Assessment
Experts recommend more than one type of
data collection method!
Choose Assessment Tool(s)
S
T
E
P
3
B
Direct methods prompt students to represent of
demonstrate their learning or produce work so that
observers can assess how well students’ texts or
responses fit expectations
Includes:
Standardized tests
Locally developed tests and assignments
Projects
Presentations
Portfolios
See handouts
Choose Assessment Tool(s)
S
T
E
P
3
B
Indirect methods capture students’
perceptions of their learning and the
educational environment that supports
that learning.
– Student surveys
– Factbook information
Definitions of direct and indirect from Assessing
for Learning by Peggy Maki p. 88
Set Timeline
S
T
E
P
3
C
– Which semester/year will each SLO be
assessed during?
– The general recommendation is to measure
one per year to give time for careful review
of results and integration into planning
– Send the timeline to the Division Chair and
to the Assessment Coordinator
Create Assessment and Rubric
S
T
E
P
3
D
– Develop the tool (test questions,
assignment, survey, etc.)
– Create the scoring guide or rubric
– Get input from all of the faculty who will be
involved.
For Future Workshops?
• Designing assessments and rubrics
• Collecting assessment data, analyzing
and reporting
• Using results to improve the program
You tell us!
Thank You For Attending!
• Please turn in:
The Worksheet
The Assessment Needs Survey
The evaluation form
• Presentation will be available on the
campus assessment web pages.
• Questions?
– Lora Lane x4178 or campus email
– Dean Humphreys x4025 or campus email