2006
Assessment Symposium
WELCOME!
May 22, 2006
Traditional Undergraduate
Retention
Cathy Alexander, Maria Kohnke,
Damien Peña
Assessment Goal & Design
• Review the status of
first-time traditional
undergraduate
freshmen by entering
cohorts
• Identify best practices
on campus
• Implement programs
aimed at improving
retention and
graduation rates
• Data reviewed using
Fall 99 through Fall 05
census snapshots
• Data separated by
categories identified by
team
• Reported data as
attrition versus retention
for simplification
Data Analysis
California Lutheran University
First-Time Freshmen 4 Year Average Attrition
Fall 99 - Fall 04 Cohorts
34.60%
34.10%
37.60%
26.30%
4th Year
41.80%
30.30%
30.20%
33.60%
26.30%
3rd Year
37.80%
41.70%
47.40%
20.20%
18.60%
21.80%
16.70%
2nd Year
29.30%
32.00%
33.60%
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
Attrition Percentage
Qualifying Spring Freshmen
Conditional Admits
SSS Eligible (not serviced)
SSS
Underrepresented
Note: Some categories Include duplicated counts
Caucasian
Overall
Findings: Attrition After Start of Senior Year
6 year Data (some data includes duplicated counts)
CLU Overall = 38%
Total Students FY 03-04
372
35% cumulative fourth year
3% after start of senior year
Underrepresented Students = 56%
82
38% cumulative fourth year
18% after start of senior year
Caucasian = 36%
261
34% cumulative fourth year
2% after start of senior year
SSS Students = 26%
26% cumulative third year (0% fourth year)
0% after start of senior year
24
includes 17 underrepresented of which 3 are
conditional admits
6 total conditional admits
Changes Implemented & Results
• Created Academic Assistance Program
(AAP) modeled after student success
plans in Student Support Services
• Funded program by assessing an
advising fee
• 6% improvement in retention for all
freshmen and 10.4% increase for
conditional admits
Goals for 06-07 & Beyond
• Longitudinal study using quantitative and
qualitative data (retention data & exit surveys)
• Expand services provided beyond entering
conditional admits and probation students
• Monitor third year retention and graduation
rates
• Gather and analyze data around “stuck
seniors” and determine methods to increase
degree completion
Exercise Science and Sports
Medicine
Program Review
Michele LeBlanc
Exercise Science and Sports
Medicine Department
• Offers a Bachelor of
Science degree with 4
emphasis options
–
–
–
–
Athletic Training
Human Performance
Pre-Physical Therapy
Physical Education
(Teaching)
• Offers activity program
which ALL CLU students
are required to participate
in for at least one
semester
• Four full-time faculty
– Jim Hand, M.S., A.T.C.
– Kavin Tsang, Ph.D.,
A.T.C.
– Michele LeBlanc, Ph.D.
– Janie Rider, Ph.D.
• Approximately 17 adjunct
instructors each semester
• Approximately 140
majors
• About 60 different classes
scheduled each semester
2005 – 2006 Program Review Goal
• Timeline/Context
– Started Program Review during 2004-2005 AY
– Repeated change in departmental personnel/leadership
– Department changed from Social Science Division to
Natural Science Division
• Program Review Considerations
–
–
–
–
Facilities and Faculty
Curriculum
Engagement/Scholarship
Etc.
Qualitative Research Design
• Purpose: Discover with faculty most pressing issues for PR
• Process:
- Conduct multiple focus group interviews
- Interview faculty one-on-one
- Plan to use Flashlight survey for student input also
• What data was collected?
– What are the major goals of our department?
– What are the department’s 3 greatest strengths/
weaknesses?
– What do you see as the greatest priorities in the department
this year? Next 5 years?
– What are your personal goals for this year? Next 5 years?
Analysis of Data: Emerging themes
• Need for departmental unity/community
– Improve communications within department
– Factions of students within dept. have established
community, but there is no cohesiveness to collective
department
• Raising the bar – upgrading overall quality of
department
– Involve students in activities outside the classroom (e.g.
research, conferences)
– Have faculty more involved across campus
• Concern for over-responsibilitied faculty and high
turnover, desire to do more, but how?
Lessons Learned/Benefits
• Common ground identified
– Curriculum  Community
• January workshop tied these together
– Enhanced curriculum created
– Cannot implement without additional faculty
• Moving forward…
– External reviewer – coming to terms with reality
– Position request – hopes for saner reality 
– Future external reviews/accreditation – need to take
baby steps
Media Habits Survey
Cody Hartley
Marketing for
Graduate & Adult Programs
July 2005 Assessment Goal
Survey Purpose: To determine the media
preferences of current ADEP and MBA students
so that chosen media targets would be accurate
• Understand current student media consumption
• Investigate if promotional tactics were
connecting with students in our target population
• Create media tracking measurement system
• Assess the deployment of resources
Assessment Design
Paper survey
• administered in class
• administered by faculty
Two week period
Survey results were entered and tabulated
electronically
Data Collection
•
•
•
•
•
•
Logistical information
TV usage and preferences
Newspaper readership
Radio listenership
Internet preferences
Other relevant information
Analysis of Data
•
•
Logistical and Demographic Information
Media Information Sources
–
–
–
–
•
Television
Newspaper
Radio
Internet preferences
Other relevant information
– Referrals
– Incentives
Outcomes and Future Goals
Outcomes:
• Confirmed low priority for television
• Shifted advertising from print to internet
• Refined radio advertising based on geography
• Developed a “Free Books” referral program
• Created media tracking measurement system
Future Goals:
• Increase by 20% the total amount of traffic driven to the
microsite via advertising
- Radio
- Print
- College Guides
- Internet
• Harness the power of referrals
Communications Program Review
Sharon Docter, Chair and Professor
Beverly Kelley, Professor
Dru Pagliassotti, Associate Professor
Russell Stockard, Assistant Professor
Jing Jiang, Instructor
Beth Le Poire Molineux, Associate Professor (half-time)
The mission of the Communication Department is to graduate majors who are
highly skilled, critical, and ethical media producers, practitioners, and analysts
who are able to do socially meaningful, theoretically rich work.
2005 – 06 Program Review Goals
Examined:
• Trends (Enrollment and Faculty-Student Ratios)
• Teaching Effectiveness
• Scholarly Productivity
• Challenge and Engagement
• Diversity Initiatives
• Critique of Curriculum
• Co-Curricular and Internship Opportunities
• Departmental Student Learning Outcomes
Design:
Challenge and Engagement
• Examined aggregate scores on student course
evaluation questions pertaining to challenge and
engagement
• Compared departmental score to score for all
traditional undergraduates
• Compared departmental grade distributions to
mean grade distributions for Arts and Sciences,
Creative Arts Division, and Social and
Behavioral Sciences Division
Data
Challenge and Engagement Course Evaluation Questions:
Communication Department: Engagement
4.37
Traditional Undergraduate: Engagement
4.43
Communication Department: Challenge
4.24
Traditional Undergraduate: Challenge
4.39
Grade Distributions:
Spring ‘05
Fall ‘04
Spring ‘04
Fall ‘03
Spring ‘03
Communication
3.24
3.18
3.23
3.27
3.15
Arts &Sciences
3.23
3.13
3.22
3.15
3.25
Soc. Sciences
3.20
3.12
3.19
3.04
3.0
Creative Arts
3.59
3.51
3.46
3.53
3.52
Analysis of Data
• Slightly below traditional undergraduate
average on measures of challenge
• Above traditional undergraduate average
on measures of engagement
• Departmental grade distributions
comparable to Arts and Science averages
• Departmental grade distribution slightly
above Social and Behavioral Sciences
average and below Creative Arts average
Recommendations
• Sustain high level of student engagement
• Institute research methods course requirement
– (Need additional faculty FTE)
• Implement Departmental Honors Program
• Require students to read more primary
research and reflect on readings through
response papers
• Research papers at the lower division level
• Provide faculty with data concerning grade
distributions and work with faculty to curb grade
inflation
A Problem-Based Learning (PBL)
Approach to Teaching
Information Literacy (IL) Concepts
to Freshman Seminar Students
Henri Mondschein
ISS/Pearson Library
Purpose
To determine information literacy
concept levels of freshmen seminar
students and to discover student
perceptions of information literacy
concepts.
Assessment Design and
Data Collection
• Quantitative Design:
– Pre-test/Post-test of Information Literacy Knowledge (UCLA
Library Instructional Services Advisory Committee
Questionnaire)
– Work samples rubric
• Qualitative Design:
– Direct observation of research skills 5-10 seminar
participants
– Interviews of 5-10 seminar participants
• Data Collection
– Pre-test/Post-test of Information Literacy Knowledge
– Student work samples
Analysis of Data
• Student Work:
– Pre/post-tests
– Samples using rubrics
– Artifacts (papers, journals)
• Interview Transcripts:
– Transcribing, coding and analyzing interviews for patterns
and themes using qualitative analysis software (Qualrus)
– Identify major themes re: PBL & IL
• Evidence of:
– enhanced information literacy competencies
– engagement, intentional learning, and self-directed learning
Preliminary Findings
•
Freshmen enter CLU with diverse information
literacy competencies
•
Freshmen identify engaging classes as consisting
of active learning, debates, discussion and group
learning activities
Future Goals
• Implement Assessment Design
• Create lesson plans and assignments based
on findings
• Share results with faculty
• Design a freshman seminar that increases
information competencies and promotes
student engagement through problem-based
learning model
Computer Science
Undergraduate Program Review
Chang-Shyh Peng
Computer Science Department
• Degrees offered (Traditional & ADEP)
– B.S. in CS, CIS (48 credits)
– Certificate in IT, IS (32 credits)
– Minor in CS, CIS (20 credits)
• Faculty
– Full time: Chang-Shyh Peng, Myungsook Klassen, Fenglien
Lee, Craig Reinhart
– Additional Adjuncts
• Milestones
–
–
–
–
Data collection & analysis: Fall 2004 ~ Fall 2005
Completion of review report: 3/16/2006
External reviewer’s visit: 4/25/2006
External reviewer’s report: 5/4/2006
External Reviewer
Process
• Included in Program
Review process
• Reviewer’s Contribution
– Outsider’s viewpoint
– Evaluation of current
status
– Feasibility of future plan
Goals
• Curriculum
– Current structure
– Areas for improvement
– Direction for
advancement
• Faculty
– Expertise
– Load
– Future growth
• Student body
– Enrollment
– Diversity
• Supporting resources
– Existing
– What it takes
Data Collection
• Internal
– Class evaluations
– Departmental assessment plan
– Data from Registrar and Provost’s Office
• External
– Curriculum at comparable institution
• APU, Chapman, Cal Poly, CSUN, Humbolt, PLU, Pomona,
Redland, Westmont, …
–
–
–
–
ACM/IEEE Curriculum Task Force Report
ACM Job Migration Task Force Report
US Dept of Labor Reports
Industry outlooks
Analysis and Outcomes
• Commendations
–
–
–
–
Dedicated faculty
Advanced Curriculum
Innovative 5-yr BS/MS Program
Well-designed labs
• Recommendations
–
–
–
–
Support and retain qualified faculty
“Fit" of the department
Recruitment of female and international students
Course redesign (discrete) and realignment (subjects with
language emphasis)
– Exploration of “Project Center”
– Sufficient and modern lab facility
– Supporting personnel
Lessons Learned
• Know your external reviewer
–
–
–
–
Credentials
Academic experience
Program Review experience
Current knowledge of the field
• Start early
– Availability of report
– Prepare review report and external reviewer
questions long before reviewer’s arrival
• Utilize valuable internal resources
Pearson Library’s Role with
Challenge and Engagement
Julius Bianchi
Introduction
• Regents requested a strategic plan for Pearson
Library (Spring 03)
• Draft Plan presented Fall 04
2005-2006 Goal
• What goal was set?
– To process all Inter-Library Loans (ILL’s) in a
timely manner
– To provide easy access to and streamlined on-line
forms
• What aspects of student learning / service will be
assessed?
– Circulation transactions and ILL’s
Assessment Design
• Identified peer
institutions to establish
base line data and
benchmarks
• Used Education
Department’s National
Center for Educational
Statistics Bi-Annual
Academic Library
Census
Data Collection
• Pearson Library
collects:
– Circulation statistics
– Inter-Library Loans
– Materials budget
• Pearson Library reports
this data to Education
Department including
IPEDS
Group
CLU
Peer
State
Nation
2000 loans received 2002 loans received
1,410
1,680
4,435
4,333
1,886
1,990
2,506
2,404
2006-2007 Actions to Promote ILL
• Free information specialists from reference
desk responsibilities
• Promote liaison program - information
specialist / faculty collaboration
• Advertise and market ILL
• Include information on ILL in all bibliographic
instruction sessions
Course Schedule Submission
Continual Improvement Process
Melanie Clarey
Fall 2005 Assessment Goal
CIP identified two areas for review:
1. Review the processes within the two
month timeline
2. Ensure the integrity of the data
– Control of input
– Control of sources
– Review communication process
•
•
Sources
Budget implications
Assessment Design
Measures used to collect data:
1. Design flow charts to track processes and
times
2. Integrity of data count:
–
–
–
The number of corrections to adjunct contracts
necessary.
The number of times the data was being entered
into a spreadsheet or Datatel.
The number of people “handling” the data
throughout the current process.
Data Collection
• Each course required input on information into 7
separate screens of Datatel.
• Each semester information was being entered into a
spreadsheet or Datatel no fewer than 5 times.
• Each semester 4 people have to enter the same
information into different systems are different times.
• Each semester the Provost office received “endless”
phone calls from adjunct professors regarding changes
to contracts.
Analysis of Data
• The number of times information was entered increased
the chances for error and discrepancies.
• Program Directors and Deans were being held
accountable for their adjunct budget yet were not given
the tools or control over the system necessary to monitor
these things appropriately.
• Discrepancies were found in data housed in Datatel.
• The entire review process took the Committee 2 months
to complete.
Recommendations & Implications
Recommendations:
• Create one screen in Datatel for information input.
• Input information directly into Datatel by the Schools and College.
• Retrieve data from Datatel. Provost and Registrar Offices each
obtain identical information.
Implications:
•
•
•
•
Timeline for the entire process has been cut in half.
Datatel will be able to give an accurate account of historical data.
Fewer corrections will be needed for adjunct contracts.
More accurate predictions of budget implications of adjunct
contracts.
The SOAR Program:
A summer bridge component of the
Student Support Services Program
Funded by grants from the U.S.
Department of Education & the
James Irvine Foundation
Tuula Mattson, MS
SSS Director
Damien A. Peña, MSW
Dir. of TRIO Programs
SOAR Mission
To provide incoming Freshman Student Support Services students an orientation to a
college environment and provide them with opportunities to connect with other first
generation students.
To assist participants in the process of academic exploration by introducing them to
relevant resources and challenging them to examine their values and interests.
To facilitate participants' transition to a college environment and to track their adjustment
through the SSS Peer Mentor program.
To provide parents of the incoming students a opportunity to answer questions and gain
information on their student's academic journey.
2005 – 2006 Goals
• Increase the total number of SOAR Program days from 3 to 7 days.
• Subsidize housing and meals costs that are not allowable for SSS as
outlined in the federal guidelines.
• Increase participation in SSS Programs and services for SSS-eligible but
not served students.
• Secure funding to cover the costs after the Irvine grant expires.
• Instill a sense of student identification and involvement with the University.
Assessment Design
Criteria for student selection
• Deposited FTF entering CLU.
• First-generation and/or low income status
as defined by Department of Education
federal guidelines.
• Confirmed through information filed in
FAFSA and conversations with Financial
Aid and Admissions staffs.
Data Collection
• Collected evaluations at the end of the
program to rate the effectiveness of the
presentations, activities, workshops, and
staff.
• SSS reviewed the following:
– Retention data for participants at the end of
fall and spring.
– GPA’s in fall and spring.
– Involvement of participants in CLU and SSS
Activities.
Analysis of Data
SOAR Particpation:
• 87 students have taken part in the SOAR program thus far.
• All 87 received an educational plan and advisement for their
academic career at CLU.
• 25 students attended SOAR in 2005, and 40 students will have an
opportunity in summer 2006.
CLU Involvement:
• All 87 were assigned an SSS Peer Mentor.
• 8 SOAR students ran for offices in the ASCLU elections; with four
serving as Freshmen Senators in their first year.
• 3 SOAR students started new clubs on campus and were
recognized by ASCLU: Crusade for Justice, Purple Pit, and H2O.
• 11 SOAR students have served as Peer Mentors for the SSS
program with 5 serving as tutors for the program.
Academic Success:
• 97% of all SSS students have higher than a 2.25 GPA.
• AY 2005-2006 will mark the first graduating SOAR class and
tracking / retention numbers will follow.
2006 – 2007 Goals
• Program Goals
– Continued funding of the SOAR program
– Expand the participation of the program
– Expand the program to two weeks with the last day serving as
Orientation Weekend
– Gain campus-wide respect for SSS and SOAR students
• Assessment Goals
– Administer Exit interviews once after SOAR and right before
Graduation
– Compile graduation data on class of 2002-2003
Noel-Levitz
Student Satisfaction Inventory
Angela Naginey
Director of Residence Life
Robby Larson
Director of Student Programs
Survey Goals
To administer the inventory in order to
determine opportunities to enhance
overall traditional undergraduate student
satisfaction relating to Student Life areas.
Instrument Design
• Established National Instrument
• Measuring Level of Importance
• Measuring Level of Satisfaction
• Majority using 1-7 Scale
• Cal Lutheran supplemental questions
• Categories for Analysis
– Recruitment & Financial Aid
– Academic Advising
– Campus Climate
– Campus Life
– Campus Support Services
– Concern for the Individual
– Instructional Effectiveness
– Registration Effectiveness
– Safety & Security
– Service Excellence
– Student Centeredness
Data Collection
• Administered in Fall 2002 & Spring 2005
• Distributed to Sophomores and Juniors
– E-mail/Web version for Commuter Students
– Paper version for Residential Students
• 49% overall response rate in both years
– 12% Commuter Response Rate (2002); 15% (2005)
– 74% Residential Response Rate (2002); 73% (2005)
Data Analysis (2002)
• Identified Performance Gaps
– Category Analysis
– Question-specific Analysis
• Student-led Focus Groups
– Qualitative discussions regarding reasons for, and potential
solutions to, performance gaps
• Student Affairs Divisional Workshop
– Discussed potential causes for performance gaps
– Brainstormed actions for improvement
• Informed Student Life Strategic Goal Development
Implications & Implementation
• Qualitative and quantitative data via student voice
• Springboard for communication
• Focused staff efforts on behalf of students
• Informed Student Life Strategic Goal Development
– Enhancing leadership opportunities for males
– Better communicating student fee allocations
• Developed 3-year cycle for administration of NoelLevitz Student Satisfaction Inventory
2006
Assessment Symposium
LEARNING TOGETHER
May 22, 2006
2006 Assessment Symposium Presentations
Written by:
Cathy Alexander, Maria Kohnke, Damien Pena,
Michele LeBlanc, Cody Hartley, Sharon Docter,
Henri Mondschein, Chang-Shyh Peng, Julius Bianchi,
Melanie Clarey, Damien Pena, Tuula Mattson,
Angela Naginey, Robby Larson
Produced by:
Amanda McClendon
May 22, 2006