Assessment Activity Report
Spring 2005
Program Review
The first activity of the semester was to revisit the original cycling sheet and determine where
each program was in the review process (see Table 1). Meetings were held with two departments
and eight department/program chairs to assist them in meeting review expectations.
Concurrently, I have met with Steve Hora to discuss the process and expected outcomes of
program reviews as well as plan to complete the pending reviews.
The program review self-study consists of an executive summary, background information
(Mission, Curricular Goals/Program Student Learning Outcomes and Role in UH system and in
UH Hilo mission and strategic plans), program organization and performance, student learning
outcome assessment, current resources, program chair’s evaluation and a statement of future
goals. Based on the first completed program reviews, the Assessment Support Committee revised
the program review guidelines in an attempt to reduce the amount of qualitative narrative in the
description of program organization and performance and increase the amount of quantitative
data and analysis (see tables attached). However, the Congress did not mandate the changes and
instead added the following preamble:
The following is a suggested format for use in developing the self study report
during a program’s seven-year review. The intention of the format is to permit the
presentation of objective data in as clear and concise a form as possible in order to
enhance the readability of the report. Use of the proposed format should be
considered voluntary by the program undergoing review except for the BOR
required table. The Institutional Researcher will provide the data for the required
table. Any part, or all, of the proposed format may be incorporated into the
program’s review.
Beginning in February, Lynne Stamoulis, Randy Hirakawa and I met bi-weekly to analyze the
current guidelines and explore alternatives that might better meet faculty and administrative
needs. The emphasis was on simplifying the process while still providing data and analysis on
which to base program changes, increase or decrease the number of faculty positions and allocate
resources. We examined models from other universities that included indicators of efficiency:
the degree to which a program uses its current resources and manages enrollment to provide both
program specific and general education courses. In addition, we are concerned about the quality
of each academic program as demonstrated by faculty productivity and evidence that students
learning outcomes are being achieved.
Efficiency and Enrollment Management
Currently, the efficiency and enrollment data for programs are combined in the BOR required
data table with the addition of a graduation index (used by the CAS Dean and VCAA for
determining resource allocations). The table includes:
1
Required Data
Number of Majors
Number of degrees earned in major (or # of graduates)
Graduation index
SSH Taught (Student Semester Hrs) (Fall)
FTE Enrollment
Crossover data % own majors
% within college
% all others
Number of classes or sections offered
Student Registrations
Avg. class size
FTE Faculty (Fall)
Fall
Annual
(4*grads / majors)
LD
UD
Grad
Undergrad
(SSH/15)
Grad
(SSH/12)
Fall
Fall
Fall
I-2
I-3
I-4
I-5
FTE Student FTE Faculty Ratio
Majors per FTE Faculty
Budget Allocation
Cost per Credit Hour
Sources of Data for Each Indicator on the Required Data Table (Excluding the Graduation Index)
Indicator:
Data Source:
Management & Planning Support (MAPS) Reports (Institutional
Research Office) http://www.hawaii.edu/iro/maps.htm
Headcount Majors
Course FTE Enrollment
Fall Enrollment Report,
UH Hilo, Fall Semesters
Course Registration Report,
UH Hilo, Fall Semesters
Data given only for
programs under review.
Data not available for
specific degree program(s)
under review, so are given
at the department level.
2
Number of Classes
Course Registration Report,
UH Hilo, Fall Semesters
Data not available for
specific degree program(s)
under review, so are given
at the department level.
Average Class Size
Course Registration Report,
UH Hilo, Fall Semesters
Data not available for
specific degree program(s)
under review, so are given
at the department level.
Data given only for
programs under review.
Degrees Awarded
Degrees & Certificates Earned,
UH Hilo, Fiscal Years
SSH Taught
Course Registration Report,
UH Hilo, Fall Semesters
Data not available for
specific degree program(s)
under review, so are given
at the department level.
% SSH Taken by own
Majors
Academic Crossover Study, UH
Hilo, Computer Report 1419
Data given only for
programs under review.
% SSH Taken by Nonmajors
Academic Crossover Study, UH
Hilo, Computer Report 1419
Data given only for
programs under review.
Analytical FTE Faculty
Departmental Activity &
Instructional Workload Measures,
Data not available for
specific degree program(s)
under review, so are given
at the department level.
UH Hilo, Fall Sem.
Student-Faculty Ratio
Departmental Activity &
Instructional Workload Measures,
UH Hilo, Fall Sem.
Data not available for
specific degree program(s)
under review, so are given
at the department level.
Note that beginning FY
Instructional Unit Cost Study, UH 2000-01, data will no longer
Direct Instructional Cost Hilo,
be available
(Information Technology Services),
Fiscal Years
http://drue.its.hawaii.edu/uhexpend/
Institutional Research Office, University of Hawai‘i, March 2004.
3
There are a number of problems associated with the current table. First, the data are for Fall
semester only which may provide an inadequate and/or incomplete picture of program
efficiency. In addition, the systemwide Institutional Research Office Management & Planning
Support (MAPS) Reports no longer collect and make available all of the required data. While
this data provides an overview of performance in the fall semester, it doesn’t tell enough about
program organization.
Program Quality and Student Learning Outcome Assessment
Universities are being asked to be accountable for student learning. Instead of measuring inputs
like the number of journal subscriptions in the library or the number of faculty members with
terminal degrees, programs are being asked to demonstrate that students have gained skills and
knowledge as a result of program completion. During the first round of program reviews under
the new guidelines, each program was expected to present an assessment plan. These plans
would be implemented and provide student learning outcome data for the next seven year
review. For example, the English Program Self-Study stated:
Preparing to Assess the Major.
English 300, the introduction to the major,
was offered for the first time in Fall 2003, specifically for literature track majors.
In Spring 2004, English 303 Backgrounds to English Studies will be offered. The
two courses will be offered annually. In the second offering of Eng 300 in Fall
2004, students will develop learning portfolios to be reviewed by the faculty for
the purposes of program assessment.
However, programs have been slow to respond. One possible solution would be to ask programs
to either develop and implement an assessment plan or adopt a nationally normed test such as
Excelsior College Examinations (ECE) and the College Level Exam Program (CLEP). For
example, according to the Chemistry self-study:
Standardized ACS exams are utilized by the Department in all areas of the
program. These exams are made available through the ACS with national
norms. Our students have consistently been within these norms. The
General Chemistry norms are in the 60 percentile nationally and our
students are in that range for the mean of the exam. The norms for
Organic Chemistry are in the 57 percentile and our students have scored in
that range.
If other programs adopted such standardized tests, they would report the scores, analyze student
performance and make changes in the program based on the data.
In addition to student performance, an assessment of program quality should include
stakeholders such as program graduates, employers, and successful entrance into graduate
schools. The Office of Institutional Research conducts and reports graduating student, alumni
and community surveys. These can be customized to provide feedback to programs for the
4
purposes of program review. Dean Hirokawa has encouraged Division and Department chairs to
take advantage of this opportunity to use data that is already being collected and analyzed and to
participate more actively in formulating questions central to program improvement. Nationally
normed data is available through the National Survey of Student Engagement that is conducted
and reported each year. Programs are encouraged to compare their program to peer and
benchmark institutions accessible through the Office of Institutional Research.
Tasks Remaining
Collaboration with administration should continue. The focus should remain on developing the
least labor intensive and most simple system that will provide the necessary data for program
improvement and university decision-making. A faculty member needs to monitor, guide and
provide assistance to programs that are preparing self-studies. In addition, professional
development opportunities need to be provided for new faculty members and those responsible
for program review. These should include workshops and individualized instruction.
5
Proposed Tables in the Voluntary Guidelines
Efficiency and Enrollment Management Data Tables
List Degrees (include tracks, options, and areas of specialization), certificates, minors, etc.
offered by your program on the table below. (Please asterisk core courses that every major must
take in order to meet major requirements.)
Specific Courses
Required in the Major
Tracks, Options,
Areas of
Specialization
Certificate Courses
Minor Course
Requirements
List GE courses provided by your program. (See
http://www.uhh.hawaii.edu/uhhl genedfac/courseproposalreview. php for information about the
UH Hilo GE program.)
Course
Hallmark
Frequency of
offering (Sem., Yr.,
Infreq.)
Enrollment
List service courses your program provides for other programs. For example, the English
program provides Eng 225 Writing for Science and Technology for science majors. Include
cross-listed courses where your program provides the course for the cross-listed program.
Service Course
Program
Frequency
Average
Enrollment
List student-faculty collaboration with dates during the past 7 years in the following
categories:
a. Directed Studies
b. Student-faculty research projects c. Student theses/ senior projects
c. Student theses/ senior projects
Show by flowchart or in lists with narrative the sequence of courses that would enable a
freshman to graduate with your major in four years. See http://cs.uhh.hawaii.edu for samples.
Year 1
Fall
Spring
Year 2
Fall
Spring
Year 3
Fall
Spring
Year 4
Fall
Spring
6
Please complete the following faculty productivity table. For a seven years since your last
program review, include the number of refereed publications (RP), book chapters (Ch), books
(B), other publications (OP), grants received (G - number/total dollar amount) for the past seven
years. Please explain the nature of "Other" creative/scholarly activities.
Faculty
Rank
#
SSH Fall
Courses
Taught
Creative/Scholarly Activities
RP
Ch
B
OP
G (#/$)
Other
-5
-4
-3
Please list service to the department, college/university and the community provided by faculty
at each rank below:
Faculty
Rank
1-5
1-4
1-3
#
Department
College & University
Big Island Community
Program Quality and Student Learning Outcome Assessment
Complete the following Opportunity to Learn Matrix. Indicate where each student learning
outcome is introduced (I), practiced (P), and demonstrated (D). Click here to view a sample.
Regularly Offered
Program Courses
Program Student Learning Outcomes
2
3
4
5
1
6
Sample from the Mathematics Program Self-Study
205/6
231/2
300/1
303/4
310
311
317
421/2
431/2
454/5
Understanding different areas of
math and how they interrelate
Classical theorem-proving skills
X
X
x
x
x
X
x
X
X
X
Refined understanding of the
problem solving process
Ability to independently develop
and deliver all pre-college math
Working knowledge of
technology appropriate to field
Skills:
a. Read, write, translate and
articulate mathematics
b. Solve problems algebraically,
numerically, and graphically
c. Make Inferences &
generalizations
x
x
x
x
x
x
X
X
X
X
X
x
x
x
X
X
X
X
X
X
X
X
X
x
X
X
x
X
X
x
X
X
X
X
X
x
x
X
X
X
X
x
x
x
x
7
In each major course, how are students assessed? From these assessments, which ones are used
to provide data for program improvement? Place an asterisk next to each measure that is used for
program improvement. Please choose at least one assessment that program faculty assess to get
feedback about program effectiveness. Click here for an example.
Course Level Assessment of SLO
Measures (Please add or
delete rows as
appropriate)
Objective Tests
Essav Exams
Research Papers
(Library)
Research Papers
(Original data-based)
Pre/Post Test
Comparisons
Group Projects
Portfolios
Simulations
Presentations
Service Learning
Overall Pass Rates
How effectively does this program meet the stakeholder expectations including alumni,
employers and other community members? Please complete the table below adding or deleting
rows as appropriate.
Targeted Outcome
Measurement (data
collection process)
Results
How results used to
improve the program
8
Program Review Schedule 2002 – 2009
(Revised 2005)
2002/2003
Chemistry
Computer Science
Biology
Geology
Mathematics
CAFNRM
Occupational Safety & Health (minor)
Women's Studies
Externally Reviewed Programs
Business Administration
Education
Nursing
2003/2004
Hawaiian Studies
English
Philosophy/Religious Studies
Communication
Linguistics
2004/2005
Anthropology
Art
Japanese Studies
2005/2006
Economics
Physics/Astronomy
History
Political Science
2006/2007
Geography
Administration of Justice
Liberal Studies
Marine Science
Psychology
Sociology
2007/2008
Performing Arts
(Other Programs & Certificates)
Environmental Studies
Honors
International Studies
Natural Sciences
Pacific Islands Studies
9