April 8, 2010: Drafted by Perry Deess, Fadi Deek, and Norbert Elliot
May 11, 2010: Presented to President Altenkirch, Provost Gatley; Committee for
Department and Program Assessment (Steve Tricamo, Chair as Associate Provost, Joel
Bloom, Sunil Saigal, Urs Gauchat, Fadi Deek, Narain Gehani, Bob English, Rich
Sweeney, Jack Gentul); and Sub-Committee on Assessment (Perry Deess, Co-chair as
IRP Director, John Bechtold [Member as Master Teacher], James Geller, Norbert Elliot
[Member as MSCHE Chair], John Cays, Shanti Gopalakrishnan, Bruce Bukiet, Ala
Saadeghvaziri, Barry Cohen)
Updated: September, 2012: Perry Deess, Judy Redling
Updated Guidelines reflect Committee structure changes, edits to Table of Contents
(including Part III-A-iv, details on external assessment plan), and additional
requirements for the Appendix: syllabi for core courses (with states learning outcomes)
and NJIT Quality in E-learning rubric for all online courses.
NJIT Program Review Process: Guidelines
Background
The enhanced program review process is responsive to increased demands for
accountability. A synthesis of program review guidelines from accreditation agencies
such as the Accreditation Board for Engineering and Technology, the Association to
Advance Collegiate Schools of Business, the National Architectural Accrediting Board,
and the Middle States Commission on Higher Education—as well as university strategic
planning initiatives—the program review process allows both transparency and
collaboration: In reviewing all NJIT program, the process allows NJIT curriculum
directors to learn from each other and to create innovative strategies for achieving
enhanced program outcomes.
Introduction
Beyond a mere audit, the Program Review Process has four innovative objectives: 1) To
provide a forum for the assessment and improvement of all degree and university
programs; 2) To demonstrate continuous improvement in the delivery of educational
curricula; 3) To promote a culture of assessment by building a cohesive assessment
strategy; 4) To create a central, web-based repository for assessment design and
supporting documents. These four objectives will yield a university-wide forum for the
following: 1) collaboration on establishing program vision, strategies, tactics, and
metrics; 2) investigation of new ways of understanding student achievement; 3)
collaboration on common ways of reporting student outcomes; and 4) creation of a webbased system of promoting program assessment models.
While all degree-granting programs will participate in program review to advance the key
objectives, programs accredited through the Accreditation Board for Engineering and
Technology, the Association to Advance Collegiate Schools of Business, and the
National Architectural Accrediting Board will not be required to complete a separate
program review report. Program directors, working with a review team within their
academic units, will be asked to submit their existing accreditation reports with key
sections highlighted that address the NJIT program review guidelines. Program directors
will also submit a brief cover memo, of no more than 2 pages, identifying the key
sections and describing changes made in response to the accreditation self-study. The
reason for requesting review of accredited programs is to allow a sense of universitywide collaboration, to study new ways of coming to terms with the complexities of
student achievement, to encourage collaboration on strategies of reporting student
outcomes, and to create a new model by which we may promote our innovative
assessment models.
Vision
The NJIT Program Review Process will answer the following questions:







How are institutional and program level goals integrated and articulated?
How have planned assessment processes been implemented?
How do assessment results provide convincing evidence of student
learning?
How do program administrators use assessment in decision making?
What changes have been made based on assessment?
How have program assessment results been communicated to
shareholders?
What is the cohesive framework of program assessment at NJIT?
Governance and the Program Review Process
The NJIT Program Review Process is governed by a standing university committee, The
Committee on Assessment of Student Learning, reports to the Deans’ Council (the
Provost and deans from Newark College of Engineering, the College of Architecture and
Design, the School of Management, the College of Computing Sciences, the College of
Science and Liberal Arts, and the Albert Dorman Honors College. Committee members
include the Associate Provost for Academic Affairs, The Director of Institutional
Research and Planning, Associate Provost for Graduate Studies, Associate Deans,
identified faculty, and identified administrators (UCRC Chair, TLT Chair, Graduate
Council Chair).
The process begins with the selection of the university’s degree programs to move
through review on a five-year rotating basis. That is, the process calls for approximately
2
22 programs to be reviewed each year. Once these programs are identified by the
Committee on Assessment of Student Learning within the five year cycle, selected
programs will work collaboratively with the assessment team in Institutional Research
and Planning to add available measures to the existing review strategy.
Program directors will then develop an analysis focusing on student learning outcomes.
Directors will include a detailed description of how data have been used to make
improvements in the past and how they will use the self-study process to drive future
improvements. Once the report is ready, the Committee on Assessment of Student
Learning will undertake the review on behalf of the Dean’s Council.
After the program review is completed, the Committee makes one of three levels of
recommendations: an exemplary recommendation; a qualified program recommendation;
or review for monitoring. If a program receives an exemplary recommendation, a
concerted effort will be made to strengthen other degree programs by advocating process
similar to those practiced by the exemplar program. If a program receives a qualified
program recommendation, a concerted effort will be made to identify the standards of
achievement of the qualified program so that all programs may achieve similar standards;
if a program needs further monitoring, the committee will document the case, work with
the program director to improve program effectiveness, and report that process to the
Committee for Assessment of Student Learning. The Committee may then refer the
program to the provost for further action.
Timeline
The timeline consists of five phases that begin with the agenda established by the
Committee on Assessment of Student Learning.
1. Initiate Process
Operating within the five year cycle, the annual process begins with a review of the
yearly roster of the programs to be reviewed in the five year process. Concurrently, the
NJIT Office of Institutional Research and Planning plans to provide technical assistance
to directors of the programs to be reviewed. The fall process also includes the charge to
the Committee.
2. Orientation of the Program Directors
Members of the Committee orient the identified program director to the review process.
The orientation consists of a review of guidelines, a review of procedures, and an
overview of resources available to assist the director.
3. Assemble Data
The Office of Institutional Research and Planning provides the program director with the
necessary data to needed to complete or enhance the program review document.
3
4. Submit the Report
With the support of Institutional Research and Planning, the program director submits the
report to the Committee.
5. Committee Review
The Committee reviews the report and makes recommendation with a 1 page summary of
program strengths and weaknesses to the Dean’s Council.
6. The Deans’ Council Makes Final Recommendations
The Deans’ Council makes final recommendations and issues a brief report citing
program strengths and weaknesses to the provost.
The timeline is described in the figure below:
Program Review Process: Timeline of Annual Activities
Committee on Assessment of Student Learning initiates process (Co-chairs)
Orientation of the program director to the NJIT Program Review Process (Co-chairs)
Data Assembly (Office of Institutional Research and Planning)
Program Review Report submitted
Committee on Assessment of Student Learning reviews Program Review Report
Deans review Committee report and make final recommendations to the Provost
4
Charge to Program Directors
In preparing the self-study, the program and the academic unit’s program review team
director should keep two goals of this process in mind:
1. To ensure that NJIT is offering quality academic programs (e.g. has
appropriate educational objectives, and is accomplishing those objectives as
evidenced by student learning outcomes assessment)
2. To demonstrate continuous improvement in the delivery of the educational
curriculum.
In preparation for the review, the program director should review the following:
•
•
•
•
•
•
•
•
•
•
•
Program oversight and management
Program objectives
Core course objectives
Relationship between curriculum and objectives
Resources—human, physical, financial—required to support the curriculum
Evaluation student learning outcomes according to a previously established
assessment plan
• Measurement of student performance in courses
• Measurement of student performance across courses
• General measurement of student learning outcomes
• Measurement of student attitudes
Obstacles to meeting objectives
Continuous curricular improvement
Signature features, such as undergraduate research experience
Submission of the Institute Information Literacy Report
If eLearning is involved, review the Institute eLearning Quality Assurance Report
Charge to Institutional Research and Planning
The Office of Institutional Research and Planning will provide each director with the
following information:






Graduation rate
Retention rate
Time to degree
Five year enrollment
Most recent diversity measures
Average GPA by student level
Collaboration on the NJIT Program Reporting Process
Program directors, members of the academic unit review team, and representatives from
Institutional Research and Planning will work together to assemble the following
evidence related to program review.
5




Describe relationship of program outcomes and educational objectives;
Describe the processes used to assure that graduates have achieved the
program outcomes: identify direct and indirect measures of the learning
outcomes.
Provide qualitative and quantitative data to demonstrate that graduates satisfy
planned program outcomes;
Describe how the assessment results are implemented to improve the program.
The NJIT Program Report Document
The NJIT program report provides an overview of the program in general and it’s the
assessment of student learning outcomes in particular. It directly addresses the twin
objectives of assessment and continuous improvement by demonstrating how the
program meets its own educational objectives and how it has made changes to improve
the program. The program review report should employ Middle States terminology,
unless the program has external accreditation. For additional assistance see the Middle
States guide to assessing student learning:
http://www.msche.org/publications/Assessment_Expectations051222081842.pdf
For a more comprehensive discussion see:
http://www.msche.org/publications/SLA_Book_0808080728085320.pdf
Programs participating in an external accreditation process such as ABET, NAAB, or
AACSB will be asked to submit existing reports with a detailed cover letter and relevant
sections marked.
An additional program review report will be scheduled for the assessment of student
learning outcomes in the GUR. This report will follow a similar model, but with a
different structure because it involves the assessment of learning across the curriculum.
Definitions:
Institutional Learning Goals:
As students seek disciplinary mastery, they will achieve skills and knowledge
in:
 Research-based Inquiry: Students employ investigative methods
 Ethical Conduct: Students understand professional and civic
responsibility
 Economic Opportunity: Students understand economic reasoning and
demonstrate that they are able to allocate resources effectively and
logically under operating constraints
 Collaboration: Students work effectively in teams to engage
multidisciplinary perspectives
 Engagement: Students are active and committed learners
6
Program Learning Goals:
Statements that describe what students are expected to know and be able
to do by the time of graduation. These relate to skills, knowledge, and behaviors
that students acquire as they progress through the program.
Course Learning Goals:
Formally defined expectations for what students should know and be able
to do after they complete a course. These relate to skills, knowledge, and
behaviors that students acquire as they progress through the course.
Learning Outcomes:
The knowledge, skills, and abilities that students have attained as a result
of their involvement in program based educational experiences.
Assessment of student learning:
Assessing student achievement employs evidence from student courses,
projects, activities, or performances to determine the extent to which student
learning achieves the stated program goals
http://www.msche.org/publications/examples-of-evidence-of-student-learning.pdf
Sample Table of Contents
I. General program information (2 page)
A. Program background information:
B. Specify program title(s): all degrees awarded for the program under review,
including options, as specified in transcripts;
C. Indicate program mode: (e.g., face-to-face or distance learning; percentage of
students enrolled part-time).
D. Program details: discuss metrics and implications for program, including how
data have been used to make improvements in the past and how they will use the
self-study process to drive future improvements.
1.
2.
3.
4.
5.
6.
Graduation rate
Retention rate
Time to degree
Five year enrollment
Most recent diversity measures
Average GPA by student level
II Program mission and goals (1 page)
A. Describe program mission and program learning goals
B. Provide a matrix showing how the institutional learning goals are mapped to the
Program Learning Goals (See Table 1)
7
Table 1. Matrix of Institutional Learning Goals and Program Goals
Institutional Institutional Institutional Institutional Institutional
Learning Learning Learning Learning
Learning
Goal 1:
Goal 2:
Goal 3:
Goal 4:
Goal 5:
Researched- Ethical
Economic Collaboration Engagement
based
Conduct
Opportunity
Inquiry
Program
Learning
Goal:1
Program
Learning
Goal: 2
Program
Learning
Goal:
ETC.
III Learning outcomes (4 pages)
A. Discuss how the student achievement of program learning goals is
assessed
i. Describe the program assessment plan:
How are student learning outcomes assessed over the course of
the program? Indicate measures of student learning and
criterion-based standards (rubrics) against which learning is
evaluated.
ii. If the assessment plan includes assessment at the course
level, how are course learning outcomes related to the
achievement of program learning goals.(see Table 2). Syllabi
with learning outcomes for all core courses should be provided
in the appendix.
iii. If the assessment employs portfolio analysis or the application
of rubrics to specific student work describe the process,
including how standards are defined.
iv. Provide details on external assessment plan for comparative
quality assurance: discipline-specific exam, external review of
final project, portfolio, or exam.
v. For programs where there are no well defined core courses,
assessment should be based on a senior project, thesis,
qualifying exam, publications or dissertation.
8
vi. For undergraduate programs, identify core courses with an
information literacy component. Describe the information
literacy assessment process and results.
vii. Undergraduate programs should address the contribution of the
GUR to learning outcomes in the program. (For GUR program
review guidelines see Appendix 1)
viii. If students can enroll in a online courses, include in the
appendix a completed NJIT Quality in eLearning Rubric for
each course:
www.njit.edu/tlt/quality-assurance/quality-assurance.pdf
Table 2. Matrix of Program Learning Goals to Core Course Goals
Program Program Program
Learning Learning Learning
Goal:1 Goal:3 Goal:
ETC.
Core
Course 1
Goals
Core
Course 2
Goals
Core
Course 3
Goals,
etc.
IV Using student learning assessment (1 page)
A. Discuss the results of the assessment and to what extent these results
provide evidence that program learning goals have been achieved.
B. Describe what specific program strengths and weaknesses are revealed
by the assessment of student learning outcomes.
C. Show how the assessment results are being used to implement
changes. (What are those changes?)
V. Assessment in the next cycle (1 page)
A. Discuss the sustainability of assessment and how it is integrated into
the program.
B. Describe planned improvements in the assessment process.
VI Conclusion (1 page)
A. Summary
9
B. Future improvements
Appendices—Supporting Documents (possible examples shown below)
A)
B)
C)
D)
E)
F)
Syllabi with learning outcomes for all core courses
Program assessment supporting data including rubrics
Summary of test results
Summary of ePortfolio evaluation
NJIT Quality in E-learning rubrics and evaluation report
Presentations and publications associated with student learning assessment
Implementation
The guidelines described in this document are effective for program review reports
written after Spring 2012 and will remain in effect until superceded by a new set of
guidelines.
10
APPENDIX I
ASSESSMENT OF THE GENERAL UNIVERSITY REQUIREMENTS: A fundamental
guiding principle in the development of the General University Requirements (GUR) is
the formulation of a foundational curriculum encompassing the necessary preconditions
for success in undergraduate disciplines, a curriculum that establishes a devotion to
lifetime intellectual discovery and personal development. Learning Outcomes relevant to
this guiding principle are articulated in the NJIT Core Competencies. Assessment of the
Core Competencies is achieved through the use of locally-developed methods and
nationally-based tests.
The NJIT Program Review Process for GUR will answer the following questions:







What learning outcomes are specific to a thematic area within the GUR?
Which Core Competencies are addressed? How is a given Core Competency
integrated and articulated in the GUR thematic area?
How have planned assessment processes been implemented?
How do assessment results provide convincing evidence of student learning?
How do program administrators use assessment results in decision making?
How have actions been taken as a result of assessment?
How have assessment results been communicated to a variety of shareholders?
Review Process
The GUR program review report will be prepared by the Program Review Committee as
a whole, based on separate contributions from each of the disciplines offering GUR
courses. The review will follow the overall structure discussed in the Program Review
Guide, but with separate sections for each discipline and will cover all standard GUR
courses in the GUR disciplines. The process of aligning and assessing goals from the
institutional goal to the program goal, to the discipline goal, to the course level will be
followed with an additional focus on the extent to which core competency benchmarks
have been achieved.
The assessment of the GUR differs in several ways from the other program reviews.
First, it must add a discipline level for learning goals because the GUR includes one
step between the program goal and the course goal and this is the goal of the specific
GUR discipline. Second, the emphasis for the review will be on the core
competencies. This is not to suggest that other goals of the GUR including general
education, engaged citizenship, and a love of life-long learning are of lesser
importance, but they are notoriously difficult to assess. The extent to which they are
addressed by individual courses will be considered but beyond this, assessment of
these goals will belimited.
11
Table 1. The NJIT General University Requirements
GUR
DISCIPLINE
DESCRIPTION
NUMBER OF
CREDITS
Computing
Sciences
An understanding of the nature of computing, its impact on society, and
the driving forces behind its pervasive deployment is integral to
effective functioning as a professional and as a citizen.
3 credits
Cultural History
All educated individuals are expected to understand and appreciate
history and the world’s cultures.
3 credits
English
The ability to communicate ideas is an essential characteristic of
educated individuals. All students are expected to achieve proficiency
in both oral and written English.
6 credits
Humanities
The ideals of a liberal education transcend particular major fields and
career goals. All students are expected to develop an interest in specific
areas within the humanities.
Management
All students are expected to develop the management skills needed to
function effectively in an organizational setting.
3 credits
Mathematics
The ability to reason qualitatively and quantitatively, to understand
probability, and to apply mathematical models to a variety of
circumstances is fundamental to making informed decisions in the
modern world.
6 credits, including at
least 1 credit of
probability and statistics
Natural
Sciences
The natural sciences provide the basis for our knowledge of the
physical universe and for technological progress. All students are
expected to develop a thorough understanding of at least one laboratory
science.
7 credits, including at
least 1 credit of lab
Social Sciences
An understanding of the social sciences is essential in order to
understand the economic, social, and political forces at work in our
world.
6 credits
6 credits of junior-level
electives;
3 credits of senior
seminar
Core Competencies:
As all students, regardless of discipline, experience General University Requirements,
we establish benchmarks and measure these three Core Competencies to support the
improvement of student learning:
A. Writing, Reading, and Critical Thinking
B. Quantitative Reasoning
C. Information Literacy
12
Table of Contents
I. General program information (2 pages)
 Brief description of GUR and its relationship to the institutional learning goals
II Program mission and goals (3 pages)
C. Describe GUR mission program learning goals, and discipline level learning goals
D. Provide a matrix showing how the institutional learning goals are mapped to the
Program Learning Goals (See Table 1) and how the Program Goals are mapped to
discipline goals (Table 2)
Table 1. Matrix of Institutional Learning Goals and Program Goals
Institutional Institutional Institutional Institutional Institutional
Learning Learning Learning Learning
Learning
Goal 1:
Goal 2:
Goal 3:
Goal 4:
Goal 5:
Researched- Ethical
Economic Collaboration Engagement
based
Conduct
Opportunity
Inquiry
Program
Learning
Goal:1
Program
Learning
Goal: 2
Program
Learning
Goal:
ETC.
13
Table 2. Matrix of Program Goals and Discipline Goals
Program
Program Program
Program
Program
Goal 1:
Goal 2: Goal 3:
Goal 4:
Goal 5:
Researched- Ethical Economic Collaboration Engagement
based
Conduct Opportunity
Inquiry
Discipline
Goal:1
Discipline
Goal: 2
Discipline
Goal:
ETC.
III Learning outcomes (10 pages)
B. Discuss how the student achievement of program learning goals is
assessed in each discipline
ix. Describe the program assessment plan:
How are student learning outcomes assessed over the course of
the program? Indicate measures of student learning and
criterion-based standards (rubrics) against which learning is
evaluated.
x. Most assessment will be conducted through the analysis of
results from of nationally normed tests. If the assessment plan
in a discipline includes additional assessment at the course
level, how are course learning outcomes related to the
achievement of program learning goals.(see Table 3).
xi. If theadditional assessment employs portfolio analysis or the
application of rubrics to specific student work describe the
process, including how standards are defined.
xii. Identify core courses with an information literacy component.
Describe the information literacy assessment process and
14
results. (Most of this will be conducted through the iSkills
exam, but additional evidence may also be provided.)
xiii. If students can enroll in an online course, include a completed
NJIT Quality Matters Rubric
www.njit.edu/tlt/quality-assurance/quality-assurance.pdf
Table 3. Matrix of Discipline Goals to Course Goals
Discipline Discipline Discipline
Goal:1
Goal:3
Goal:
ETC.
Course
Goal 1
Course
Goal 2
Course
Goal 3
etc.
E. Provide a matrix showing mapping of Core Competencies to GUR courses (Table
5)
Table 5. Matrix of Core Competencies and Courses
CC1: Writing,
Reading, Critical
Thinking
CC2:
Quantitative
Reasoning
CC 3:
Information
Literacy
(Other)
Learning
Outcome 1
(Other)
Learning
Outcome 2
Course1
Course 2
Course 3ETC.
F. For the overall GUR program the program review committee will establish core
competency benchmarks that indicate a target achievement level. For example, at
least 70% of students must achieve Core Competency #1.
IV Using student learning outcomes assessment overall
15
D. Discuss the results of the assessment and to what extent these results provide
evidence core competency benchmarks and program goals have been achieved
E. Describe what strengths and weaknesses are revealed
F. Show how the assessment results are being used to implement changes. (What are
those changes?)
IV Using student learning assessment, for each separate discipline (4 page)
A. Discuss the results of the assessment and to what extent these results
provide evidence that program and discipline learning goals have been
achieved.
B. Describe what specific program strengths and weaknesses are revealed
by the assessment of student learning outcomes.
C. Show how the assessment results are being used to implement
changes. (What are those changes?)
V. Assessment in the next cycle (2 page)
A. Discuss the sustainability of assessment and how it is integrated into
each discipline
B. Describe planned improvements in the assessment process.
VI Conclusion (1 page)
C. Summary
D. Future improvements
16