Program Evaluator (PEV)
Face-to-Face Training
Copyright © 2014 by ABET
Welcome!
“Our duty is to believe that for which we have
sufficient evidence, and to suspend our
judgment when we have not.”
~ John Lubbock
Facilitator Introductions
List of Participants – on each table
Introduce yourself – first time you speak
Copyright © 2014 by ABET
ABET Welcome
ABET Headquarters
Representative
Copyright © 2014 by ABET
Why Are We Here?
Course Purpose
• Provide you with the knowledge, skills,
•
•
and abilities to conduct an ABET
accreditation evaluation.
Learn how to provide guidance to
programs for improvement.
Our goal: To improve consistency of
program evaluation.
Copyright © 2014 by ABET
Course Purpose (cont.)
Pre-Work
Process
Outputs from
Pre-Work
Face to Face
Training
Outputs
from Face to Face
• This Face-to-Face Training is an experiential workshop
•
that simulates a real visit as closely as possible.
It works in conjunction with the on-line Pre-Work
modules, designed for your independent and convenient
learning.
Copyright © 2014 by ABET
Why Are We Here?
Learning Outcomes
At completion of the training, you will be able
to:
• Plan an effective, cordial, efficient campus
visit.
• Interact with your team during a visit.
• Given a specific situation or evidence (e.g.,
transcripts, student work):
 Determine which criteria are relevant to determining
the accreditation action.
 Determine if the program is in compliance with a
criterion. If not, determine the level of compliance
and the accreditation action.
Copyright © 2014 by ABET
Learning Outcomes (cont.)
At completion of the training, you will be able to:
• Conduct interviews of faculty, staff, and students.
• Write an exit statement to the institution.
• Conduct an informal debrief with the program
head and exit meeting with leaders of the
institution.
• Complete Program Evaluator forms.
• Demonstrate “in practice” effective use of PEV
competencies in the PEV Competency Model.
Copyright © 2014 by ABET
Agenda - Day One Morning
Module Name
Timing
1 – Welcome and Introduction
8:00 – 8:15
2 – Connection to Pre-Work and Assessment Fundamentals
8:15 – 9:30
Break
9:30 – 9:45
3 – Display Materials and Facilities
9:45 – 11:00
4 – Sunday Team Meeting
11:00 – 12:30
Lunch
Copyright © 2014 by ABET
12:30 – 1:15
Agenda - Day One Afternoon
Module Name
Timing
5 – Interviews
1:15 – 2:05
6 – Monday Night Meeting
2:05 – 3:35
Break
3:35 – 3:50
7 – Writing Exit Statements
3:50 – 4:55
8 - Homework
4:55 – 5:15
Wrap Up Day One
Copyright © 2014 by ABET
5:15 – 5:25
Agenda – Day Two
Module Name
Timing
Day Two Overview
8:00 – 8:10
9 – Context for Judgment
8:10 – 9:00
10 – Informal Program Debrief & Exit Meeting
9:00 – 10:05
Break
10:05 – 10:20
11 – Exit Statements
10:20 – 11:00
12 – Connection to PEV Competencies
11:00 – 11:20
13 – Post-Visit Work
11:20 – 11:29
14 – Post Training
11:29 – 11:38
15 – Closing Exercise
11:38 – 11:45
Copyright © 2014 by ABET
Agreements
•
•
•
•
•
•
•
Speak with intention.
Listen with attention.
Know that the wisdom is in the room.
Ensure representation from all pertinent voices.
Model the PEV competencies in all facets of the
training.
Turn off/silence electronic devices (cell phones,
laptops, etc.).
Start/stop on time.
Copyright © 2014 by ABET
Training Materials
• Notebook
 Slide Presentation
• Reference Notebook
 General Accreditation Materials
 Commission Specific Materials
• Exercise Handouts
• Table Supplies
• Issue Bins
Copyright © 2014 by ABET
Module 2: Connection to Pre-Work
Pre-Work Modules
1.
2.
3.
4.
Overview
The Accreditation Process
The Role of the Program Evaluator
Continuous Improvement of Student
Learning
5. Applying the Criteria
6. The PEV Appraisal Process
Copyright © 2014 by ABET
Completed Pre-Work
Pre-Work
Process
Outputs from
Pre-Work
Face to Face
Training
Outputs
` Face to Face
Pre-Requisite Work Modules:
Completed:
• Overview
• Proficiency Assessments (3)
• The Accreditation Process
• Upper State University Self-Study
and Transcript Analysis (PEV Report
and PEV Worksheet as appropriate
for Commission)
• The Role of the Program Evaluator
• CI of Student Learning
• Applying the Criteria
• The PEV Appraisal Process
Copyright © 2014 by ABET
Pre-Work Refresher Cards Exercise
In your table groups:
• Use the refresher cards to test your knowledge of Pre-Work
topics:
 Each take a turn asking, answering, and recording the result.
 If you get an incorrect response, another group member may
‘help’ out.
• Your Support Facilitator has the answers and is available for
•
•
assistance.
Record the answers your team gets correctly the first time,
second time, and third time AND the number of cards you
have gone through.
YOU HAVE 50 MINUTES.
Handout: Refresher Cards
Copyright © 2014 by ABET
Upper State University
Pre-Work Analysis
• Pre-work analysis provides starting point for evaluation
•
of Upper State University.
Face-to-face training simulates a campus visit to Upper
State University.
 “On-site” activities reveal additional insight and information about
the program.
Therefore:
• Keep an open mind.
• Stay in information-gathering mode.
• Do not look through the lens of “reinforcing your predrawn conclusions.”
Copyright © 2014 by ABET
Upper State University
Site Visit
At the end of the training you’ll leave with:
• Table team decision whether the
program meets the criteria.
 Team consensus.
 My interpretation and your interpretation of
like circumstances must be the same.
Copyright © 2014 by ABET
ABET Code of Conduct
(Rules of Procedure Section EIGHT A & B)
•
•
•
•
•
•
•
•
•
•
Accept responsibility for decisions you make ...
Serve only in areas of competence.
Act as agent of ABET and avoid conflicts of interest.
Keep decisions confidential.
Be objective and truthful.
Behave honorably, ethically and lawfully.
Report concerns with respect to internal financial matters.
Treat all persons fairly.
Assist colleagues … in following the Code of Conduct.
Support prompt and fair adjudication of alleged
violations...
Copyright © 2014 by ABET
Conflict of Interest
(Rules of Procedure Section SEVEN A & B and APPM II.C)
• Current or past employment as faculty, staff, or
•
•
•
•
•
consultant by the institution or program;
Current or past discussion or negotiation of employment
with the institution or program;
Attendance as student at the institution;
Receipt of an honorary degree from the institution;
An institution or program where a close family relative
is, or was, a student or employee within the past 10
years; or,
An unpaid official relationship within the past 10 years
with an institution.
Copyright © 2014 by ABET
Dress Code for Visits
• Day 0: Business Casual (Usually a
Sunday afternoon campus visit. Time
used to review course materials,
assessment information, and tour
laboratories.)
Copyright © 2014 by ABET
Dress Code for Visits
• Day 1: Business Professional
•
(Meetings with Faculty, Students,
Support Areas)
Day 2: Business Professional (Exit
Meeting with CEO of Institution)
Copyright © 2014 by ABET
JIT
Evaluation
Copyright © 2014 by ABET
Module 3
Display Materials and
Facilities
Copyright © 2014 by ABET
Interview with
Laboratory Manager
Copyright © 2014 by ABET
What to Look For: Facilities
• Labs/Design Studios
 Sufficient number and size of labs.
 Appropriate coverage across the breadth of
specializations within the program.
 Appropriate equipment, in good repair.
 Appropriate student access (including evening and
weekend access).
 Appropriate technician support and instructional
support in lab.
Copyright © 2014 by ABET
What to Look For: Facilities
• Labs/Design Studios, cont.
 Safe physical arrangement and appropriate
safety practices followed.
 Comments from students about the lab
experience.
 Is there some type of “Laboratory Plan” for
maintaining and upgrading the instructional
laboratories? This is not required by ABET, but,
in general, a formal or informal plan of some sort
should exist.
Copyright © 2014 by ABET
What to Look For: Facilities
• Classrooms
 Appropriate physical arrangement, support for computer
projection, etc.
 Not overcrowded.
• Support facilities
 Sufficient computer access, with appropriate off-hours
access.
 Appropriate spaces for students to gather (not an explicit
criterion but relates to several criteria).
 Appropriate shop with parts, repair facilities, etc.
• Faculty offices
 Sufficient size, privacy.
Copyright © 2014 by ABET
Panel Discussion
Copyright © 2014 by ABET
Display Materials
Copyright © 2014 by ABET
Display Materials
Copyright © 2014 by ABET
Display Materials (APPM II.G.6.b)
• Course materials including course syllabi, textbooks,
example assignments and exams, and examples of
student work, typically ranging from excellent through
poor.
• Evidence program educational objectives are based on
the needs of the stated program constituencies.
• Evidence of the assessment, evaluation, and attainment
of student outcomes.
• Evidence of actions taken to improve the program.
Copyright © 2014 by ABET
Small Group Sessions
with USU Display Materials
• Descriptions of materials for:
 Curriculum/Course Work
 Senior Design Projects, Capstone or
Integrating Experience (if required)
 Demonstration of Student Outcomes
 Assessment Materials
Handouts 3-1 & 3-2
Copyright © 2014 by ABET
What to Look For: Course Materials
• Courses appear appropriate to accomplish the
•
•
•
Program Educational Objectives and Student
Outcomes.
Student work indicates active engagement,
demonstration of learning, and reasonable grading
standards
Evidence with respect to specific Student Outcomes
(including those in the criteria) as appropriate to the
assessment plan.
A few missing items, or even entire missing courses,
are not necessarily systemic problems. Pursue any
major gap to see if it represents a serious problem.
Copyright © 2014 by ABET
What to Look For:
Laboratory Reports
• Evidence (lab reports) of appropriate
•
student learning (not just cookbook).
Evidence to support Student Outcomes.
Copyright © 2014 by ABET
What to Look For:
Major Design or Integrating Experience
• Evidence that all students complete a major
design, capstone or integrating experience (if
criteria require) that draws on previous
courses and incorporates standards and
realistic constraints. Student work should
demonstrate this via a complete project
report or some other mechanism.
Copyright © 2014 by ABET
What to Look For:
Assessment Materials
• Examples of instruments used and
•
•
•
actual data collected.
Summaries of the data with results
reported in a usable form.
Recommendations for program
improvement based on the data.
Implementation and results.
Copyright © 2014 by ABET
38
Documentation (APPM II.A.6)
• Institutional catalogs and similar publications must clearly
indicate the programs accredited by the commissions of
ABET as separate and distinct from any other programs or
kinds of accreditation.
• Each accredited program must be specifically identified as
“accredited by the _________ Accreditation Commission
of ABET, http://www.abet.org.”
• Each ABET-accredited program must publicly state the
program’s educational objectives and student outcomes.
• Each ABET-accredited program must publicly post annual
student enrollment and graduation data per program.
Copyright © 2014 by ABET
JIT
Evaluation
Copyright © 2014 by ABET
Module 4:
Sunday Team Meeting
Copyright © 2014 by ABET
Sunday Team Meeting Agenda
• Agenda Review
• Visit Schedule
• Program Evaluator Pre-Visit Briefings



Program Strengths
Potential Program Shortcomings
Recommended Action
• Display Material and Facility Findings
• Agree on Common Findings and Plans to
•
Investigate
Debrief
Copyright © 2014 by ABET
Handout 4-1
Actual Visit Sunday Meeting
• Usually involves review of multiple
•
•
programs with one PEV per program.
Discussions will focus on common
shortcomings across programs and what
the team needs to do to ensure consistent
evaluation and recommended action.
Usually two meetings:
• Before review of display materials and facility tours.
• After review of display materials and facility tours.
Copyright © 2014 by ABET
JIT
Evaluation
Copyright © 2014 by ABET
Module 5:
Interviews
Copyright © 2014 by ABET
Why Interview?
Direct interface with the members of the
program:
• Facilitates understanding of the program.
• Provides an opportunity to clarify items in
the Self-Study and to probe for further
details.
• Effective way to identify problems not
addressed in the Self-Study.
Copyright © 2014 by ABET
Panel Discussion
Copyright © 2014 by ABET
Faculty Interview
Copyright © 2014 by ABET
Managing Problem Interviews
Exercise
In your table groups:
• Draw card from scenario card deck.
• Share ideas for dealing with the situation:
 What would you say in this situation?
• Facilitator: Capture ideas on sticky-notes.
• Post ideas/responses for each scenario on
appropriate flipchart in room.
Handout: Interview Cards
Copyright © 2014 by ABET
The Art of Asking Questions
• All interview questions should be
DESIGNED to elicit a detailed response.
 Determine and clarify objectives BEFORE
the interview.
 Prepare questions.
Copyright © 2014 by ABET
The Art of Asking Questions
• Begin with “why”, “how”, “what”, “when”,
•
•
and “where”.
Avoid leading questions that assume an
answer.
Prompt for specific information: “Show
me.”
Copyright © 2014 by ABET
The Art of Asking Questions
• Turn the interviewee into a teacher:
•
•
•
“Can you explain that to me?”
Ask an obvious question to establish
interviewee as the expert.
Periodically verify what you heard: “Do I
understand you to say…?”
Silence is golden.
Copyright © 2014 by ABET
Student Interviews
• Suggest Interview a junior/senior level
•
•
class (~20 minutes w/o instructor
present)
Briefly describe ABET and
Comprehensive Review
Possible Questions:




Do the laboratory facilities and equipment work well?
Are the computer laboratories satisfactory & available?
Are enrollment and career advising helpful?
Do the courses from the Chem; Phys; Math support the
engineering courses?
Handout
Copyright © 2014 by ABET
5-1
JIT
Evaluation
Copyright © 2014 by ABET
Module 6:
Monday Night Meeting
Handout 6-1
Copyright © 2014 by ABET
Monday Night Meeting
• Each PEV Candidate in turn brief team on
potential finding including:
 Criterion to be cited
 Evidence gathered
 Compliance level
• Discuss evidence, criteria and compliance as
•
team and develop consensus for the finding.
You have a maximum of 60 minutes.
Copyright © 2014 by ABET
Actual Monday Team Meeting
• Similarities with simulation
• Differences with simulation
Copyright © 2014 by ABET
JIT
Evaluation
Copyright © 2014 by ABET
Module 7
Writing Exit Statements
Copyright © 2014 by ABET
Types of Statements
Exit
statement for
Program A
Exit
statement for
Program B
Exit
statement for
Program C
Team Chair
Exit
Statement
Draft Statement
Final Statement
Handout 7-1
Copyright © 2014 by ABET
Types of Findings
• The statement will generally include one or
more types of findings:





Strength
Deficiency
Weakness
Concern
Observation
• The format will vary depending on type of
visit: General Review or Interim Visit.
Copyright © 2014 by ABET
GENERAL REVIEW
Copyright © 2014 by ABET
Outline of Exit Statement
• Introduction
• Program Strengths
• Program Shortcomings
 Deficiencies (in order of Criteria)
 Weaknesses (in order of Criteria)
 Concerns (in order of Criteria)
• Observations
***Applies to CAC, EAC, & ASAC – ETAC a bit
different
Handout 7-2
Copyright © 2014 by ABET
Introduction Statement
Include information such as:
•
•
•
•
•
•
Type or special characteristics of the program.
Emphasis area(s).
Number of enrolled students.
Size of most recent graduating class.
Number of faculty members.
Other information that could be helpful to the
next team.
Copyright © 2014 by ABET
Introduction Example
• The industrial engineering program prepares
students for careers in system design and
optimization and ergonomics. All courses in the
major are available by distance education, but
students must come to campus or have facilities
available through their employers for laboratory
experiences in two courses. The current student
enrollment is 150 students. There are 14 fulltime faculty members supporting the program.
Copyright © 2014 by ABET
Program Shortcoming Statements
• Contents:
 Criterion/policy citation
 What was observed
 Effect on the program
Copyright © 2014 by ABET
Criteria/Policy Citation
• Criteria/Policy Citation
 State which criterion or policy applies
 Quote excerpts as needed
Copyright © 2014 by ABET
What Was Observed & Effect
• What was observed
 Describe what was observed (evidence).
 Describe how your finding points to lack of
compliance with the criterion or policy or points to
lack of strength of compliance.
• Effect
 Describe the negative effect it has on the program
(D or W) with respect to the criterion.
 Describe the potential future effect on the
program (C) with respect to the criterion.
Copyright © 2014 by ABET
68
Example of a Deficiency (Part 1)
Students Criterion
Criterion 1 requires that students who
graduate from the program meet all
program requirements.
Copyright © 2014 by ABET
69
Example of a Deficiency (Part 2)
While most students who graduated successfully
completed all requirements, there were a few
students who did not pass all of the required
courses designated by the program.
Students can graduate from the program without
passing all the required courses. Thus, the
program is not in compliance with the Student
criterion.
Copyright © 2014 by ABET
Example of a Weakness (Part 1)
Program Educational Objective Criterion
This criterion requires that each program
have a documented process for involving
program constituencies, for the periodic
review of these program educational
objectives.
Copyright © 2014 by ABET
Example of a Weakness (Part 2)
The program has relied on a review of the PEOs every
three years by the faculty only. The other
constituencies (alumni, employers, students) were not
involved in this review.
Without involvement of all constituencies, the program
can not ensure that the PEOs meet the needs of all
constituencies. Thus strength of compliance with this
criterion is lacking.
Copyright © 2014 by ABET
Example of a Concern (Part 1)
Institutional Support Criterion
The criterion requires that resources be
sufficient to attract, retain, and provide for
the continued professional development of
the faculty. Resources must also be
sufficient to acquire, maintain, and operate
infrastructures, facilities and equipment
appropriate for the program.
Copyright © 2014 by ABET
Example of a Concern (Part 2)
At present, it appears that resources are
adequate to support the program. However,
there have recently been large reductions in
the operating budget of the department.
If these budgetary reductions continue, future
compliance with this criterion may be
jeopardized.
Copyright © 2014 by ABET
STATEMENT CRITIQUING
EXERCISE
Handouts 7-3 & 7-4
Copyright © 2014 by ABET
Additional Guidelines for Writing
Statements
• Must be based on conclusions from
•
•
evidence found by the team to be credible
and significant.
Names of individuals or titles that identify
individuals must not be used.
Avoid attributing opinions, conclusions or
recommendations to other individuals or
groups of individuals besides the
Commission.
 Poor: The team discovered that….
 Better: Evidence indicated that…..
Copyright © 2014 by ABET
Additional Guidelines for
Statements
• Provide enough detail that the program and
•
•
institution will know precisely what is
inadequate.
Provide enough detail to the Commission to
justify the type of finding.
Provide enough detail that the team making the
next general or interim review will be able to
determine the amount of progress that has been
made since the original finding.
Copyright © 2014 by ABET
Additional Guidelines for
Statements
• If wording does not add to or clarify the
•
•
finding, then do not include it.
Avoid the use of acronyms or other
abbreviations.
Avoid combining findings on two criteria
into one statement.
Copyright © 2014 by ABET
Program Audit Form (PAF)
• ONLY document left with the institution
•
at the end of the visit.
MUST be consistent with the Exit
Statement.
 Level of compliance.
 Wording to describe what was observed and
the effect on the program.
Handout 7-5
Copyright © 2014 by ABET
Program Audit Form (PAF)
PROGRAM AUDIT SUMMARY
(PROVIDE A COPY TO INSTITUTION AT EXIT MEETING)
Use “C” for concern, “W” for
Shortcomings
weakness, and “D” for deficiency in from Previous
the appropriate line. 1
Review
If the program has no deficiencies
or weaknesses, check this line.
1. STUDENTS
2. PROGRAM EDUCATIONAL
OBJECTIVES
3. STUDENT OUTCOMES
4. CONTINUOUS
IMPROVEMENT
5. CURRICULUM
6. FACULTY
7. FACILITIES
8. INSTITUTIONAL SUPPORT
PROGRAM CRITERIA
ACCREDITATION POLICIES
AND PROCEDURES
Copyright © 2014 by ABET
Exit
Meeting
Seven
Day
Response
Due
Process
Program Audit Form (PAF)
Copyright © 2014 by ABET
Format of Exit Statement
• Introduction
• Program Strengths
• Program Shortcomings
 Deficiencies (in order of Criteria)
 Weaknesses (in order of Criteria)
 Concerns (in order of Criteria)
• Observations
Copyright © 2014 by ABET
Reviewing Your Statement
• Use the statement rubric to carefully
•
•
review your exit statement before
turning it in to the Team Chair.
Ask another PEV to review the
statement for clarity and consistency
with the PAF.
Read the statement out loud to your
fellow team members.
Copyright © 2014 by ABET
STATEMENT WRITING
SUMMARY
• Table Discussion
Rubric for Evaluation of Exit Statement & PAF
#
Needs Improvement
Developing
1
Finding statements do not cite the criterion or policy
that applies to each finding.
Finding statements do not consistently cite the criterion
or policy that applies or quote excerpts as needed.
Finding statements cite the criterion or policy that
applies or quote excerpts as needed.
2
Finding statements do not describe what was
observed for each finding.
Finding statements do not consistently describe what
was observed for each finding.
Finding statements describe what was observed for
each finding.
3
Most finding statements incorrectly interpret the
criteria or policy.
Some finding statements incorrectly interpret the criteria
or policy.
All finding statements correctly interpret the criteria or
policy.
Deficiency or
Weakness
Statement
4
Statements do not describe the impact each finding
has on the program.
Statements do not consistently describe the impact each
finding has on the program.
Statements describe the impact each finding has on
the program.
Concern
5
Statements do not describe the potential future
impact on the program.
Statements do not consistently describe the potential
future impact on the program.
Statements describe the potential future impact of
each finding on the program.
TAC only
6
Statements do not include a directed action for each
finding.
Statements do not consistently include a directed action
for each finding
Statements include a directed action for each finding.
Consistency of
Language
7
Most language used in the statements is not
consistent with the definition of the findings or level
of compliance.
Some language used in the statements is not consistent
with the definitions of the findings or level of
compliance.
All language used in the statements is consistent with
the definitions of the findings or level of compliance.
Consistency with
PAF
8
Most finding statements are inconsistent with the
findings indicated on the Program Audit Form.
Some finding statements are inconsistent with the
findings indicated on the Program Audit Form.
All finding statements are consistent with the findings
indicated on the Program Audit Form.
Deficiency,
Weakness &
Concern Findings
Statements
Meets Expectations
Handout 7-6
Copyright © 2014 by ABET
JIT
Evaluation
Copyright © 2014 by ABET
Module 8
Homework
Copyright © 2014 by ABET
Homework Assignment
• Update your PEV Worksheet to reflect new
information from today’s activities:
 Interviews, display materials, information from
Dean, and team discussions.
• Draft your Exit Statement following the
statement format and writing guidelines.
 Introduction, and the shortcomings based on your
team consensus for Upper State University.
• Complete the Program Audit Form.
• Read the two Judgment Scenarios.
Handouts 8-1 & 7-6
Copyright © 2014 by ABET
Homework Assignment
• You will email the completed Exit Statement
and Program Audit Form to the ABET Staff
person whose e-mail address will be provided.
• The PAF for your commission is available
online in Module 7.
• They will be evaluated by your table Support
Facilitator using the statement rubric.
• We will discuss the Judgment Scenarios
Sunday morning.
Copyright © 2014 by ABET
Day One Closing
• Day One once-around the room
 What do you leave with today?
 What’s “top of mind” about the day?
• Thank you for your effort today!
Copyright © 2014 by ABET
Welcome Back!
PEV Face-to-Face Training
Copyright © 2014 by ABET
Agenda - Day Two
Module Name
Timing
Day Two Overview
8:00 – 8:10
9 – Context for Judgment
8:10 – 9:00
10 – Informal Program Debrief & Exit Meeting
9:00 – 10:05
Break
10:05 – 10:20
11 – Writing Exit Statements
10:20 – 11:00
12 – Connection to PEV Competencies
11:00 – 11:20
13 – Post-Visit Work
11:20 – 11:29
14 – Post Training
11:29 – 11:38
15 – Closing Exercise
11:38 – 11:45
Copyright © 2014 by ABET
Module 9
The Context for Judgment
Copyright © 2014 by ABET
Context for Judgment
• Evaluation of a program for accreditation
•
•
requires PEV judgment on the extent to
which each criterion is satisfied.
PEV judgment is based on evidence
obtained during the evaluation process.
Judgment was demonstrated in the exit
statements written last night for Upper State
University as discussed yesterday.
Copyright © 2014 by ABET
Context for Judgment
Throughout the case study:
• Where did you use evidence as a basis for
•
•
•
judgment?
How and when did you see evidencebased judgment applied by others?
What did you struggle with?
Why is evidence-based judgment
important?
Copyright © 2014 by ABET
Context for Judgment
• Accreditation has moved away from a
•
rigid “bean counting” process to one that
is based on continuous quality
improvement processes.
ABET Criteria are based on principles of
Continuous Quality Improvement (CQI).
Copyright © 2014 by ABET
Context for Judgment
• An educational program CQI process should reflect
a clear understanding of:








Mission
Constituents
Objectives
Outcomes
Processes (internal practice to achieve the outcomes)
Facts (data collection)
Evaluation (interpretation of facts) and
Action (feedback to support decision making and improve
processes)
Copyright © 2014 by ABET
JUDGMENT SCENARIO
EXERCISE
Handout 9-2
Copyright © 2014 by ABET
Context for Judgment
• Many questions of judgment center around
•
assessment and continuous improvement.
When such issues arise, ASK:
 Are assessment processes adequate to determine
achievement of each of the criterion?
 Are assessment processes robust enough to identify
shortcomings and achievements?
 Are assessment processes sustainable?
 Will assessment processes lead to program
improvements if appropriate?
Copyright © 2014 by ABET
Program Educational Objectives
• Definition:
 Program educational objectives are broad
statements that describe what graduates are
expected to attain within a few years of
graduation. Program educational objectives
are based on the needs of the program’s
constituencies.
• Key Words
 Broad Statements; Expected to Attain;
Needs of Constituencies
Copyright © 2014 by ABET
PEOs: Important Concepts
• Should be Forward Looking
• Should use Expected to Attain language
•
•
•
and not Preparing Graduates to Achieve
Must periodically review PEOs
Must ensure PEOs remain consistent with
the institutional mission, the program’s
constituents’ needs, and these criteria
Can include language from Student
Outcomes
Copyright © 2014 by ABET
Summary
• Judgment centers on the observed
•
evidence that supports achievement of
each criterion.
When issues arise about specific
components of the criteria, ASK:
 Does the evidence support my decision?
 What additional evidence could I seek?
Copyright © 2014 by ABET
JIT
Evaluation
Copyright © 2014 by ABET
Module 10
Informal Program
Debrief & Exit Meeting
Copyright © 2014 by ABET
Informal Debrief Preparation
• Must have a complete Exit Statement and
•
•
Program Audit Form at this stage.
Must be able to elaborate on the statement
and explain its conclusions without falling
into a debate with the program head.
Must be able to relate every shortcoming to
the Criteria.
Copyright © 2014 by ABET
Informal Debrief Preparation
• Must understand what not to say, particularly
•
•
the recommended accreditation action.
May make comments that are intended to be
helpful while making it clear that these
comments (which may appear in the report as
Observations) have no relation to the findings
with respect to the Criteria.
Must remain completely consistent in all
statements to every person.
Copyright © 2014 by ABET
INFORMAL DEBRIEF
DEMONSTRATION
• Note: This is NOT Upper State
University
Handout 10-1
Copyright © 2014 by ABET
EXERCISE
SIMULATING THE INFORMAL
DEBRIEF
Copyright © 2014 by ABET
Exit Meeting
• Must have a complete Exit Statement and
•
•
•
Program Audit Form that follow all ABET
guidelines.
The statement is read verbatim.
There must be no deviation from the written Exit
Statement.
If questions are asked, the answers must
essentially re-state the Exit Statement without
introducing new material.
Copyright © 2014 by ABET
Exit Meeting
• Always consult with the Team Chair if in
•
•
doubt about the appropriate response to
a question.
Always be courteous and respectful.
Thank institution for campus hospitality.
Copyright © 2014 by ABET
What NOT to Do at the Exit Meeting
• Do not deviate from the Exit Statement.
• Do not make comments regarding any aspect of the
•
•
•
•
•
program or institution other than those in the Exit
Statement.
Do not compare the visited institution to your own
institution.
Do not make disparaging comments about students or
anyone else.
Do not disagree with the other members of your team in
any way.
Do not make any social conversation that could possibly
be misinterpreted.
Do not arrive late or leave early.
Copyright © 2014 by ABET
JIT
Evaluation
Copyright © 2014 by ABET
Module 11
Writing Exit Statements
& Other Items
Copyright © 2014 by ABET
Writing Exit Statements
& Other Items
• Observations & Statement Format
• Interim Actions
• Evaluation of On-Line Programs
Copyright © 2014 by ABET
Types of Statements
Exit
statement for
Program A
Exit
statement for
Program B
Exit
statement for
Program C
Team Chair
Exit
Statement
Draft Statement
Final Statement
Handout 11-1
Copyright © 2014 by ABET
Strengths & Observations
Copyright © 2014 by ABET
Strength Statements
• A strength is an exceptionally strong, effective
•
practice or condition that stands above the
norm, and has a positive effect on the program.
For each strength, state:
 What was observed
 What makes it stand above the norm
 The positive effect it has on the program
• ETAC Note: Strengths are considered
observations.
Copyright © 2014 by ABET
Strength Example
The Office of Student Services and Career
Development is highly successful in placing
the graduates of the college.
Important services provided by this office include
career counseling, workshops on interviewing
readiness, and training in study skills. The office
uses JOBTRAK to assist with referrals and job
placement.
These activities help the individual programs
achieve their objectives in the successful job
placement of their graduates.
Copyright © 2014 by ABET
Statement of Observation
Definition:
“a comment or suggestion which does
not relate directly to the accreditation
action but is offered to assist the
institution in its continuing efforts to
improve its programs”
- ABET Accreditation Policy & Procedures Manual Section
II.G.9.a.(3)
Copyright © 2014 by ABET
Example
Statement of Observation
Use of CSE 2221, Computing in
Engineering & Science, as a prerequisite to senior-level courses could
be considered as a way to enhance the
computational experience of the
students.
Copyright © 2014 by ABET
Strength & Observation Exercise
• Take a 5 minutes and read the Strength
•
•
•
statements on page 1 of the handout.
Determine whether each statement is an
appropriate Strength statement or not.
Discuss as a table.
Repeat for the Observation statements
on page 2 of the handout.
You have 20 minutes.
Copyright © 2014 by ABET
Handout 11-2
Interim Actions
• Programs that receive one or more
•
•
•
Weakness or Deficiency will be subject to
an Interim Action
Interim Report/Interim Visit
Interim Report – Team Chair Writes Draft
Statement
Interim Visit – Exit Statement, PAF, and
Draft Statement Written – Some
Differences From General Review –
Experienced PEVs Involved
Copyright © 2014 by ABET
Evaluation of On-Line Programs
• Growing need to evaluate programs
•
•
offered in on-line format only
ABET has evaluated a few of these
programs in recent past
On-line programs are evaluated using
the same Criteria as used for traditional
programs.
Copyright © 2014 by ABET
Visit to On-Line Program a Bit
Different
• The Evaluation visit is usually made to a
•
“central office” rather than a traditional
campus.
Faculty and student interviews usually
held using Internet based
communication utilities like Skype,
GoToMeeting, etc.
Copyright © 2014 by ABET
Courses and Assessment
• Assessment of Student Outcomes in many
•
•
organizations done by professional
organizations
Faculty involvement needs to be
determined, particularly in “closing the
loop”
Curricula and courses may be more
practically oriented. Display materials may
be different than found in more traditional
programs.
Copyright © 2014 by ABET
Faculty and Facilities
• Qualifications of faculty may be different
•
than found at more traditional programs.
Part-time faculty from industry may have
more to offer to students than those with
more traditional academic programs.
Hardware and course software needs to
be evaluated at “central office”. Library
access will probably be on-line.
Copyright © 2014 by ABET
JIT
Evaluation
Copyright © 2014 by ABET
Module 12
Connection to Competencies
Copyright © 2014 by ABET
ABET PEV Competency Model
•
•
•
•
•
•
Technically Current
Effective Communicator
Professional
Interpersonally Skilled
Team-Oriented
Organized
Handout 12-1
Copyright © 2014 by ABET
PEV Competency Exercise
At your table, discuss:
• Which competencies were effectively
demonstrated within the group? How?
• Which competencies do we individually and
collectively need to improve?
Copyright © 2014 by ABET
Competency Based Performance
Appraisal
• The competencies form the basis of
performance appraisal.
 Evaluation of PEV Candidates
• You will be evaluated by your table Support
Facilitator and PEV Candidate peers at the
end of training.
 Evaluation of PEVs
• Each PEV will be evaluated by the institution,
Team Chair and peers after each visit.
Copyright © 2014 by ABET
Performance Appraisal Process
• Serves as a vehicle for professional
•
•
•
•
development.
Tool for continuous improvement.
Clearly communicates what is expected.
Everyone gets evaluated every time.
Evaluate against Competency Model.
Copyright © 2014 by ABET
Competency-Based
Performance Appraisals
• 360-Degree Process:
 Team Chair
 Institution
 Team Members
• “Met Expectations” (rating of 3) describes
•
how the competency is demonstrated previsit and on-site. This is the benchmark.
Area for actionable comments for items
below “3.”
Copyright © 2014 by ABET
Performance Appraisal Process
ABET Reviews
• Exceptional
Performers
• Consent Agenda
• Flags below 3
Copyright © 2014 by ABET
Executive
Committee
Review
Aggregate
Report;
Preliminary
Decision
• Recognition
• Remediation
• Removal
Professional
Society Additional Input
• Extenuating
Circumstances
• Add, but not delete
Final Decisions
Notification to
PEV
• Executive
Committee to
Society
• Society to PEV
• ABET send
performance
appraisal results
JIT
Evaluation
Copyright © 2014 by ABET
Module 13
Post-Visit Work
Copyright © 2014 by ABET
The Accreditation Timeline
January
Institution requests
accreditation for
programs
March - June
Team chairs
assigned, dates
set, team members
chosen
December - February
Draft statements edited
and sent to institutions
Year 1
February - May
Institution prepares
self-evaluation
(Program Self-Study
Report)
Copyright © 2014 by ABET
September - December
Visits take place, draft
statements written and
finalized following
7-day response period
May - June
Necessary changes
to statement,
if any, are made
August
Institutions notified
of this action
Year 2
February - April
Institutions respond
to draft statement
and return to ABET
July
Commission meets
to take final action
Post-Visit Process
• 7-Day Response from institution.
• Draft Statement prepared, edited, and sent
•
•
•
•
to institution.
30-Day Due Process Response from
institution.
Final Statement prepared and edited.
Commission takes final accreditation
action.
ABET sends Final Statement and
accreditation letter to institution.
Copyright © 2014 by ABET
PEV Responsibilities – Post-Visit
• Within one week after the visit is
completed, the Program Evaluator
MUST:
 Submit visit expenses using the on-line
expense system. Include original itemized
receipts.
• And, the PEV should:
 Complete Team Chair and Peer PEV
Performance Appraisal Forms online.
Copyright © 2014 by ABET
PEV Responsibilities – Post-Visit
• Between Visit and Commission Vote,
PEV may be asked to:
 Review Due Process materials.
 Consult with Team Chair on accreditation
action.
Copyright © 2014 by ABET
ABET Travel Policy
• Always download and use the latest ABET
•
•
•
Travel Policy & Procedures Manual.
Make reservations through ABET’s travel agent.
Submit expenses online using the ABET on-line
expense web tool.
Scan original receipts for ALL lodging, meals,
and transportation (tolls, parking, taxi, bus, rail)
or submit original receipts by mail.
Copyright © 2014 by ABET
Module 14
Post-Training
Copyright © 2014 by ABET
What Happens Next?
• Leave behind completed Exit Statement and
•
•
•
•
Program Audit Form (PAF).
Evaluation of Exit Statement & PAF by Facilitator.
Evaluation by Facilitator and peers using modified
PEV performance appraisal.
Member society notified of completion of training
and availability of evaluation results.
Update your bio on the ABET secure website with
dates available for a fall visit.
Copyright © 2014 by ABET
Contact Your Training Mentor
• Your Training Mentor is there to support
•
you!
Debrief with your Training Mentor soon.
 Focus on key learnings from the training.
 Ask any follow up questions.
 Share any other feedback on the training
(pre-work or face-to-face).
Copyright © 2014 by ABET
Getting Assigned
• Member society makes assignments based
•
•
•
on institutions’ program needs and PEV
pool.
Some societies will require Observer visit
first.
Institutions check team list for any conflict
of interest.
Team Chair approves your participation
and obtains approval from the institution
then contacts you.
Copyright © 2014 by ABET
Questions and Answers
• Any questions on what happens
after the training?
Copyright © 2014 by ABET
MODULE 15
CLOSING
Copyright © 2014 by ABET
What Have We Accomplished?
A Lot!
• Compared to yesterday morning, how
prepared do you now believe you are to go
out on an ABET accreditation visit?
Copyright © 2014 by ABET
Evaluations
• Complete evaluations and leave in the
evaluation envelope at your table.




Support Facilitator
PEV Candidate Peers
Just-in-Time Evaluations
Overall Training Evaluation
Copyright © 2014 by ABET
Thank you.
Travel safely.
Copyright © 2014 by ABET