National Commission for
Academic Accreditation and
Assessment
PLANNING AND IMPLEMENTATION
OF SELF-STUDY
for Institutional Accreditation
Dr. George Peterson
Dr. Eqbal Darandari
May. 21-22, 2011
training@ncaaa.org.sa
Copyrights © for materials are reserved for NCAAA
AIMS OF the WORKSHOP
Providing
• Understanding of the process of Self Study
• Guidance on the content of a Self Study Report
(SSR) and its role in the process of External
Review/Accreditation
– Meeting generally accepted standards in higher
education.
– Institutional mission and objectives.
– Research report on the quality of the institution.
– Self assessment against given criteria
•
Copyrights © for materials are reserved for NCAAA
Learning Outcomes
By the end of the workshop, the participant will be able to:
Discuss
and explain
how to plan,
manage,
and
implement
Institutional
Self Study
Implement
Skills of
Institutional
Self Study
using
NCAAA
templates
Lead the
design and
writing of
Institutional
Self Study
Report.
Copyrights © for materials are reserved for NCAAA
Day 1
S1
Introduction and
Purpose of the SSR
S2
S3
Planning the SSR
Using NCAAA
standards and
evidence
Day 2
S1
S2
S3
Structure and
Content of the SSR
Recommendations,
Action Plans and
Conclusions
Applications,
Questions, and
Sharing Experiences
Copyrights © for materials are reserved for NCAAA
Day 1
S1
Introduction
and Purpose
of the SSR
S2
S3
Planning the SSR
Using NCAAA
standards and
evidence
Copyrights © for materials are reserved for NCAAA
Introduction
Participants Preparing for accreditation
Participants Preparing for developmental
reviews
Participants about to start institutional self
study
Copyrights © for materials are reserved for NCAAA
Introduction:
Important Facts
• In the last 6 years, NCAAA and its
partners made a lot of improvements.
• More than 6000 faculty members have
participated in the NCAAA's local and broad
training programs.
• 90% of the institutions have establish their
own quality units or centres.
• Over than 70% of HEIs have made initial self
evaluations at institution and program levels
• Many institutions have started a planimplement-monitor-review cycle.
Copyrights © for materials are reserved for NCAAA
Introduction: Important Facts
• Many institutions conducted strategic
plans for quality improvement and
included actions to implement the
requirements of the quality assurance system
dealing with significant deficiencies
discovered.
• 5 institutions and 41 programs have
undergone the developmental reviews.
• 6 institutions and 32 programs have been
reviewed for accreditation
Copyrights © for materials are reserved for NCAAA
Evaluation and
Accreditation Steps
1. Establish Quality Assurance System
2. Meet Institution Goals and NCAAA
Standards
3. Meet Eligibility Requirements
4. Prepare Self Study Report (SSR)
5. External Review and Report
6. Accreditation
Copyrights © for materials are reserved for NCAAA
NCAAA Accreditation Flowchart
NCAAA Secretary
General approves
sending Review Panel
NCAAA
Secretary General
approves to start
accreditation
process
Recommends
Corrections
Institution or
Program
with proper
government
approvals
Application
to start
accreditation
process
On-site ASG
inspection,
meeting
Liaison Officer
assigned by ASG
Accreditation NCAAA Applicant
Time Line
3 months to 1 year
NCAAA
Board
warns or
revokes
NCAAA
grants accreditation
Recommends
Review Panel
vi
Recommends
Corrections
Recommends
Accreditation
Accredited
Institution/Program
Institution
Program
Preparation
to meet
NCAAA
requirements
Evaluation
date set
Liaison Officer
works with
Institution
Program Quality
Assurance
Center
Eligibility
Requirements
Met
Complete
Scales Report
Self Study
Strategic Plan
Specifications
KPIs in place
Standards Met
Show Evidence
Preparation to be Accredited
1 to 2 years
On-site visit
Review Panel
Evaluation
and Report
Annual Reports
Annual Review
Inspections
Monitoring
Strategic Plan Results
Training
Every 5 years -new
Self–Study and onsite Review Panel
Evaluation
NCAAA Accreditation
continue to meet standards
and follow protocol
NCAAA QUALITY PLANNING
AND REVIEW CYCLE
Environment
Scan
Five yearly
Review
Review
Performance
Action
Plan
Define/Review
Mission/Goals
Set
Objectives
Yearly
Monitor
Results
Implement
Plan
Copyrights © for materials are reserved for NCAAA
Develop
Plan
Institutional and Program
Accreditation
Accreditation is public recognition that
quality assurance Standards are met
in the management and delivery of
learning outcomes achieved by
students.
Standards must be equivalent to high
quality international Standards.
Copyrights © for materials are reserved for NCAAA
Accreditation Process
Process involves a rigorous self evaluation
in relation to
Institution Goals + the eleven NCAAA
Standards followed by an external review.
An external Review Panel will verify the
conclusions of the Institution’s self
evaluation according to its quality of
performance in relation to the Standards.
If an Institution is to be accredited then ALL
eleven Standards must be met (including
Mission and Goals).
Copyrights © for materials are reserved for NCAAA
TRUST
It means that we can trust the university,
institution, college, program, or
department
It means the institution or program is
doing what it says it is doing.
Copyrights © for materials are reserved for NCAAA
Foundations of
Accreditation
•
•
•
•
•
•
•
•
Trustworthiness
Commitment
Quality Standards
Common Protocol
Transparent
Internal Self Evaluation
External Evaluation / Peer Reviews
Continuous Improvement
INSTITUTIONAL SELF STUDY
•
•
•
•
•
Primary focus of External Review
Takes place before accreditation
Institutional Quality Unit and Committee
Consideration of performance
Report and conclusions
Copyrights © for materials are reserved for NCAAA
NCAAA: “the BIG 6”
Internal Evaluation Includes:
Surveys
SWOT Analysis
Specifications
Self-Evaluation Scales
Self Study Report (SSR)
Strategic Action Plan
Purposes of
Self Study
• A powerful systems approach to
improving
student performance and
system effectiveness
providing results over time
Copyrights © for materials are reserved for NCAAA
Purposes of
Self Study
By conducting a data-driven-statistical
Self-Study of its own strengths and
opportunities for change, the institution
or program will…
Involve the campus community to
evaluate its mission, educational
programs, activities, ongoing processes of
planning, resource allocation, and
university renewal; SURVEYS
involvement = investment
Purposes of
Self Study Report (SSR)
• Identify Strengths and weaknesses
• Framework for Accreditation Processes
• Enhances Development and
Effectiveness
Copyrights © for materials are reserved for NCAAA
Purposes of
Self Study Report (SSR)
Provides framework for accreditation
‘should provide sufficient description to enable the review team
to understand the key features of the institution’s approach to
assuring quality and securing standards, but should focus on
the effectiveness of that approach. Where the institution
expresses confidence in its own effectiveness, the evidence
upon which its view is based should be made clear’
‘The periodic institution self study provides the basic resource
for the external review for re-accreditation .’
NCAAA: Handbook for Quality Assurance and Accreditation
DESCRIPTION – ANALYSIS – EVIDENCE – REFLECTION –
ACTION
Copyrights © for materials are reserved for NCAAA
Purposes of
Self Study Report (SSR)
Provide a framework for continuous improvement of
the institution;
Initiates an attitude or culture of growth and
improvement
Enhance the shared understanding of its mission as
an educational university or college; UNITY and
team play
Ensure that the university’s or college programs and
activities strive towards a common vision; all on the
same spacecraft going to the same planet
Copyrights © for materials are reserved for NCAAA
Purposes of
Self Study Report (SSR)
Provide comprehensive, tangible, and
coherent recommendations for the
university’s future plans—
recommendations that have been
developed, evaluated, and committed to
by the entire community; are
comprehensive and measurable VS
personal agenda
Copyrights © for materials are reserved for NCAAA
SSR WAY FORWARD
‘Periodic self-study examines the institution in greater
depth, checking on how effectively it is achieving its
mission and objectives, and planning for any changes
that are needed.’
‘A self-study should also consider how an institution has
evolved over time in response to evaluations and
changing circumstances and the probability that it will
continue to be refined and improved in the future.
Consequently developments over the period of review
need to be considered as well as quality issues at the
time of the self-study.’
NCAAA: Handbook for Quality Assurance and Accreditation
Copyrights © for materials are reserved for NCAAA
SSR Stakeholders
University leaders institutions
Program leaders colleges /
departments
Professors classrooms
Students enrolled and alumni
Public employers and community
MOE, MHE, and government leaders
PROPERTIES OF A
SELF-STUDY REPORT (SSR)
‘Many self-study documents (report) were found to be
frank, honest and self-critical, though occasionally
criticisms directed by institutions at their own
arrangements for quality and academic standards
were found to have overstated their deficiencies.
Conversely, in a few cases,
other self-study
documents were found to have overstated the
strengths of the institution's arrangements.’
QAA, 2007
Institution Eligibility for Applying for
Accreditation and Conducting SSR
Institutional Eligibility Check List
1.
2.
3.
4.
5.
6.
7.
8.
9.
Final license or approved government institution
Activities consistent with license or approval
Approved and consistent with license or approval
Strategic plan including plan for quality
Availability of policies, regulations and terms of reference
Published guides or handbooks for students
Program specifications for all programs
Course specifications
Regulations and descriptions of processes for program approval,
changes, and review
10.
Systems for monitoring quality and improving programs
11.
Central maintenance analysis and reporting of statistical data
Student surveys
Quality assurance system covering all standards
Data on Key Performance Indicators
Arrangements for comparative benchmarks
Systems for maintenance of data on research (if applicable)
Systems for maintenance of data on community service activities
Students graduated
Compliance with standards for accreditation.
12.
13.
14.
15.
16.
17.
18.
19.
Criteria
Criteria
Confirmed
Met
(NCAAA)
Group Activity 1
20 minutes
(10 minutes for discussion and 10 minutes to report)
Has your institution met
eligibility requirements?
Use the Checklist
provided.
Copyrights © for materials are reserved for NCAAA
Day 1
S1
S2
Introduction and
Purpose of the SSR
Planning the
SSR
S3
Using NCAAA
standards and
evidence
Copyrights © for materials are reserved for NCAAA
SSR STARTING POINT
1. First, read and analyze the NCAAA
standards;
2. Second, make a commitment to
meeting NCAAA Standards and protocol.
3. Third, assess the University or Program
status in relationship to the NCAAA
standards
Copyrights © for materials are reserved for NCAAA
SSR is WORK…WORK…
and more WORK
1. TIME is needed for university staff to
do all the work that is required. Deans
must consider reduced teaching and/or
research loads, or additional
compensation
2. Strong and meaningful support is
needed from each college dean and
university administrator to create
enthusiasm and positive attitudes.
Copyrights © for materials are reserved for NCAAA
PLANNING THE SSR
1.
2.
3.
4.
5.
6.
7.
What are we trying to do?
Why are we doing it?
How are we doing it?
Why is that the best way to do it?
How do we know if it works?
How do you change if need to?
How evaluate success of change?
Copyrights © for materials are reserved for NCAAA
PREPARATION OF INSTITUTION
Self Study– CONSIDERATIONS?
•
•
•
•
•
•
•
•
•
Led by senior member of faculty
Guided/approved by a Steering Committee
Working parties
Training
Buy-in of Senior Staff
Annual reports
Editorial control
Benchmarks
Timing
Copyrights © for materials are reserved for NCAAA
PROCESS FOR SS
•
•
•
•
Resources
Communication
Independent analysis and comments
Concluding statement and Action
Plan
Copyrights © for materials are reserved for NCAAA
KEY is the FACULTY
NCAAA accreditation and the whole
university or college level of success
is directly tied to the faculty.
The faculty is the fulcrum for quality, a
SSR, and for academic success.
Maximum faculty involvement
Strong faculty = Strong university
Copyrights © for materials are reserved for NCAAA
SSR Leadership Team
NCAAA accreditation SSR is the work
of the whole faculty team and not
just 20% of the faculty.
Team LEADERS are needed…
LEADERSHIP is fundamental
SSR Accreditation Leadership
(Deanship of Evaluation, Quality, and
Academic Accreditation)
Copyrights © for materials are reserved for NCAAA
Steps in Conducting an
Institutional Self Study
1.Nominate
Chair &
Coordinator
2. Appoint
Steering
Committee
3.Plan
Strategy for
Self Study
5.Appoint Sub
Committees
5. Special tasks that may need attention for the review
Provide information about the self study to
faculty, staff and students.
Coordinate activities for the institutional review
and the program reviews to minimize duplication.
6.Review
Performance on
Indicators
10
Self-Study Report
Prepared
8.Gather Information
Complete Self Eva
Scales
11. Report given to
Rector and
Governing Board
9. Sub
Committees
Draft Reports
12
Report given to NCAAA
for External Review
SSR Accreditation
Leadership Team
The Accreditation Leadership Team consists of key individuals
that will drive the Self-Study process to a successful
completion.
Deanship of Evaluation, Quality, and Academic
Accreditation
. Its primary task is to plan, guide and coordinate the Self-Study
Report (SSR). It’s responsibilities are:
1. Appointing necessary subcommittees,
2. Preparing the Self-Study Report with all supporting
documents and evidence,
3. Setting the schedule for the Review Panel site visit.
4. Participating as hosts and resources to the Review Panel
team and to fellow-up with faculty members and
administrators for sessions.
5. Reviewing the report of the visitation Review Panel.
6. Make recommendations for continued improvement.
7. Follow-up on Review Panel Report recommendations
SSR Leadership Chair
1.Keep everyone on task
2.Ensure the SSR subcommittees are
functioning and carrying out their tasks
ON TIME
3. Encourage and motivate those working with
the accreditation process
4. Work closely with the university’s administrator
5. Attend the pre-visit with the NCAAA Visitation
Team and Liaison Officer
6. Chair the SSR Leadership Team meetings
7. Ensure that the work of the SSR TEAM stays on
the task timeline.
Copyrights © for materials are reserved for NCAAA
Qualifications of the SSR
Leadership Team & Chair
Dedication to the Mission of the university
Trusted members of the university
Familiarity with the university
No conflict of interest employment, either
personally or of a family member, within the
university
Strategic planning, technological, and
organizational skills
Strong interpersonal skills
TEAM BUILDER
Copyrights © for materials are reserved for NCAAA
Forming Steering
Committee
• Based on Institution size, type, teams
• Based on Standards, Group of
Standards, or functions
Decide on Model of Self Study
Defined a detailed Plan with
responsibilities and timeline
Copyrights © for materials are reserved for NCAAA
Forming Steering Committee:
Example
Steering Committee
Sub-committee 6
Follow up coordination
Sub-committee 1
Standard 1
Standard 2
Sub-committee 3
Standard 5
Standard 6
Sub-committee 5
Standard 10
Standard 11
Standard 3
Sub-committee 2
Standard 4
Sub-committee 4
Standard 7
Standard 8
Standard 9
Finding the Story:
Meaning and Interpretation
Self-STUDY Examination and Observation
How do we collect and arrange the data? DESIGN
How do we interpret the data? MEANING
What does the meaning imply? IMPLICATIONS
How do we apply the data? ACTION?
Copyrights © for materials are reserved for NCAAA
Thinking about SSR
We need to understand that NCAAA
accreditation and writing a SSR is
not just meeting Standards and
following a process protocol.
We need to understand that NCAAA
accreditation and writing a SSR are a
comprehensive quality assurance
system.
System Elements
Colleges &
Technology
Universities
MIS
Transportation
Community
Executive
Staff
Classified
Programs
Staff
Local
Food
Accounting
Business
Service
Finance
Facilities &
Parents
Teachers Maintenance
Students
45
Systemic University Improvement
Alignment
At
ALL
Levels
College
Program
DEPARTMENT
CLASSROOM
STUDENTS
IMPROVEMENTS
University
All-Comprehensive
Institutional
System for Systems
Horizontal
and
Vertical
Copyrights © for materials are reserved for NCAAA
SSR in Three Steps
Step One – Study
Step Two – Analyze
Step Three - Show Evidence
Copyrights © for materials are reserved for NCAAA
3 Steps of Self-Study
Step Two: Analyze Data,
Descriptions, Quality Indicators,
Evidence and Documents…
• In this step, the Institution does a self-rating
based on the Standard and KPI Quality
Indicators and analyzes the information.
• The analysis identifies findings, strengths,
and recommendations (weaknesses and
needs).
• After the analysis, intervention actions will
address the needs that were suggested.
Copyrights © for materials are reserved for NCAAA
Reason: What rational?
What is the thinking that is behind
the analysis or assessment?
How are the statistics, graphs, or
tables interpreted or given
meaning?
How is the MEANING applied?
What is the logic behind the
evaluation?
Copyrights © for materials are reserved for NCAAA
3 Steps to Self-Study
Step Three: Show Evidence
• In this step, the Institution or Program
collects documents and artifacts that
provide verification that the Standards are
met and the process was scientific.
• This information is either included in the
Self-Study Report or available for the
Review Panel to evaluate during the visit.
• Interviews and observations take place
during the accreditation review visit to provide
professional opinions.
Copyrights © for materials are reserved for NCAAA
Support Evidence
What evidence supports that a
quality indicator is met and
evaluated accurately?
What evidence documents that
the answers to the questions
are accurate?
Copyrights © for materials are reserved for NCAAA
Group Activity 2
20 minutes
(10 minutes for discussion and 10 minutes to report)
• What steps have you already taken
to conduct your institution Self Study
and to prepare your SSR?
• What difficulties have you
experienced?
• How have you resolved these
difficulties?
Copyrights © for materials are reserved for NCAAA
Day 1
S1
S2
Introduction and
Purpose of the SSR
Planning the SSR
S3
Using NCAAA
standards and
evidence
Copyrights © for materials are reserved for NCAAA
NCAAA Standards
1)
2)
3)
4)
5)
Mission and Objectives
Governance and Administration
Quality Assurance and Improvement
Learning and Teaching
Student Administration and Support
Services
6) Learning Resources
7) Facilities and Equipment
Copyrights © for materials are reserved for NCAAA
NCAAA Standards
8) Financial Planning and
Management
9) Faculty, Staff and Employment
Processes
10)Research
11)Institutional Relationships with
the Community
Copyrights © for materials are reserved for NCAAA
Using Self Evaluation
Scales (SES)
Initial Self
Evaluation
Guide External
Review Panel
Self Study
Continuous
Monitoring
Mid Periodic
Reviews
Copyrights © for materials are reserved for NCAAA
Self Evaluation Scales (SES)
11 Standards
Sub- Standards
How well is this done?
(enter stars)
NA
Using Scales
N
Y
Is this true?
Y/No/NA
-------
0
*
**
***
****
*****
Copyrights © for materials are reserved for NCAAA
Improvement
Required
Good
Performance
High Quality
Performance
Self Evaluation Scales (SES)
Using stars for evaluations
Yes/No/Not applicable
IMPROVEMENTS REQUIRED
No star - practice, though relevant, is not followed
One star - practice followed occasionally but quality is poor or not
relevant
Two stars -practice usually followed but quality is less than
satisfactory.
GOOD PERFORMANCE
Three stars - practice followed most of the time
- evidence of the effectiveness of activity is usually
obtained
- evidence indicates satisfactory performance normally
achieved
- plans for improvement in place and progress monitored
Copyrights © for materials are reserved for NCAAA
Self Evaluation Scales (SES)
Four stars-
HIGH QUALITY PERFORMANCE
practice consistently followed
indicators of quality of performance are established
mainly high quality performance - still some room
for improvement
plans for improvement implemented and progress
monitored and reported
Five stars - practice is followed consistently
- high standard supported by evidence
- evidence indicated superior to comparable
institutions
- realistic strategies for improvement
Copyrights © for materials are reserved for NCAAA
Good Practices Relating to Standard 3: Quality Management
3.1
Institutional Commitment to Quality Improvement
An institution must be committed to maintaining and improving quality through effective
Is this true?
leadership and active involvement of teaching and other staff.
Y/No/NA
3.1.1 The Rector or Dean strongly supports involvement in quality assurance processes.
3.1.2 Adequate resources are provided for the leadership and management of quality yes
assurance processes, and provision of assistance where it is needed.
yes
3.1.3 All teaching and other staff participate in self-assessments and cooperate with
reporting and improvement processes in their sphere of activity.
yes
3.1.4 Creativity and innovation combined with clear guidelines and accountability
no
processes are actively encouraged at all levels.
3.1.5 Mistakes and weaknesses are recognized and used as a basis for planning for
yes
improvement.
3.1.6 Improvements in quality are appropriately acknowledged.
yes
3.1.7 Evaluation and planning for quality improvement are integrated into normal
yes
administrative processes.
Overall Assessment
Comment_____________________________________________________________
How well is
this done?
(enter stars)
***
****
***
0
**
***
****
***
_____________________________________________________________________
Priorities for improvement_______________________________________________
____________________________________________________________________
Independent Opinion
Comment_____________________________________________________________
***
4.1 Student Learning Outcomes
Intended student learning outcomes must be consistent with the National Qualifications
Framework, and with generally accepted standards for the field of study concerned
including requirements for any professions for which students are being prepared.
4.1.1 Intended learning outcomes are specified after consideration of relevant academic
and professional advice.
4.1.2 Intended learning outcomes are consistent with the Qualifications Framework.
(covering all of the domains of learning at the standards required)
4.1.3 Intended learning outcomes are consistent with requirements for professional
practice in Saudi Arabia in the fields concerned. (These requirements should include
local accreditation requirements and also take account of international accreditation
requirements for that field of study, and any Saudi Arabian regulations or special regional
needs.)
4.1.4 If an institution has identified special attributes to be developed in students
graduating from the institution comprehensive strategies are established for these to be
developed. (This means that the attributes to be developed in students are clearly defined,
strategies for developing them planned and implemented across all programs, and
mechanisms for assessing and reporting on the extent to which graduating students have
developed them, are in place.)
4.1.5 Appropriate program evaluation mechanisms including graduating student surveys,
employment outcome data, employer feedback and subsequent performance of graduates
are used to provide evidence about the appropriateness of intended learning outcomes and
the extent to which they are achieved. (see also sections 4.3 and 4.4 dealing with
verification of standards and program evaluation processes)
Overall Assessment
Comment_______________________________________________________________
Is this true?
Y/No/NA
yes
yes
yes
yes
yes
How well is
this done?
(enter stars)
***
***
***
****
***
***
_______________________________________________________________________
Priorities for improvement_________________________________________________
______________________________________________________________________
Independent opinion
Comment_______________________________________________________________
***
4.3 Program Evaluation and Review Processes
The quality of all courses and of the program as a whole must be monitored regularly through
appropriate evaluation mechanisms and amended as required, with more extensive quality
reviews conducted periodically.
4.3.1 Courses and programs are evaluated and reported on annually with information about the
effectiveness of planned strategies and the extent to which intended learning outcomes are
being achieved.
4.3.2 When changes are made as a result of evaluations details of those changes and the
reasons for them should be retained in course and program portfolios.
4.3.3 Quality indicators that include learning outcome measures are identified and used for all
courses and programs.
4.3.4 Records of student completion rates in all courses and the program as a whole are kept
and used as quality indicators...
4.3.5 Quality indicators for programs are kept on regularly reviewed by senior administrators
and quality committees.
4.3.6 A selection of quality indicators including progression and completion rates are
compared across the institution and by reference to appropriate external benchmarks, and
action taken when problems are identified.
4.3.7 If problems are found through program evaluations appropriate and timely action is
taken to make improvements
4.3.8 In addition to annual evaluations a comprehensive reassessment of the program should
be conducted at least once every five years. Procedures for conducting these reassessments
should be consistent with policies and procedures established for the institution.
Is this true?
Y/No/NA
yes
yes
yes
yes
yes
yes
yes
yes
How well is
this done?
(enter stars)
***
***
***
****
**
***
***
***
***
Overall Assessment
Comment_______________________________________________________________
Priorities for improvement_________________________________________________
Independent opinion
Comment_______________________________________________________________
***
Evidence Wheel
Profile
Surveys
NCAAA
Accreditation
SWOT
Analysis
Strategic
Plan
Self
Study
Report
Program &
Course
Specifications
Copyrights © for materials are reserved for NCAAA
EVIDENCE
What EVIDENCE is needed to support that
this Standard or quality indicator is met?
What REASON or RATIONAL is given to
support that the quality indicator is met?
What DATA or STATISTICAL
DOCUMENTATION is needed to support
this quality indicator is met?
List evidence after each quality indicator
on the lines provided
How are we going to get
information???
What methods are used?
1. Statistical Surveys WHO? For WHAT?
2. Historical Records WHAT? For WHO?
3. Structured-Unstructured Interviews
WHO? WHAT to ASK?
4. Observations WHAT DO TO SEE?
WHY?
5. Questionnaires WHO? WHAT to ASK?
Copyrights © for materials are reserved for NCAAA
Two Kinds of Data
Quantifiable Data (objective)
What is this? How do we gather it? How do
we use it for NCAAA accreditation?
Qualitative Data (subjective)
What is this? How do we gather it? How do
we use it for NCAAA accreditation?
Copyrights © for materials are reserved for NCAAA
Apply to….
Self Study Report
Profile: Data, statistics, description
Standards Self Study Report
Course Specifications and Report
Program Specification and Report
Strategic Plan and Action Points
Evidence portfolio
Copyrights © for materials are reserved for NCAAA
Basic Questions for
Interpretation are….
Ask: How does this information relate to our
GOALS? …our VISION?
Ask: How does this information align to our
core values and beliefs?
Ask: What strengths and what weaknesses
do we see?
Ask: What problems still need to be
resolved?
Ask: What additional information is needed?
Copyrights © for materials are reserved for NCAAA
Basic Questions for meaning
and application are…
Ask: How do we improve?
Ask: What immediate action is needed?
Ask: What context or perspective is required
for this information to be understood?
Ask: What are the assumptions or givens or
presuppositions of this information?
Ask: What is the next step?
Ask: What bias or prejudices are present?
Copyrights © for materials are reserved for NCAAA
Key PERFORMANCE Indicators
and Benchmarks
What is a Key Performance Indicator
(KPI)?
KPIs are intended to provide specific
quantifiable data that can be used to
monitor quality and become selected
for benchmarking purposes.
Copyrights © for materials are reserved for NCAAA
Examples of Benchmarks
1. 15 students for every 1 teaching
staff
2. 75% of the teaching staff with
verified doctoral qualifications
3. 25.5 book titles held in the library
per student enrolled
4. 0.75 computer terminals per
student enrolled
Copyrights © for materials are reserved for NCAAA
SOURCES OF
EVIDENCE for SSR?
Copyrights © for materials are reserved for NCAAA
STUDENT EVIDENCE
• Achievement
– Completion/wastage rates/retention
– Entry qualifications
– Academic distance travelled (value added)
• Experience
– Contact hours
– Staff/Student Ratios
– Complaints/Appeals
– Selected Student Survey Indicators
Copyrights © for materials are reserved for NCAAA
STUDENT EVALUATION
SURVEYS
•
•
•
•
•
•
•
Anonymous
Common question sets
Open ended questions
Trend analysis
Frequency and Length
Course Evaluation Survey (After completion of each course)
Student Experience Surveys (After completion of half the
program)
• Program Evaluation Survey (After/ at the last semester of
program)
Copyrights © for materials are reserved for NCAAA
Course Evaluation Survey (CES)
Course Title _______________________________Program________________________________
Semester__________________________________Year___________________________________
Feedback from students is very important in trying to improve the quality of courses.
This is a confidential survey. Do not write your name or identify yourself in any way. Your responses
will be combined with the responses of others in a process that does not allow any individual to be
identified and the overall opinions will be used to plan for course improvements.
Please respond to the following questions by completely filling a response for each of your answers.
Questions about the start of the course:
1. The course outline (including the knowledge and skills the course was
designed to develop) was made clear to me.
2. The things I had to do to succeed in the course, including assessment tasks
and criteria for assessment, were made clear to me.
3. Sources of help for me during the course including faculty office hours and
reference material, were made clear to me.
Copyrights © for materials are reserved for NCAAA
Stron
g
ly Di
sagre
e
•
Disag
ree
e
 Strongly agree means the statement is true all or almost all of the time and/or very
well done.
 Agree means the statement is true most of the time and/or fairly well done.
 True sometimes means something is done about half the time.
 Disagree means something is done poorly or not often done.
 Strongly disagree means something is done very badly or never or very rarely done.
True
Some
times
Do not use red, green or yellow
Do not use highlighters
×
Agre
Use a pencil or blue/black pen only
Make heavy marks that fill in your response
√
ly A
gree
Do not mark in any of these ways
Stron
g
Do this
Questions about what happened during the course:
4.
The conduct of the course and the things I was asked to do were consistent
with the course outline.
5.
My instructor(s) were fully committed to the delivery of the course. (Eg.
classes started on time, instructor always present, material well prepared,
etc)
6. My instructor(s) had thorough knowledge of the content of the course.
7. My instructor(s) were available during office hours to help me.
8. .My instructor(s) were enthusiastic about what they were teaching
9
My instructor(s) cared about my progress and were helpful to me.
10. Course materials were of up to date and useful. (texts, handouts, references
etc.)
11. The resources I needed in this course (textbooks, library, computers etc.)
were available when I needed them.
Copyrights © for materials are reserved for NCAAA
12. In this course effective use was made of technology to support my learning.
13. .In this course I was encouraged to ask questions and develop my own ideas
14. In this course I was inspired to do my best work.
15. The things I had to do in this course (class activities, assignments,
laboratories etc) were helpful for developing the knowledge and skills the
course was intended to teach.
16. The amount of work I had to do in this course was reasonable for the credit
hours allocated.
17. Marks for assignments and tests in this course were given to me within
reasonable time.
18. Grading of my tests and assignments in this course was fair and reasonable.
19. The links between this course and other courses in my total program were
made clear to me.
Evaluation of the Course
20. What I learned in this course is important and will be useful to me.
21. This course helped me to improve my ability to think and solve problems
rather than just memorize information.
22. This course helped me to develop my skills in working as a member of a
team.
23. This course improved my ability to communicate effectively.
Overall Evaluation
Copyrights © for materials are reserved for NCAAA
24. Overall, I was satisfied with the quality of this course.
25. What did you like most about this course?
26. What did you dislike most about the course?
26. What suggestions do you have to improve the course?
Copyrights © for materials are reserved for NCAAA
KPI Quality Indicators
What quality indicators or KPIs
support or verify that a
Standard is met?
What questions should be
answered to indicate that the
Standard is met? (What evidence
and for what reason?)
Copyrights © for materials are reserved for NCAAA
Example: Mean Rating by Final Year Students on
Q 21 of Program Evaluation Survey
(Q21: I have developed the knowledge and skills
required for my chosen career)
College
Eco & Com
Education
Science
Islamic Stud
Medicine
Nursing
University
2007-8
Male
Female
2.98
3.43
3.22
3.16
3.60
3.65
4.12
3.89
3.31
3.58
3.22
3.88
3.55
3.72
2008-9
Male
Female
3.01
3.55
3.30
3.08
3.50
3.49
4.00
3.80
3.76
3.92
3.33
3.91
3.77
3.80
2009-10
Male Female
3.12
3.61
3.63
3.22
3.58
3.61
3.96
4.02
3.85
3.89
3.26
3.98
3.76
3.84
Questions
1.Which (if any) of these results would you regard as satisfactory?
2.What target(s) could you set for improvement?
3.What strategies could you suggest to improve the results of some
colleges?
Copyrights © for materials are reserved for NCAAA
Group Activity 3
20 minutes
(10 minutes for discussion and 10 minutes to report)
A- Examine Standard 4 and create KPI quality
indicators for it in the connecting circles.
B- Complete the Self Evaluation Scales for a SubStandard under this Standard for your institution
C- Think about the following:
– What evidence would you use to complete the
scales?
– From where would you obtain this evidence?
– Based on your experience, what difficulties do you
expect in collecting and using evidence and KPIs?
Copyrights © for materials are reserved for NCAAA
4.6
Educational Assistance
for Students
Representative
Substandards
4.2
Student
Learning
Outcomes
Standard 4
Learning and
Teaching
4.7
Quality of Teaching
4.3
Program Development
Processes
4.5
Student Assessment
4.2
Student Learning Outcomes
Intended student learning outcomes must be
consistent with the National Qualifications
Framework, and with generally accepted
standards for the field of study concerned,
including requirements for any professions
for which students are being prepared.
Programs must be planned in a way that
ensures that all courses contribute to
program learning outcomes in a coordinated
way.
Copyrights © for materials are reserved for NCAAA
4.2 Student Learning
Outcomes
KPIs
1. Ratio of Students to Teaching
Staff
2. Percentage of students entering
programs who successfully complete the
first year
3. Proportion of graduates from
undergraduate programs who within six
months of graduation are:
• Employed
• Enrolled in Further Study
• Not Seeking Employment or Further
Study
Copyrights © for materials are reserved for NCAAA
How do we use KPIs to determine how
well we perform?
1. Ratio of Students to Teaching
Staff
University 27:1
Benchmark 30:1
2. Percentage of Students entering
Programs who Successfully Complete
the First Year
University: 77%
Benchmark 85%
Copyrights © for materials are reserved for NCAAA
% Student Completion
1st Year
100
University
90
80
70
60
50
40
30
20
10
0
2011
2012
2013
2014
Academic Year
Copyrights © for materials are reserved for NCAAA
End of Day 1
Copyrights © for materials are reserved for NCAAA
PLANNING AND IMPLEMENTATION
OF SELF-STUDY
for Institutional Accreditation
Day 2
Copyrights © for materials are reserved for NCAAA
Day 2
S1
S2
S3
Structure and
Content of the
SSR
Recommendations,
Action Plans and
Conclusions
Applications,
Questions, and
Sharing Experiences
Copyrights © for materials are reserved for NCAAA
MATTERS FOR INCLUSION
IN AN INSTITUTIONAL SSR
•
•
•
•
Institutional Profile
Context – internal and external
Mission – Goals – objectives
Evaluation in relation to Quality
Standards (KPIs+ Benchmarks)
• Conclusions and Recommendations
Copyrights © for materials are reserved for NCAAA
MATTERS FOR INCLUSION
IN SSR
• General Information
• Self study process
• Mission and goals of the
programme
• Key performance indicators [KPIs]
Copyrights © for materials are reserved for NCAAA
CONTENT OF SSR
•
•
•
•
Authorative – evidence based
Evaluative – analyse and reflect
Fit for purpose – required coverage
Constructive – enhancement and good
practice
• Action plan – by whom and when
Copyrights © for materials are reserved for NCAAA
INTERPRETING SELF
EVALUATION SCALES
• Broad evaluation possible?
• Particular importance attached to what student
learnt
• Have Learning Objectives been met?
• Standards of student achievement
• Benchmark against standard
• Can evidence be verified?
Copyrights © for materials are reserved for NCAAA
SSR Template
A. General Information
Name of Institution
Date of Report
B. Institutional Profile
An institutional profile should be prepared including the following material:

A brief summary of the institution’s history, scale and range of activities;

A description of the management and organizational structure using an organizational
chart, list of colleges and departments, and the names and contact details of key
individuals;

A list of campus locations indicating programs offered and student numbers;

Teaching and other staff and student numbers in total and by college, department,
and program;

Summary information about the institution’s accreditation status including the
outcomes of any previous institutional reviews, and any conditions that were
established;

A description of the institution’s quality assurance arrangements, priorities for
development, and any special issues affecting its operations;

A summary of the institution's strategic plan. (A copy of the actual strategic plan
should be available for reference if required.)

A list of matters that are of particular interest to the institution and on which the
institution is seeking comment and advice in the review.
Handbook for quality assurance and accreditation in Saudi Arabia
What is the Profile?
A picture of the University using
data…
Using data to improve what you are doing!
• University Descriptions
• List Accomplishments
• Enrollment Data – statistical trends over
several years, with predictions
Copyrights © for materials are reserved for NCAAA
What is the Profile?
A picture of the Institution using data
Using data to improve what you are
doing!
Student Academic
Achievement and
Performance
Outcomes
Copyrights © for materials are reserved for NCAAA
What is the Profile?
A profile is assessment data….
 Student Achievement Test Scores or licenses
earned…
 Failures/Drop-out rates
 Acceptance rates, profile of students
 Follow-up on alumni or graduates
 Comparative statistics
Satisfaction of University or College Programs
Survey Statistics and Analysis
SSR Template
C. Self-Study Process
Provide a brief description of procedures followed and administrative
arrangements for the self study. Include an organization chart. Membership
and terms of reference for committees and /or working parties should be
attached.
D. Context of the Self Study
1. Environmental Context.
Summary of significant elements of the external environment in which the institution is
operating and changes that have occurred recently or are expected to occur (eg.
economic or social developments, population changes, government policies,
developments at other institutions with implications for this institution’s programs).
2. Institutional Context.
Brief summary of recent developments at the institution with implications for the
review.
Handbook for quality assurance and accreditation in Saudi Arabia
SSR Template
E. Mission, and Goals and Strategic Objectives
for Quality Improvement
1. Mission of the Institution
2. Summary of Strategic Plan for Quality Improvement
( The institution's major goals and strategic objectives for quality improvement should
be listed, indicating for each objective, performance indicators and benchmarks or
standards of achievement the institution wishes to achieve.)
F Special Emphasis in the Self-Study (if any)
Indicate any areas of particular interest to the institution in the review. (These may relate to
responses to changes in the external or institutional environment, to planning priorities that
may have been determined as a result of quality assessments or other strategic priorities, to
government policies, etc.)
SSR Template
G. Progress Towards Major Quality Objectives
(Refer to Item D2 above)
Assessment of institutional performance in relation to plans or any major quality
improvement initiatives in the period under review. These may have been undertaken
in response to a previous self study, recommendations or requirements following an
external review, or for other reasons.
Brief reports should be provided on each major initiative citing the objective(s),
specific data indicating the results achieved, and a comment on reasons for success or
failure to achieve the desired results.
Objective 1
Result Achieved (Performance in relation to indicators and
benchmarks)
Comment
Objective 2
Result Achieved (Performance in relation to indicators and
benchmarks)
Comment
(continue for other strategic quality improvement objectives)
Copyrights © for materials are reserved for NCAAA
Group Activity 4
20 minutes
(10 minutes for discussion and 10 minutes to report)
How you would consider assessing your institutional
performance in relation to plans or major quality
improvement initiatives in the period under review.
What indicators would you use? And Why?
Copyrights © for materials are reserved for NCAAA
SSR Template
H. Evaluation in Relation to Quality Standards
Reports should be given on performance in relation to each of the standards set
out in the Standards for Quality Assurance and Accreditation of Higher
Education Institutions. The reports should refer to areas of strength and
weakness as indicated by the rating scales in the accompanying document—
Self Evaluation Scales for Higher Education Institutions.
1. Standard ……..
(Overall Rating _______Stars)
•Explanatory note about development and use of the mission.
•Description of process for investigation and preparation of the
report on this standard.
•Report on subsections of the standard
•Overall Evaluation of Performance on standard.
Refer to evidence obtained and provide a report based on that evidence and including a
summary of particular strengths, areas requiring improvement, and priorities for action
SSR Template
H. Evaluation in Relation
to Quality Standards
EXAMPLES
Writing Strengths
-Areas for Improvement
- Priorities for Improvement
Copyrights © for materials are reserved for NCAAA
Enrollment
80
70
60
50
Sophomore
Junior
Senior
40
30
20
10
0
2002
2003
2004
2005
Findings:
•3rd enrollment decreases every year
•3rd decline causes future enrollment
problems
•Senior enrollment has steady increase
•Junior enrollment is up and down
Strength:
• Senior enrollment
Weakness:
• Enrollment fluctuation
• Revenue instability
• Class size change
Copyrights © for materials are reserved for NCAAA
Examples of Reports on Progression Rates
Progression Rate from First to Second Year –
(Proportion of all students enrolling in first year in each college who
continued to the second year of their program the following year)
Colleges
Eco & Com
Education
Engineering
Law
Science
Medicine
Nursing
University
2007-8
Male
Female
48.38
55.33
66.3
78.34
41.33
55.33
62.29
57.83
59.22
58.91
61.33
58.22
55.13
48.38
55.33
2008-9
Male
Female
45.8
56.98
62.87
76.44
44.36
58.35
49.35
53.52
56.91
57.34
63.89
56.81
59.16
45.8
56.98
2009-10
Male Female
50.1
52.3
63.98
77.32
42.11
46.1
51.23
55.3
57.26
61.2
66.87
54.35
58.2
50.1
52.3
Questions
Would you consider the proportion of students who complete first year
courses but do not proceed to be a problem? How could you investigate
the reasons for this drop out rate? What improvements are needed?
Copyrights © for materials are reserved for NCAAA
Mean Rating by Students on Satisfaction
with the Quality of Their Courses
(All Students, 2007/8 t0 2009/10)
Colleges
Eco & Com
Education
Engineering
Law
Science
Nursing
University
2007-8
Sem 1
Sem 2
3.03
3.29
3.32
3.11
3.70
3.75
4.12
3.89
3.01
2.98
3.62
3.78
3.55
3.48
2008-9
Sem 1
Sem 2
2.98
3.25
3.20
3.29
3.68
3.69
4.00
3.80
3.16
3.22
3.77
4.01
3.42
3.56
2009-10
Sem 1
Sem 2
3.13
3.33
3.09
3.22
3.68
3.65
3.96
4.02
3.45
3.49
3.99
4.02
3.61
3.68
Questions
1.Which college do you believe has the most satisfying results?
2.What action need to be taken to improve the ratings by students? What
target would you set for improvement?
Copyrights © for materials are reserved for NCAAA
Group Activity 5
15 minutes
(10 minutes for discussion and 5 minutes to report)
Complete your Self Evaluation Scales by:
- Identifying KPIs and evidence used,
- Areas of Strengths
- Priorities for improvement
Copyrights © for materials are reserved for NCAAA
Day 2
S1
S2
S3
Structure and
Content of the SSR
Recommendati
ons, Action
Plans and
Conclusions
Applications,
Questions, and
Sharing Experiences
Copyrights © for materials are reserved for NCAAA
Writing Recommendations
• Write Priorities
• Write Recommendations
• Write Interventions
Recommendations Are…
A recommendation is written sentence
with a challenging verb that directs
tangible, measurable, or quantifiable
action that is directed toward
improvement.
Verbs with clear reference to
measurable action
SECTION I SSR
INDEPENDENT EVALUATION
• Outside Institution has experience
• Verify Evaluation Accuracy and results
interpretations based on evidence
• Comment on relative standards
• Critical friend
• Summarise matters raised
• Opportunity to respond
I Independent Evaluations
1. Describe the process used to obtain independent comment on the self
study.
Processes may include a review of documentation by experienced and independent
persons familiar with similar institutions and who could comment on relative
standards, consultancy advice or a report by a review panel, or even the results of
an accreditation review by an independent agency. An independent evaluation may
be conducted in relation to the total self-study, or involve a number of separate
comments by different people on different issues.
2. Summary of matters raised by independent evaluator(s)
3. Comment on matters raised by independent evaluator(s)
(Agree, disagree, further consideration required, action proposed, etc.)
Developmental Review
Mock Review
Copyrights © for materials are reserved for NCAAA
J Conclusions
1. List and briefly describe institutional activities that are
particularly successful or that demonstrate high quality.
2. List and briefly describe institutional activities that are less than
satisfactory and that need to be improved.
K Action Recommendations
These should be based on the matters identified earlier in the report for further
improvements or to overcome problems or weaknesses identified indicate specific actions
proposed to deal with the most important priorities for action identified in those sections.
Matters of greatest urgency or highest priority should be identified.
For each action proposed recommendations should be made on who should be responsible
for the action, timelines specified, and any necessary resources required.
Action Recommendation 1 …..
Person (s) responsible
Timelines (For total initiative and for major stages of development)
Resources Required
CONCLUSIONS
Base SSR preparation on Annual Monitoring
Reports
Use for Quality Assurance and Quality
Improvement
–
–
–
–
–
AUTHORATIVE – evidence based
EVALUATIVE – analyse and reflect
FIT FOR PURPOSE –coverage
CONSTRUCTIVE - good practice and improvement
ACTION PLAN – responsibilities and time lines
Copyrights © for materials are reserved for NCAAA
RACI
R - Responsible – One who owns the project
A - Accountable -To whom R is responsible Must sign
off on the work before it becomes effective, veto
(yes/no) power
To be Consulted - Has information/or capacity
necessary to complete the work
To be Informed - Must be notified of the result but not
consulted
R A C I -Example
Activity
Learning Resources
Facilities and Equipment
Learning and Teaching
Student Administration
and Support Services
Faculty and Staff
Employment Processes
Faculty Member
Department Chair
Vice Rector
Rector
R
A
C
I
R
?
?
?
C
R
A
I
?
R
?
?
?
?
R
A
Copyrights © for materials are reserved for NCAAA
ACTION PLANS GENERAL
•
•
•
•
Rolling or fixed date
Tabular
Assign responsibility
Base on reality – recorded issue leading to
achievable solution
S – Specific
M – Measurable
A – Achievable
R – Responsible – Result oriented
T – Time lines
Copyrights © for materials are reserved for NCAAA
ACTION PLAN SPECIFIC
•
•
•
•
Sign off Action Plan
Changes
Staff development needs
Seize initiative
LINK ALL ACTION POINTS TO
ISSUES RAISED IN ANNUAL REVIEWs
Copyrights © for materials are reserved for NCAAA
ACTION PLAN EXAMPLE
Action
Date
Responsibility Resources
1.
2.
3.
Evaluate if completed
Link to summaries in SES
Copyrights © for materials are reserved for NCAAA
Documentation and Attachments
The Institutional Self Study Report should be on A4 paper,
unbound, printed on one side, page numbered, and with a table of
contents for easy reference.
Attachments
Membership and terms of reference for sub-committees and
working parties
Reference list of key reports and other documents cited in the
report
Glossary of acronyms and abbreviations used in the report
Copy of report(s) by independent evaluator(s)
Copyrights © for materials are reserved for NCAAA
Documentation and Attachments
with SSR
Self–evaluation Scales for Higher Education Institutions.
The completed scales should include star ratings, independent
comments and indications of priorities for improvement as
requested in the document, and should be accompanied by a
description of the processes used in investigating and making
evaluations.
A copy of the institution's strategic plan.
A copy of the institution's strategic plan for quality
improvement (which may be included within the broader
institutional strategic plan).
Current student catalogue, prospectus, bulletin or handbook
including descriptions of the curriculum, admissions requirements,
degree completion requirements, and related information.
Copyrights © for materials are reserved for NCAAA
•
•
•
•
•
•
Documentation to be available in
the review visit
Faculty handbook or similar document with information about staffing policies,
professional development policies and procedures and related information.
Administrative and financial policies manual or similar document including the
institution’s bylaws and regulations, roles and responsibilities of administrative and
academic officers and major committees, and an explanation of the institutions
governance and administrative structure.
Quality assurance manual or description of procedures including information about
the institutions system of assessing programs and services, the role of the
institution’s quality center and systems for gathering and analyzing data on quality of
performance and planning for improvement.
Current data on faculty and other teaching staff including tables with numbers by
academic rank, by highest qualification, teaching staff/student ratios for each
department and college, and for the institution as a whole.
For a university (optional for a college) information should be provided on
research output for each department, college and for the institution as a whole.
Current teaching staff CVs should be on file and available for the review panel if
required.
Copyrights © for materials are reserved for NCAAA
Group Activity 6
30 minutes
(15 minutes for discussion and writing
and 15 minutes to report)
Use the Ratings of the Scales and available documents
and evidence to write an SSR report on Standard 4
Standard 4 Teaching and Learning
(Overall Rating _______Stars)
1-Explanatory note about development and use of the mission.
2- Description of process for investigation and preparation of the
report on this standard.
3-Report on subsections of the standard
4-Overall Evaluation of Performance on standard including:
- summary of particular strengths,
areas requiring improvement,
- and priorities for action
Copyrights © for materials are reserved for NCAAA
Day 2
S1
S2
S3
Structure and
Content of the SSR
Recommendations,
Action Plans and
Conclusions
Applications,
Questions, and
Sharing Experiences
Copyrights © for materials are reserved for NCAAA
Continue Group
Activity 6
Copyrights © for materials are reserved for NCAAA
Group Activity 7
15 minutes
(10 minutes for discussion and 5 minutes to report)
Using your previous report, draw up an
Action Plan which will lead to
improvement in your institution regarding
standard 4.
Actions
Date
Responsibility
1.
2.
3.
Copyrights © for materials are reserved for NCAAA
Resources
Common
Challenges and Problems
Success Factors
in Conducting Institutional SSR
Copyrights © for materials are reserved for NCAAA
End
Thank you
Copyrights © for materials are reserved for NCAAA