Overview of ODRC’s ORAS Quality Assurance Program
What is Quality Assurance?
ORAS File Review
Development of the QA Program
Results of the QA Program
Recommendations

A planned and systematic pattern of actions
necessary to provide adequate confidence
that the product optimally fulfills customer
expectations.
Proactive
Improve
Confidence
Credibility



Create a culture of quality
Good fidelity = replication of results
Mission “reduce recidivism”






FAQ
Flow charts
Internal audit standards
Quality Assurance Reports
Case file reviews
Quality Assurance Program
Keep these 12 things clean for good
assessments




Duplicates
Sources used
Verifying collateral information
Notes




Pilot site evaluations
Benchmarking
Literature Review
University of Cincinnati


Goal = accurate assessments
What contributes to the goal?
◦ Fidelity to the assessment process
◦ Good interview skills

How will the QA program be delivered?
◦ Create measures/tools
◦ Timelines



Implement the QA program
Analyze and share results
Evaluate program and make adjustments.






QA Coordinator
Random selection of staff
Scheduling
Communication
Work flow
Selection of Evaluators

ORAS Direct Observation Tool

Performance Measurement Guide





Pre-brief meeting
Direct Observation of the interview
Individual Scoring of the tool
Debrief and Coaching
Results to the QA Coordinator





Goals
Program process
Results
End users
All levels of management




Subject matter experts/Certified ORAS
Trainers
IPC skills
Recommendation from their immediate
supervisor.
Completion of evaluator training.







Direct Observation Tool
Performance Measurement Guide
Double coding
General process
Common rating errors
Providing good feedback that is balanced
Giving and receiving criticism

Survey Evaluation
◦ Evaluators (19)
◦ Assessors (116)
◦ Do survey results demonstrate gaps? Perceived
value verses real value.

Share results

Value of the pre-brief meeting
◦ 65% of the assessors and 59% of the evaluators rated it
successful or very successful.

Concentrated areas of coaching
◦ Assessors and evaluators rated using the interview guide
and using collateral information highest.

Did assessors learn anything new?
◦ 61% of the assessors and 90% of the evaluators
responded yes.

Ability of the evaluators to add value
◦ 82% of the assessors rated the evaluators with the ability
or strong ability to add value.

How important are accurate ORAS assessments?
◦ 65% of the assessors rated it as important or very
important.

What are the barriers to completing accurate
ORAS assessments?
◦ The length of time it tales to complete the interview
and input the information into the portal. Also the
program runs slow.
◦ Clients not being fully truthful.
◦ Not having an investigation in the file.
◦ Time consuming when you have a lot of other
things to do.

What was most valuable about the ORAS QA
process?
◦ Getting reassurance that I was doing some things
right.
◦ The tool has potential if done accurately.
◦ Helped understand my struggles were being felt by
others also being evaluated.
◦ One on one discussion and feedback.
◦ Don’t think we need this, to concerned about QA
and not the time consuming ORAS tool.

How would you improve the ORAS QA
process?
◦ It was fairly upfront and open. The evaluator did a
good job with me. Maybe assess everyone at some
point.
◦ I think the ORAS QA process can be very effective, if
administered by someone that has actually
completed an ORAS assessment of some kind.
◦ Have the assessor sit in on more than one
assessment.
◦ Figure out a way across the state to bring all
officers to the same page period. Good Luck!
Interview
Skills

Reliability


Direct Observation
Tool
Double Coding
Survey
Survey
Trends
Domains 6 and 7
Interview Skills








Resources
Subject Matter Experts
Have a QA Coordinator
Explain QA process to the offender
Scoring Guide
Have follow up plan ready to implement
Staff support – supervisors need to be more
knowledgeable.
Sustainability







Interview training
Booster trainings focused on weak areas
and/or changes
RIB rules card
Supervisor staffing requirements
Individual Improvement Plan
Consider assessor models
Continue QA efforts statewide with all user
groups










Goals
Concerns/barriers for staff
Resources
Process flow
Direct Observation
Training
Communication
Implementation
Evaluation
Follow up – use results to impact positive change
Proactive
Improve
Confidence
Credibility
Deborah Herubin
Deborah.Herubin@odrc.state.oh.us
614-995-0181
John Geras
John.Geras@odrc.state.oh.us