Evaluator Workshop
Fall Visits 2011
Tuesday, May 24, 2011
Please join the audio portion of this training:
866-740-1260, Access Code: 7489001
ReadyTalk Help Desk: 800.843.9166
International Help: 303.209.1600
1
Announcements
This presentation and the accompanying materials are
available for download from:
http://www.wascsenior.org/fall2011
For assistance with Voice and Web connections please contact:
ReadyTalk Help Desk, 800.843.9166
International Help: 303.209.1600
Please mute your microphone if you are not speaking.
If you have questions, please enter them into the Chat window.
2
WASC Evaluator Workshop
Fall Visits 2011
3
Workshop Outcomes
•
•
•
•
Know how to prepare for and conduct an
effective visit
Be prepared to produce a useful, highquality team report
Be ready to make sound judgments about
institutions under the Standards
Be familiar with resources that support
your work on a team
4
Agenda
•
•
•
•
•
Context for the Visit/Accreditation
Preparing for the Visit
Conducting the Visit
Developing Team Recommendations
Writing the Team Report
5
Context for Accreditation
and Visits
• The continuing evolution of the WASC
process and Standards
• The accountability movement
– Moving from assessment for improvement to
assessment for accountability
– New areas of emphasis
• The Handbook revision process
6
Recent Changes in the
Institutional Review Process
and Standards
• Changes to Institutional Review Process re: Student
Success, Program Review and EE Sustainability
• Changes to CFRs
• Clarifying the scope of the CPR visit to review the
“infrastructure” for assessment of student learning
• Program Review and Program-Level Student Learning in
a systematic way
Tool: Table A & B
(EVG pg. 74 & 80); SVG pg. 68 & 74)
7
Covering the Impact of the Financial
Recession on Institutions
Questions to ask the institution:
• How has the financial recession affected your
institution?
• How has your institution responded?
• What plans are in place in case the current state
of affairs becomes permanent?
8
Q&A
• Please feel free to type in your questions
using the chat window or just chime in.
9
The Three-Stage
Review Process
1.
Institutional
Proposal
2.
Capacity &
Preparatory
Review
3.
Educational
Effectiveness
Review
10
The Three-Stage Review Process
1. Institutional Proposal
2. Capacity and Preparatory Review
3. Educational Effectiveness Review
11
Timeline for Three-Stage
Review Process
Proposal
Extended
period of
time to
sustain
Initiatives
(7-10 years)
Submitted 2
Years before
CPR review
Capacity And
Preparatory
Review
Educational
Effectiveness
Review
18-24
months to
prepare for
EER
(or less for
Candidacy or
Initial Accreditation)
12
Institutional Self-Review
• The heart of accreditation
• Built upon an effective internal process of
– Evaluation
– Self-reflection
– Recommendations from previous reviews
– Plans for action
13
Stage 1:
The Institutional
Proposal
14
The Institutional Proposal
• Guides the entire accreditation review process
• Connects institution’s context and priorities with
the Standards of Accreditation
• Provides primary basis for both institution selfreview and team evaluation
• Allows alignment of accreditation activities to
institutional strategic plan and key areas chosen
for improvement
15
The Letter of Intent
• Submitted by institutions seeking
Candidacy or Initial Accreditation, the LOI
serves the same purpose as the proposal
• Includes suggestions from Eligibility
approval letter
• Submitted to assigned WASC liaison, one
year in advance of CPR Review
• Instructions are in “How to Become
Accredited” on WASC website
16
Stage 2:
The Capacity and
Preparatory Review
17
Purpose of the CPR
• Review and verify the information in the
institutional presentation (report and data)
• Evaluate key institutional resources, structures,
processes in light of Standards
• Evaluate institution’s infrastructure to support
and assess student learning
• Assess institution’s preparedness to undertake
the Educational Effectiveness Review
18
Stage 3:
The Educational
Effectiveness Review
19
Purpose of the EER
• Invite sustained engagement by the
institution on the extent to which it fulfills
its educational objectives
• Enable the Commission to make a
judgment about extent to which institution
fills its Core Commitment to Educational
Effectiveness
20
Q&A
• Please feel free to type in your questions
using the chat window or just chime in.
21
The Two Core
Commitments
22
Core Commitment 1
“The institution functions with
clear purposes, high levels of
institutional integrity, fiscal
stability, and organizational
structures to fulfill its
purposes.”
23
Core Commitment 2
“The institution evidences clear and
appropriate educational objectives and
design at the institutional and program
level. The institution employs processes of
review, including the collection and use of
data, that ensure delivery of program and
learner accomplishments at a level of
performance appropriate for the degree or
certificate awarded.”
24
The Four Standards
Tool:
Standards of Accreditation (EVG pg. 51, SVG pg. 47)
Standards at a Glance (EVG pg. 68, SVG pg. 63)
25
Standard 1:
Defining Institutional Purposes and
Ensuring Educational Objectives
• Institutional Purposes
• Integrity
26
Standard 2:
Achieving Educational Objectives
Through Core Functions
• Teaching and Learning
• Scholarship and Creative Activity
• Support for Student Learning
27
Standard 3:
Developing and Applying Resources
and Organizational Structures to
Ensure Sustainability
• Faculty and Staff
• Fiscal, Physical, Information
Resources
• Organizational Structures & Decision
Making Processes
28
Standard 4:
Creating an Organization Committed to
Learning and Improvement
• Strategic Thinking and Planning
• Commitment to Learning and
Improvement
29
Expectations for Two Reviews
Capacity and Preparatory
Educational Effectiveness
• Preparatory = readiness for • Demonstrating student
learning
the Educational
Effectiveness Review
• Demonstrating institutional
• Capacity = purposes,
learning
integrity, stability,
• Demonstrating evidenceresources, structures,
based decision-making
policies, processes
Tool: Expectations for Two Reviews
(EVG pg. 45, SVG pg. 42)
30
Q&A
• Please feel free to type in your questions
using the chat window or just chime in.
31
Preparing for the Visit
32
Roles and Responsibilities of
Team Members and Staff
•
•
•
•
Role of Team Chair
Role of Team Assistant Chair
Role of assigned WASC staff liaison
Team assignments
Tool: Section 10 (Tips, Roles and Advice)
33
Timeline For CPR/EER Reviews
12 weeks
Institution
mails report
to team and
WASC
Team holds
conference
call
2 months
Site visit held
and team report
written
Institution
responds to
errors of fact in
team report
Institution
responds to
final team
report
Commission
acts at
February or
June meeting
34
Pre-visit Preparation
• Read all the documents from WASC
– Standards, CFRs, policies, visit guide, rubrics
– Background documents re: institution and purpose of
the visit, including Proposal and/or last action
letter/team report
• Read the institutional report
• Review the data portfolio and exhibits
– What to look for and highlight?
35
Reviewing the Exhibits
• Enrollment data
– Headcounts and FTE
•
•
•
•
•
Graduation data
Faculty data
Key financial indicators
Inventory of Educational Effectiveness Indicators
Inventory of Concurrent Accreditation and Key
Performance Indicators
Tool: How to Review WASC Data Exhibits (EVG pg. 107, SVG pg. 96)
36
Reading the Report
• Has the institution done what it said it would
do in its Proposal?
• Has it collected and analyzed data effectively?
• Are its conclusions supported by evidence?
• Are there serious problems or potential areas
of noncompliance?
• Does the report contain recommendations for
further institutional action?
37
Worksheet for Team Pre-Visit
Conference Call
• Organizes team’s responses to institutional
materials
• Helps team make preliminary evaluation under
the Standards
• Provides basis for team to work toward
consensus
• Should be submitted in advance of call
Tool:
Team Conference Call Worksheet (EVG pg. 94; SVG pg. 88)
38
Team Pre-Visit Conference Call
• Evaluates quality of institutional report and
alignment with Proposal and previous action
letter(s)
• Identifies areas of good practice, improvement,
and further inquiry
• Identifies issues, strategies, evidence needed
• Identifies persons and entities to be interviewed
• Makes or refines team assignments
• Plans visit logistics
39
Off-Campus Sites and
Distance Education Programs
(special requirement for some visits)
Prior to Visit: Sites/online programs will be identified and assignments made
•
•
Review substantive change action letters to determine if issues have been
identified
Develop plan for the review of the programs and/or sites
During Visit
•
•
•
•
•
Interview faculty, administrators and students
Evaluate facilities OR online infrastructure
Observe classes (can be done ahead for online)
Document visit and findings in the appendix
Discuss important findings with team for inclusion in report, as appropriate
Tools: Distance Ed. Summary (EVG pg. 216, SVG pg. 157)
Off-Campus Site Summary (EVG pg. 205, SVG pg. 148)
40
Compliance Audit
(special requirement for some visits)
• Required for:
– Institutions seeking Candidacy and Initial
Accreditation
– Some institutions under sanction
• Additional report submitted by institution in
advance of the visit—with links to documents
• Prepare as appendix to report
Tool: Compliance Audit Checklist (EVG pg. 103, SVG pg. 93)
41
Determining Strategy for CPR Visit
• What evidence is provided to show capacity and
readiness for EE?
• What are the strengths and weaknesses of the
evidence?
• What other evidence do you want to review to
evaluate capacity and preparation for EE?
• Do any issues arise with regard to the
Standards?
• Meetings: format/methodologies
42
Determining Strategy for EER Visit
• What evidence is provided to show EE?
• Why was it chosen?
• What are the strengths and weaknesses of the
evidence?
• What other evidence do you want to see to
evaluate effectiveness?
• Do any issues arise with regard to the
Standards?
• Meetings: format/methodologies
43
Drafting in Advance of the Visit
• Assistant Chairs draft outline of team report and
Section I
• Team members draft outline or text for which
they are responsible, using institution’s report
and data portfolio, with space for additional
evidence, analysis and conclusions
Tool: Guidelines for Drafting Preliminary Report Narratives Prior to the
Visit (EVG pg. 102, SVG pg. 92)
44
Q&A
• Please feel free to type in your questions
using the chat window or just chime in.
45
Conducting the Visit
46
Launching the Visit:
Team Executive Session
•
•
•
•
•
•
•
•
Discuss preliminary findings
Identify major issues for exploration
Refine lines of inquiry for each meeting
Confirm team assignments
Discuss use of tools and rubrics
Review preliminary outline/draft team report
Discuss options for confidential team recommendation
Review schedule
47
Visit Schedule
• Executive sessions and debriefings with
team only
• Meetings and interviews with key
individuals and groups
• Open meetings with students, faculty and
staff
• Document review
• Time for drafting report sections
• Final exit meeting
48
Confidential Email Account
• Set up by WASC as extension of open meetings
• Checked by Assistant Chair during visit
• Important emails shared with team and
investigated
• Comments included in team report only if the
institution has a chance to address them
Tool: Sample Notification re: Confidential Email Account
(EVG pg. 179, SVG pg. 123)
49
Approaches Used on Visits
• Document review
• Interviews and meetings
– Mini-questionnaires
– Techniques for small and large meetings
– Fishbowl exercises
• Audits
Plan visit methodologies in advance
as part of schedule.
50
Tips for Good Interviews
• Decide on a protocol for interview
• Prepare questions and lines of inquiry in
advance
• Ask questions that elicit information, stimulate
discussion, or require judgment
• Avoid interrogation, leading questions, or
loaded language
• Avoid consultation, giving solutions, or talking
about your institution
• Let them do the talking
51
Addressing Student Success
• All CPR and EER reports should address this
topic
• Teams should designate someone to study and
write about student success
• Discussion should include summary of data,
identification of issues, and plans for
improvement
52
Evaluating Program Review
and Student Learning
on EER Visits
Tool:
EE Toolkit (available online)
Suggested Approaches for Evaluating Program Review (EVG pg. 184)
53
EER Toolkit
54
55
56
57
Rubrics: Assessment of Student Learning
1. Academic Program Learning Outcomes
2. Use of Portfolios in Assessing Program
Outcomes
3. Use of Capstones in Assessing Program
Outcomes
4. Integration of Student Learning
Assessment into Program Review
5. General Education Assessment Process
Tool: Rubrics for Evaluating the Effectiveness of
Assessment Practices (EVG pg. 190)
58
Educational Effectiveness
Framework
• Use with team to evaluate institution’s “place”
• Use language of rubric to describe the institution in
the report
• Ask the institution to evaluate itself and discuss
• Confer with team toward end of visit to mark a copy
of the EEF
• Submit the marked EEF confidentially to WASC
Tool: Educational Effectiveness Framework (EVG pg. 188)
59
The Exit Meeting
• Team chair communicates commendations and
key recommendations that will be included in
report
• Chair may ask team members to participate
• The meeting is not a dialog, discussion or
debate
60
Q&A
• Please feel free to type in your questions
using the chat window or just chime in.
61
Developing Team
Recommendations
62
Two Kinds of Recommendations
• Team recommendations at the end of team
report, delivered at the exit meeting
• Confidential Team Recommendation to the
Commission for action
Tools:
Commission Decisions on Institutions
(EVG pg. 245, SVG pg. 177)
Commission and Team Decision Indicators
(EVG pg. 255, SVG pg. 187)
63
Team Report Recommendations
Should be:
• Overarching and important
• Supported by evidence
• Linked clearly to Standards and CFRs
• Supported by text in the report
- Distinguish recommendations from suggestions and
observations embedded in the report
Tool: Educational Effectiveness Framework
(EVG pg. 188)
64
Producing Effective
Team Reports
65
Report Preparation Logistics
• Follow report template
• Start writing before the visit
• Complete your sections on site and give to
Assistant Chair for editing together
Tool: Section 7 (Producing Effective Team Reports)
66
Using Evidence in Team Reports
•
•
•
•
•
•
Use qualitative and quantitative evidence
Select evidence carefully and purposefully
Connect evidence to an assertion or question
Analyze information; do not just set forth data
Let evidence suggest improvements
Use evidence that speaks to the institution’s
themes and the team's questions
67
Team Use of the Standards
and CFRs
• Team judgments must be linked to specific
Standards and CFRs
• CFRs must be cited in reports
• Standards and CFRs form the basis for
Commission decisions
• Standards and CFRs provide a context for
continuous quality improvement
68
What is an effective team report?
• Reflects a thorough assessment of the
institution’s capacity, preparation, and/or
effectiveness
• Is evidence based
• Cites the Standards and CFRs
• Provides the basis for a sound and supportable
Commission decision
• Identifies important areas for institution to
address
69
Tips for Writing Team Reports
• Consider multiple audiences: institution,
Commission, and next team
• Know your areas of responsibility, including length
and depth of your section
• Start writing before you arrive on campus
• Address priorities and goals set by the institution
• Address Commission’s concerns (last action letter)
• Make commendations, but don’t overdo it
• Use praise that doesn’t send wrong or mixed signal
70
More Tips on Team Reports….
• Be sure to check facts
• Support findings and recommendations with
evidence --and tie them to CFRs
• Ensure evidence is sound and valid
• Distinguish recommendations from suggestions
or observations
• Use formal language and tone (e.g., not
“we/they”)
• Don’t mention personnel by name
• Don’t prescribe solutions
71
Q&A
• Please feel free to type in your questions
using the chat window or just chime in.
72
After the Visit
73
What happens next?
• Assistant Chair prepares draft for Chair, team
and staff review; changes as needed
• Chair sends to institution for corrections
of fact
• Chair finalizes draft and submits to WASC
• Chair sends Confidential Team
Recommendation and completed EEF
to WASC
• WASC sends report to institution
74
Then…
• Staff prepares draft action letter, which is reviewed
by team Chair
• Commission Panel reads report and
documentation including institution’s written
response, meets with institutional representatives
at Commission meeting
• Panel makes recommendation to Commission,
and Commission acts
• Staff finalizes draft action letter on behalf of
Commission
75
Also after the visit….
• Team members send reimbursement forms to WASC
within 30 days
– Hotel arranged and paid directly by institution
– Travel / food reimbursed
– Rental car must be approved in advance by
WASC staff
– Spouse or assistant costs not covered
– See policy for more details
• Team members should not have any contact with the
institution
– About the visit OR
– Consult with the institution for one year
76
The Team’s Impact
• Peer review is the foundation of accreditation.
• The team report forms the basis for the
Commission action and its letter.
• The team report and action letter inform the
work of the institution for years to come.
• Why were you chosen for a team?
77
Resources for Teams
• Appendices of Visit Guide
• Team Materials and Institutional Report mailed
10-12 weeks in advance of visit
• WASC Website: www.wascsenior.org
• WASC Email Advisory (sent prior to visit)
• WASC Staff
78
Q&A
• Please feel free to type in your questions
using the chat window or just chime in.
79
Thank you
for your service to the
region
80
Announcements
The materials presented during this webinar and a
recording of this session will be posted at:
http://www.wascsenior.org/fall2011
81