Peer Reviewer Training
Workshop
Shea Wang & Trudi Ohki
CAFÉ
Sept. 22, 2014
Workshop Overview
Upon completion of this workshop, you
will be able to:
• Conduct a pre-observation meeting
• Review teaching materials
• Use the narrative, checklist, rating, and data
collection forms for classroom observation
• Conduct a post-observation debrief
• Write a final post-observation report
Activities Overview
•
•
•
•
Practise using the pre-observation meeting form
Practise using the review of teaching materials form
Practise classroom observation skills and techniques
Practise using the narrative, checklist, rating, and data
collection forms for classroom observation
• Practise post-observation debrief
• Practice post-observation report writing
Faculty Evaluation Policies:
Instructional Faculty – C5065
Non-Instructional Faculty –
C5066
Evaluation at MacEwan
Policy C5065 guided by the philosophy that
we:
•
•
•
•
•
•
promote a culture of teaching and learning
foster professional development and scholarly activity
promote fairness and transparency
timely and accurate feedback
comply with collective agreements
undertake as a process involving multiple stakeholders, and a
variety of assessment approaches
Evaluation at MacEwan
Faculty strive for excellence in teaching, which is
assessed by:
•
•
•
•
teaching (including curriculum development)
research, scholarly, or artistic engagement
service (collegiate citizenship)
collegiality (professionalism, respect for students & peers, and
ethical behaviour)
• professional development
C5065 – Instructional Faculty
Requirements based on appointment type
Continuing
Teaching Dossier
Peer Review
Probationary
Sessional/Sessionalextended
√
√
√
Sessional-extended
at renewal
(Collective
Agreement 2011-14)
Annual Report
√
√
Student Online Feedback
√
√
Performance Review
√
√
√
C5065 – Probationary Requirements
Teaching Dossier - Annually
Peer Review - Two reviews during each of the first two years (4 total)
Annual Report - Annually
Student Online Feedback - Every course during appointment
Performance Review - Annually
Introduction
What is Peer Review?
• Scholarly publications – review process to assess quality
of the work
• Evaluation of classroom/online teaching and teaching
materials
• ASAC position review committees
• Any evaluation process where the evaluators are peers
Peer Review Literature
Peer review is appropriate for…
•
•
•
•
•
Course content, expertise in field of study
Course objectives and materials
Appropriate teaching methodology
Course organization
Appropriate tests, assignments, evaluation methods
Seldin, 1999
Peer Review at MacEwan
• Teaching effectiveness
– Classroom visits
– Teaching materials
• ASAC position review committees
Qualifications of Peer Reviewers
• Criteria developed by FEC
• Completion of an appropriate peer review training
course… and / or demonstration of peer review
experience
• Minimum 3 years full-time teaching experience
• Pedagogical expertise
• Possess a continuing appointment
• Recommendation by Dean / Associate Dean / Director /
senior administrator
Attributes of Peer Reviewers
• Utilizes effective instructional practices (demonstrated
through dossiers, annual reports, awards, etc)
• Demonstrates collegiality
• Provides constructive, objective feedback focusing on
teaching behaviours and improvement
• Models ethical behaviour
• Appreciates different styles / modes of teaching
Exceptions…
• For all peer reviewers, should there be a deficit in
meeting the recommended qualifications, the Dean,
Associate Dean, Director, or senior administrator will
assess and provide final approval.
Choosing Peer Reviewers
• Chairs will support and advise faculty members in the
selection of two peer reviewers to conduct classroom
observations and review of relevant teaching
materials.
• It is not necessary that peer reviewers be contentarea experts.
• Peer reviewers can expect to be asked to perform
peer reviews outside of their home department(s).
Defining Effective Teaching
Conversations about teaching (30min)
• Pair and Share
Break into small groups and discuss the following
sentences:
I most enjoyed teaching when __________
I knew I had a problem when __________
•
Have a member of your group report on your
discussion to the main group
*Chickering and Gamson, 1987
Measuring Good Teaching
Myths
•
•
•
•
•
•
Dr. Fox…
Expressiveness…
High Ratings Reflect Lenient Grading
Workload/course difficulty…
Rank and experience…
Gender…
https://facultycommons.macewan.ca/headlines/professorsfrom-sfu-share-mean-student-evaluations
Measuring Good Teaching
Measuring in context
• Take 15 min. and complete the following
sentences in your group:
– I knew my teacher was a good teacher
when/because…
– I knew my teacher was a poor teacher
when/because…
Literature on “Good” Teaching
Eight dimensions of college instruction :
• Course organization & planning
• Faculty communication
• Faculty/student interaction
• Assignments, exams, & grading
• Instructional methods & materials
• Course outcomes
• Student effort & involvement
• Course difficulty, workload, & pace
(Marsh, 1984 & 1987)
Literature on “Good” Teaching (2)
•
•
•
•
•
•
•
Encourages student interaction
Encourages student cooperation
Active learning
Prompt feedback to students
Emphasizes time on task
High expectations
Respect
(Chickering and Gamson, 1987)
Literature on “Good” Teaching (3)
Good teachers…
• Are good learners;
• Display enthusiasm;
• Know how to modify their teaching strategies;
• Encourage learning for understanding;
• Demonstrate an ability to transform and extend
knowledge, rather than just transmit it;
• Set clear goals;
• Provide high-quality feedback
• Show respect for their students
(Ramsden, 2003)
Overall… “Good” Teaching
• Parker Palmer – “capricious chemistry”
– Good teaching is ultimately irreducible to a simple
technical formula
– Characteristics are beyond common descriptors or
attributes of a checklist
• Solution: multifaceted approach
– Student feedback, peer review, self-reflection, and
administrative evaluation
Peer Review Activity
In your small group, reflect on your past
experiences with giving and receiving
peer feedback. Address the questions:
•
•
•
•
What did you learn from the experience?
In what ways it was a positive/negative experience?
Did it influence your instruction?
How did receiving and giving feedback change the
way you thought about teaching and learning?
Peer Observation
The two forms of observation:
• Summative observation - This observation is an
evaluation. It can be done for administrative or
supervisory reports for the purpose of job retention,
promotion, salary increases, etc.
• Formative observation - The purpose of a
formative peer observation is to help instructors
become better teachers and more knowledgeable
professionals.
Peer Review Process at MacEwan
Evaluation Policy outlines two main areas of
teaching assessed by peer reviewers:
1. Teaching materials - lecture notes, course outlines,
tests and exams, assignments, grading rubrics, etc.;
and
2. Instructional delivery - observation of teaching in
formal lectures, as well as other delivery modes
such as labs, studio, clinical, and online/distance
formats
Peer Review Process at MacEwan
Five main elements:
•
•
•
•
•
Pre-observation meeting
Review of teaching materials
Classroom observation
Post-observation debrief
Post-observation report
Forms
Some of the forms available…
•
•
•
•
Narrative form
Checklist form
Rating form
Data Collection
(see appendices of workbook)
Pre-Observation Meeting
• “Emotionally, it is important to address distrust of
evaluation, violation of personal style and space,
insecurities about time, and fear of bias” (Chism, 2007)
• Essential that trust be established - Acknowledge the
emotional aspect.
• Clear objectives of observation should be outlined Communicate expectations
• Stressed that the process is collaborative - Engage in
active listening skills
(Review pre-observation forms in appendices)
Course Materials
Some examples of materials include:
•
•
•
•
Syllabus
Course guides
Course packets and assigned texts
PowerPoint Presentations , overheads, BlackBoard,
etc
• Handouts
• Videos and other multimedia supplements
• Project assignment directions and handouts
Procedure for Evaluation of Course
Materials
• Importantly, we are not focusing on student achievement
measures when reviewing teaching materials. Rather,
we are looking at content, design, clarity, etc
• Materials enable reviewers to see an instructor's
philosophy enacted, and can alert reviewers to the areas
in which a given instructor excels
• Materials can promote discussion about the overall goals
(Review course materials checklist in appendices)
Classroom Observation
• Observers should use the same observation form – they
should take separate minutes/notes
• At least two observers should attend each class
observation session
• At least two visits is preferable
• No unannounced visits
• Discuss with the instructor the format of the review prior
to visit: Where will you sit? Will you be introduced?
Classroom Observation
During the Classroom Visit…
• Be unobtrusive (basically, remain quiet)
– Do not ask questions
– Do not volunteer answers
– Do not correct the instructor
• Be attentive
• Note specific examples of observed behaviours
(positive & negative)
Classroom Observation Challenges
• One class may be not representative
• Atypical performance (good or bad)
• Observer may not have a good
understanding of the context
• Personal biases on what constitutes “good
teaching”
Some Solutions
•
•
•
•
•
•
Attend 2+ classes
Video tape class?
Focus-group students after class?
Use the data collection form, cite examples
Discuss visit with co-reviewer, debrief together
Contextualize the course – reflect back on:
– Pre-observation meeting
– Examination of course materials
Peer Observation Exercise
Pre-observation…
• In your small group, decide on the form you will be using for
review (narrative, checklist, rating)
• Review form and highlight areas that each reviewer will be
responsible for focusing on and taking notes about
Observation…
• Choose one of the two scenarios to review
• Reflect on the class from the following viewpoints:
– Viewpoint 1: students
– Viewpoint 2: instructor
Peer Observation Exercise
• Compare your list of observer behaviors with others in
your group
• Be reflective and systematic through the criteria outlined
in your observation form
• Embed your observations & conclusions in examples of
observed behaviours
– Did the instructor’s behavior appear to be what you
might expect in a formal observation?
Peer Observation Exercise
Role Play
• In your small group, designate an individual to be the
instructor, the reviewer, and a reporter
• Break into large groups based on role as “instructors”
& “reviewers” & “reporters”
• Compare the notes your groups took during the
observation and discuss strategies for approaching
the debrief (and how the characters might react)
• Break back into your small group and role play the
post-observation debrief meeting
Post Observation Debrief
• Meet with the instructors immediately following the
classroom visit
• Ask the instructor to describe how they felt the class
went in relation to the goals, objectives and
expectations outlined in the pre-observation meeting
• At this time, the peer reviewers should bring up any
immediate concerns that were highlighted by the
observation session.
**This is an important aspect of the review as it can
potentially change the observers’ perceptions about
what they saw and heard.
Post Observation Debrief
Procedure
• Ask the person who taught how s/he felt the lesson went
• Reconfirm the observation focus points for the final report
• Offer positive comments on what you think went well
• Identify any urgent issues
• Ask questions for clarification, as needed
• End on a courteous note
• Provide an expected timeline for the final report and final
meeting
Post-Observation Exercise
• Prepare yourself to care – watch that you are not overly
forgiving or overly critical (put yourself in their shoes)
• Use descriptive language rather than evaluative
• Be responsive to the goals outlined in the preobservation meeting
• Check to make sure the instructor has heard your
message as you intended
• Be forward-looking and positive in your discussion, i.e.
what can be improved over what went wrong
Peer Review Final Report
Policy C5065 states…
• Section 4.4.4, “Peer reviewers shall prepare brief written
assessments, grounded in evidence, and submit a copy
of reports to the faculty members and Chairs.”
Who Sees the Final Report?
• Faculty member – used to improve teaching
effectiveness, make adjustments as necessary for
application for continuing status
• Chair and Dean – used for administrative review
• ASACs – used for personnel decisions
Characteristics of a Good Report
• Reference the goals established during the preobservation meeting
• Provide a written record of what was observed and
what was discussed
– Consider the audience
– Use examples from the observation
• Language is clear, concise, avoids jargon and
abbreviations
• Tone is supportive and phrasing is positive
• Action-oriented and provides specific and practical
suggestions for improvement
Characteristics (cont’d)
• One, joint report per instructor - Structure of report
should be agreed upon by all reviewers
• Return report in a timely manner
– Review with faculty member within 3 days of
classroom visit
• Consider faculty member response before submitting for
administrative review
• Confidential
(Review report template in appendices)
Peer Review for Online Courses
• Online Peer Review Form (4 categories)
– Student behaviour
– Faculty-student interaction (multiple
interactive communication strategies)
– Technology support
– Learning environment
Challenges - Peer Review for
Online Courses
A connection between specific materials and
instruction and student action that manifest
learning objectives.
Technology and accessibility – Are
instructors responsible for these? Yes, no,
maybe, more nowadays?
What Is A Good Online Course?
• Content? Easy use of technology? Interaction?
• Learner-centered outcomes-based instruction:
– Clear objectives as stated in:
– Syllabus: must be clear. Super important.
– Grading Rubrics: explains how they will be graded
Cont’d.
What Is A Good Online Course?
• Content? Easy use of technology? Interaction?
• Learner-centered outcomes-based instruction
– Clear objectives as stated in:
– Models or Examples of quality work from successfully submitted
and graded student work
– Explanations of the relevance of course materials to learning.
– WHY are we doing this? (think of students perspective).
Cont’d.
What Is A Good Online Course?
• Content? Easy use of technology? Interaction?
• Learner-centered outcomes-based instruction
– Assessment tools should be used:
– Throughout the course to establish:
– Learning styles
– Skill/knowledge level
– Challenge preconceptions
– Clarify current learning
Cont’d.
What Is A Good Online Course?
• Content? Easy use of technology? Interaction?
• Learner-centered outcomes-based instruction
– Opportunities for students to construct and experience
their own knowledge:
– We all learn differently and it’s important to be clear and
understanding about this.
– Learning processes and outcomes are not all the same.
Cont’d.
What Is A Good Online Course?
• Content? Easy use of technology? Interaction?
• Learner-centered outcomes-based instruction
– Opportunities for students to become increasingly
responsible for their own learning:
– Instructor should provide consistent and timely input.
– Encouragement, good examples, and ‘instructor presence’
– Try to develop intrinsic motivation.
Cont’d.
What Is A Good Online Course?
• Content? Easy use of technology? Interaction?
• Learner-centered outcomes-based instruction
– Options for demonstrating learning outcomes (if
appropriate)
– Opportunities for reflection
– Effective student support services
Cont’d.
What Is A Good Online Course?
• Clear and captivating instructional
design
– the writing style, the images and plug-ins
selected, and the layout are appropriate to the
intellectual sophistication, technical savvy and
needs of the audience
– a pathway to guide student learning
– appealing presentation
Cont’d.
What Is A Good Online Course?
• Varied forms of interactivity with material, other
students and the instructor
– Regular Communication with peers and instructor
– Varied Interaction with Content
Source:
http://www.4faculty.org/includes/digdeeper/online/lc_outcome
s_based_instruc.htm
Online Course Material for Review
1. Instructional material and resources
2. Assignments and student assessment (clear and
detailed instructions and guidelines, purposes, formal
writing standards/academic writing requirement)
3. Discussion forums (ability for students to give
feedback and ask questions during the class is
critical)
4. E-mail messages
5. Course syllabus (the syllabus is the most important
document of the entire course. It is the road map or
‘game plan’ for the course)
6. Chat space
Where to Go For Resources
• CAFÉ website:
– Then click on “Faculty Evaluation”
– Then click on “Peer Review”
• Faculty Evaluation Coordinator