ALT-C short paper #0114: Peer observation for distance learning e-tutors
Peer observation for distance learning e-tutors: evaluation of a cross-programme trial scheme
Richard Walker & Kevin Hall
University of York
Abstract
This short paper discusses the methodology and outcomes of a trial peer observation programme for distance
learning e-tutors which was introduced in the spring and summer terms (2011) across three online distancelearning programmes at the University of York. The trial aimed to scope out a framework for peer observation,
which could be used to support the professional development of tutors who are engaged in the supervision of
online learning.
The trial was designed to support tutors in developing their skills through a process of mutual exchange
between observer and observee, with the observee selecting the focus and materials for the observation as a
way of establishing ownership for this process. As a departure though from established peer observation
models (e.g. COOLAID), participants who opted in to the trial were assigned to random pairings in the role of
observer and observee, bringing colleagues together from different disciplinary backgrounds. This was
intentional in exposing participants to different pedagogic perspectives on tutoring skills. Whilst guidance and
assessment criteria for the conduct of the observation and delivery of feedback were made available at the
outset of the trial, tutors were encouraged to develop their own strategies for completing the observation.
They were also free to determine whether the observation would be a reciprocal or one-way process.
Ten tutors from the three participating distance learning programmes completed peer observations, working
as 5 separate pairings. Tutors from 4 of the 5 pairings performed both roles as ‘observer’ and ‘observee’ in
reciprocal observations. Evaluation of the trial was based on structured interviews with tutors from each
pairing at the end of the observation, coupled with procedural evidence collected during the course of the
trial; e.g. blog postings on participants’ expectations for the trial; recorded objectives for each observation.
Notwithstanding the small sample and self-selecting nature of the tutors who engaged in the trial, the
preliminary findings indicate that the randomised pairings helped participants to focus on tutoring techniques
and approaches, rather than teaching content and key messages in their exchanges, without the distraction of
discussing disciplinary matters. The blind matching process encouraged them to seek out common ground in
their pedagogic objectives and approaches to online tutoring, although the differences in course contexts
frustrated some participants at the outset in making sense of the tutoring approaches and student activities
under observation.
The absence of prescriptive methods for the conduct of the observation resulted in pairs developing their own
working methods to negotiate their ‘contract’, observe their partner and deliver feedback on the online
practice, with evidence of alignment in the selection of objectives for each pairing. This was most noticeable in
the two pairings which met face-to-face to negotiate their contract. Whilst all pairings reported positive
learning outcomes from the observation process, i.e. tutoring techniques and ideas which could be applied to
their professional practice, the results suggest that a relationship may exist between the choice of working
methods and quality of outcomes in the observation process. Pairings which engaged in face-to-face
negotiation and discussion appeared to develop greater levels of trust, which enabled them to probe tutoring
practice in a more critical way, exploring areas of perceived weakness as well as tackling emergent themes,
moving beyond the agreed objectives. Pairings which worked at a distance (online) tended to adhere more
closely to the contract and focus on predetermined themes with objective comments, with limited evidence of
critical feedback in their exchanges. This suggests a challenge for the further development of this trial and the
conduct of peer observation at a distance in supporting observers and observees to develop relationships of
1
ALT-C short paper #0114: Peer observation for distance learning e-tutors
trust through online interaction, enabling them to draw the maximum benefits from peer observation to
inform their own professional practice. Further research is required with a larger sample of participants to
verify these findings. As a next step, the University’s Distance Learning Forum will reflect on the outcomes
from the trial with a view to running an extended programme of peer observation for distance learning etutors over the next academic year (2011-12).
1. Background
In recent years there has been growing interest in applying peer observation techniques to online
teaching and tutoring activities. Indeed a variety of methods have been developed across the HE
sector to support peer review of this kind, with a primary focus on support for distance learning
delivery. The ‘Peer-to-peer Reflection on Pedagogical Practice’ (PROPP) model developed at
University College London is one such example, providing a framework for individual tutors to come
together to discuss a problem, issue or a project from a course and use this material as a basis for
reflection on course design and delivery methods (Swinglehurst, 2006). Last year the universities of
Hull and Staffordshire showcased their collaborative model of peer observation of online practice for
online tutors (COOLAID), presenting their findings in an evidence-based practice seminar sponsored
by the Higher Education Academy (Bennett et al., 2010).
In both contexts, peer observation has been conceived as a way of meeting the development needs
of teaching staff, providing a framework for participants to develop their skills and competencies in
online teaching and support activities, drawing on a community of practice. This marks a departure
from traditional peer observation approaches for classroom teaching which have focused on quality
improvements and the evaluation of teaching performance, promoting instead a learning
opportunity for both observer and observee in sharing practice and approaches to online teaching.
1.1 Aims and rationale
This paper reports on the outcomes of a trial peer observation programme for distance learning etutors which was conducted over the spring and summer terms (2010 - 2011) across three online
distance-learning programmes at the University of York. The trial aimed to scope out a framework
for peer observation, which could be used to support the professional development of distance
learning tutors and their skills development.
In the spirit of the COOLAID model, peer observation was presented to participants in this trial as a
development technique for the individuals involved, rather than as a quality assurance mechanism
or way of monitoring performance. It was intended to assist tutors in developing their professional
practice as e-tutors through the sharing of impartial and objective feedback, based upon actual
observations of practice in order to:

enable the person being observed to better understand their behaviours and actions whilst
tutoring and so make informed decisions about how to behave and act differently in order to
become a better teacher;
2
ALT-C short paper #0114: Peer observation for distance learning e-tutors


enable the observer to objectively describe and discuss the behaviours and actions of other
tutors and to be informed by these observations. This will include the identification and
sharing of good or innovative practices; and
enable diverse and innovative tutoring methods and practices, including use of technology,
to be opened up to external friendly scrutiny and discussion, in order to ensure that good
and innovative practices are shared.
This initiative was organised by the University’s Distance Learning Forum and represented a standalone trial, unrelated to the University’s ‘Peer Support for Teaching’ (PST) programme which was
launched across the University at the same time. The University’s PST programme is geared towards
quality improvements to teaching at a departmental level, placing a strong focus on group initiatives
based upon module, team teaching or collaborative arrangements addressing issues such as
curriculum design, assessment and feedback. Peer observation, in contrast, was conceived as a
vehicle for professional development, with the observee owning the process in order to have
confidence in it, hence self-selecting the materials to be observed, the timeframe of the observation
and the aims of the observation are central to the peer observation process.
1.2 York’s approach to peer observation: methodology
Building on the insights from other institutional experiences with peer observation (e.g. COOLAID Collaborative Observation Online; PROPP - Peer-to- peer Reflection on Pedagogical Practice), the
trial was designed to support tutors in developing their skills through a process of ‘mutual exchange’
between observer and observee (McMahon et al., 2007), with the observee selecting the focus and
materials for the observation as a way of establishing ownership for this process.
As a departure though from established peer observation frameworks, participants who opted in to
the York trial were assigned to random pairings in the role of observer and observee, bringing
colleagues together from different disciplinary backgrounds with contrasting levels of tutoring
experiences. This was intentional in exposing participants to different pedagogic perspectives on
tutoring skills, as well as unfamiliar programme contexts and technologies (learning platforms), in
this way extending the opportunities for mutual learning within the pairings.
3
ALT-C short paper #0114: Peer observation for distance learning e-tutors
Table 1: Shared and contrasting features between COOLAID and York approaches to online peer
observation
Description of York trial
Shared features with COOLAID model
Features specific to York
Choice over:
 Participation
 Focus of observation
 Form of feedback
 Information flow
 Future action
Choice over:
 role (observer / observee)
No choice over:
 Partner (observer and/or observee)
Random selection – based on matching
participants from different programmes
& subject / disciplinary backgrounds
Tutors were invited to ‘opt in’ to the peer observation programme, with an online consultation
phase introduced over a one-week period prior to its launch (14th – 21st Feb 2011), which aimed to:
help tutors understand the aims of the trial; get their input on the delivery of the trial – i.e. how the
observation process would run; and foster a sense of ownership over the peer observation process.
General guidance notes were made available to participants at the outset of this process on:



the principles of peer observation;
approaches to defining the focus for an observation;
tips on the delivery of feedback to observees.
Participants were encouraged to comment on the guidance notes and the proposed pro forma
document, which were adapted from the COOLAID documentation presented at the HEA evidencebased seminar. (See the Appendix for a copy of the pro forma and guidance information which were
used by participants.) The documentation was intended to help participants record their
observation arrangements (objectives & observation aims; arrangements for
observation/information to provide to observer & tips on delivery of feedback), with individuals
requested to return the first section of the form to the programme coordinator, outlining the agreed
aims of the observation.
The process was intentionally non-directive, with participants free to determine how and when they
would conduct the peer observation – whether this would be a reciprocal arrangement (participants
experiencing both roles as observer and observee) or one-way arrangement. The only requirement
was for participants to assist with the evaluation of the trial before the end of the summer term
(2010-11), by engaging in a reflective interview on their experiences with the coordinator upon
completion of the process.
4
ALT-C short paper #0114: Peer observation for distance learning e-tutors
1.3 Profile of participating tutors
Ten tutors took part in the trial and they were drawn from three distance learning programmes at
the University of York, which reflected contrasting approaches to online tutoring and course
delivery.
Table 2: Breakdown of participants by distance learning programme
Distance Learning Programme
MA Public Policy & Management
MA TEYL
Certificate in Creative Writing (Lifelong Learning)
# of tutors completing observation
5
2
3
The MA programme in Public Policy and Management (PPM) is a fully online postgraduate
programme hosted on a Moodle VLE platform, which focuses on asynchronous group discussion as
its teaching model. The MA programme Teaching English to Young Learners (TEYL) is hosted on the
University’s Blackboard VLE, presenting its online course space as a resource area and optional
discussion space for the duration of the programme, with online tutoring restricted to the
preliminary phase of the course. The Certificate in Creative Writing (Lifelong Learning) is also hosted
on the University’s VLE, but is focused on 1:1 interaction between tutors and students with tutors
overseeing individual writing tasks, and although accredited is part of the University’s ‘learning for
pleasure’ portfolio of courses.
Consequently tutoring roles and pedagogic approaches varied a great deal between the participating
tutors, with different perspectives on expectations for student engagement online and group
management for discussion activities. Differences were also evident in the levels of tutoring
experience of participants, ranging from tutors with 8 – 9 years experience to one year or less. One
of the tutors had only taught 2 terms at York for the online programme, but had face-to-face
teaching experience to draw on. A common characteristic shared by all though was their selfconfidence and willingness to share practice with peers outside their programme. As one participant
noted in discussing her rationale for involvement in the trial:
“I am confident about my teaching style but not clear if this is working online –I wanted to
find out.” (Lifelong Learning tutor)
Another of the less experienced tutors commented on the desire to “improve my abilities”:
“I was not concerned about exposing myself – maybe yes if this had been in my first term’s
teaching – but I was confident enough to do it”. (PPM tutor)
The fact that the trial scheme was non-compulsory reassured some participants and helped them to
focus on it as a staff development opportunity, rather than a competency measurement:
“I felt in control of the process and quite relaxed. I would have reacted differently if told that
I had to do it with a random observer.” (PPM tutor)
Perhaps more significantly, features of the peer observation process were already established as
part of working practice across some of the distance learning programmes. PPM and MA TEYL tutors
5
ALT-C short paper #0114: Peer observation for distance learning e-tutors
were already familiar with peer observation as a staff development tool, having engaged with
colleagues on their programme in the observation of their tutoring. It was viewed as a supporting
mechanism. The opportunity to work with someone outside the programme offered fresh
perspectives as well as learning opportunities on how another distance learning programme works.
“I was excited by working with an external. We have done our own peer observation
internally with all following the same approach – this is an opportunity for another pair of
eyes although it was slightly nerve wracking.” (PPM tutor)
1.4 Evaluation aims & methods
The purpose of the trial was to inform thinking at the University on the development of a framework
for peer observation for distance learning tutors, which could be rolled out across the institution.
The research effort focused on the conditions for effective peer observations, with attention drawn
to the impact of the blind matching process to see which combinations of pairings worked and which
did not, as well as the range of contextual information that should be shared prior to the
observation in order to support an effective exchange between participants.
Our research approach focused on participants’ perceptions of their experiences, establishing
‘meaning’ from the standpoint of the actors, rather than through objective measurements of
performance or levels of engagement in the trial. Evaluation of the trial was therefore based on
interpretive methods through structured interviews with observers and observees after the
completion of each observation. The interviews explored the process of peer observation, focusing
on the blind matching process and the development of the informal contract between participants,
which determined the focus and criteria for the observation. The research also explored the richness
of the dialogue between participants and the opportunities for learning which emerged from this
process – specifically the actions arising from the observation exchange which contribute to the
professional development of e-tutors.
We also drew on procedural evidence collected during the course of the trial to support the research
and cross reference with the interview transcripts: e.g. blog postings on participants’ expectations
for the trial during the online consultation phase; objectives for each observation, recorded in
section 1 of the pro forma, which was returned to the programme coordinator by each pairing.
2. Findings
Of a total of 7 pairs of tutors identified at the outset of the trial, five pairs went on to complete the
observation process. 4 out of the 5 pairs conducted reciprocal observations, with participants
experiencing both roles as ‘observer’ and ‘observee’. The other pair intends to follow a similar
approach and will complete a reciprocal arrangement with a follow-up observation in September
2011.
Only two of the pairings were able to meet face-to-face, with the other pairings involving tutors
located overseas, working across different time zones.
6
ALT-C short paper #0114: Peer observation for distance learning e-tutors
Observations tended to focus on discussion areas and course materials for archived courses, with
only two tutors opting for ‘live’ courses to reflect on their tutoring practice.
2.1 Blind matching process
Participants were matched up by the trial coordinator in a random fashion, with the only criterion
for the matching of tutors focusing on the selection of tutors from different programmes.
Consequently participants were not given an opportunity to express a preference for a partner
based on level of experience, responsibility, role or teaching discipline.
Feedback from the exit interviews with participants revealed that the blind matching approach was
positively received by participants. The trial met the baseline expectations of tutors in offering a
partner with an external perspective on tutoring practice, thereby offering a contrast to programmespecific activities such as shadowing and mentoring. For tutors working overseas the trial offered a
greater connection to the University and opportunity to share practice. For the majority of tutors,
any form of matching was welcome as long as the procedure was straightforward to adopt – with no
strong preferences expressed over discipline and level of experience.
“I was not worried about the matching. I was hoping to get something from any discipline –
there will be something good in it” (PPM tutor)
“Blind matching was a really good idea as I had no idea about other programmes and didn’t
know anyone else from the other courses. It’s important that the peer observation process is
straightforward – not complicated to organise and do. Working with someone who is
assigned to you is easier to do. It worked fine for me.” (MA TEYL tutor)
The opportunity to work with someone from a different discipline and pedagogic outlook was seen
as beneficial – ensuring that the focus of the observation was on teaching practice rather than
teaching content. The involvement of an external perspective also promised a more objective
review of tutoring practices, with no preconceptions about the programme or individual’s tutoring
style:
“It brought objectivity to the process. My partner didn’t know me or my tutoring style.” (PPM
tutor)
“It allows a certain level of objectivity. You can stand back outside of your course without any
link with the set-up and comment.” (MA TEYL tutor)
Pedagogic values and levels of tutoring experience were not viewed as critical factors in determining
an effective ‘match’.
“Sharing the same pedagogic approach can shut down opportunities to learn. With blind
matching you can get a nugget from watching anyone.” (LLL tutor)
“It’s not too much of an issue whether your partner has a little or a lot of tutoring
experience.”
7
ALT-C short paper #0114: Peer observation for distance learning e-tutors
Participants noted that the natural inclination for many would be to choose someone from a similar
discipline, but this would restrict perspectives on pedagogy, terminology and teaching practice. The
key was to move away from the specifics of course to focus on the teaching, which would enrich the
opportunities for learning.
“The major objection to blind matching is that you can’t comment on content and specifics you can’t comment on the real content of courses. But the purpose is on how teaching and
tutoring are done. It’s a good idea to get a mix of academic backgrounds as you can then
hope to avoid discussing details (of a course). The internal peer observation discussion starts
on tutoring but soon shifts to the details – the content and not tutoring style.” (PPM tutor)
“You can get bogged down in your discipline – you should be focusing on teaching practice
rather than teaching content.” “..by stripping out the content I could focus on the
observation issues” (MA TEYL tutor)
Participants acknowledged though that to make the blind pairing work, they needed to seek out
common ground between them to establish an effective working relationship and rapport, which led
one participant to argue that is more important to match around personality – to ensure a
compatible working relationship – than to match around discipline or pedagogic outlook.
The effectiveness of the observation also depended upon clarifying the course context up front teasing out the information that the observer requires to make sense of the online activities under
review. This encompasses the ‘rules of engagement’ for the course: i.e. whether participation in
online activities is a requirement for the course; what’s visible online in terms of learning and
tutoring support provision and what’s happening offline – e.g. telephone contact with students. The
gaps in knowledge and assumptions about tutoring styles needed to be addressed up front or
pursued through clarifying questions as the observation progresses:
“..that aspect of not knowing the course – you have to ask a lot of questions about the
background of the course. For one course there was a lot of email / telephone traffic
between the student and tutor which was not apparent through simply observing the online
discussion. There were therefore gaps in our knowledge when reviewing the online
discussion. But if we repeated the process we would have that knowledge. Could do with that
knowledge (context of course) upfront.” (PPM tutor)
However, one participant noted that some observation issues were specific to role and level of
responsibility and might influence the matching process.
“Blind matching could be organised at the level of people. Level of responsibility is an
important factor.” (MA TEYL tutor)
Another participant highlighted the need for contingency arrangements if a matching did not work
out – a mechanism to change the pairs without anyone being offended if it is not working out.
8
ALT-C short paper #0114: Peer observation for distance learning e-tutors
2.2 Contract negotiation: observation focus and objectives
No steer was given to participants on the selection of objectives for the observation. However, a
number of shared concerns emerged across the peer discussions (see Table 3 below). The most
common objectives focused on discussion management responsibilities – how to open and close
discussions, encourage participation and contribute effectively to discussions – and these concerns
were shared across all 3 programmes. Tutors from the PPM and Creative Writing (Lifelong Learning)
programmes also focused on activity design (structure of tasks) and make-up of course materials.
Table 3: Breakdown of objectives for peer observation – presented by programme:
Theme
Frequency of
Total
objectives identified
by participants
PPM
MA
LLL
tutors TEYL tutors
(n=5) tutors (n=2)
(n=3)
Theme 1: Discussion management
Discussion management & organisation:
 Encouraging student participation
 Rounding off discussions
 Responding when issues arise
4
0
1
5
Methods for supporting student learning
2
2
0
4
2
1
1
4
1
1
0
2
Course content
 Building engaging materials
 Presentation & content of course
1
0
2
3
Activity design
 Balance of tasks (individual vs. group) and use of
media
 Expectations made of students
2
0
2
4

Balance of feedback to individuals & group
Contribution of posts to discussion
 Value of tutor’s input to discussion
 Tone & delivery of feedback
 Length / style of tutor’s posts
 Balance between teaching & facilitation
Methods for stimulating group interaction
 Stimulating student – student interaction
Theme 2: Course design
9
ALT-C short paper #0114: Peer observation for distance learning e-tutors
Theme 3: Tutor’s skills (general)
Time management
 Managing delivery of feedback to students
0
0
1
1
Technology usage
 Alternative tools / delivery modes
2
0
0
2
Technology help & guidance
 Layout of blog / forum discussion space
0
0
1
1
1
2
0
3
Theme 4: Programme management
Supporting associate tutors
 Supporting & monitoring associate tutors
 Quality assurance procedures
An analysis of observation objectives by pairings suggests a level of common ground and shared
interest across each pairing – however this may be a product of the contract negotiation between
individuals seeking common ground, rather than ‘luck of the draw’ in the blind matching of
compatible partners. However, greater alignment - i.e. evidence of shared objectives - is discernible
in those pairs that met at the outset to negotiate the focus of their respective observations. One
pair arrived at shared objectives relating to discussion management and the presentation of course
materials to students, which they agreed on at an initial meeting to discuss the focus of the
observation. The meeting helped them to seek out the common ground between them, as described
by the following tutor:
“It worked because we got on. Subconsciously we were looking for common ground. We
reflected on our teaching styles – when we speak to each we use the same language – share
the same world view.” (Lifelong Learning tutor)
2.3 Peer observation process and working models
Typically the process of peer observation involves participants tackling three phases of engagement:
1. Preparation: scene setting; establishing roles, discussing objectives for the peer observation
and expectations on what will be observe and by when – recorded in a ‘contract’)
2. Observation: review of tutoring practice and recording of notes in line with ‘contract’
3. Discussion: delivery of feedback to observee; reflection and discussion; points for future in
professional development.
For the trial programme, no steer was provided as to how these phases of engagement should be
handled and by when. The only requirement was to complete the observation before the end of the
summer term (2011) and to feedback experiences through an evaluation interview.
10
ALT-C short paper #0114: Peer observation for distance learning e-tutors
Feedback from the interviews revealed a great deal of variation in how pairs tackled the phases of
engagement, with 3 working models emerging (see Figure 1 below). Variables such as the
geographical location of tutors (location and time zones) and time constraints (the amount of time
that could be devoted to the peer observation process) – appeared to determine the way that tutors
worked.
For example, the two pairings which were able to meet in person (Model 1) used an initial meeting
to define the scope of the observation and to surface a range of shared concerns, which they then
worked up into objectives for the observation - of which there was a close overlap between
partners. A follow-up face-to-face meeting provided an opportunity to discuss observation findings
and explore their significance for tutoring practice, before a summary report was exchanged. This
reflects a collaborative and conversational approach to the discussion of aims and findings, whereas
the others pairings (Models 2 & 3) focused on individual selection of objectives, with communication
aimed at clarifying the issues that had been identified. Models 2 and 3 differed in the way that
feedback was delivered, with Model 2 pairings discussing outcomes from the observation after the
exchange of reports, whereas the Model 3 pairing simply exchanged feedback reports.
Model
Model 1: f2f
negotiation with oral
discussion of
feedback
Preparation
Observation & Discussion
online observation
contract by f2f meeting
forms exchanged after
meeting
feedback through f2f
meeting
reports exchanged after
meeting
online observation
Model 2: negotiation
at a distance with
oral & written
feedback (interactive)
personal reflection on
aims
clarification by email
discussion of feedback
by skype
Model 3: negotiation
at a distance with
written feedback
(asynchronous)
personal reflection on
aims
online observation
forms exchanged
delivery of written
report to partner
forms exchanged
clarification by email
delivery of notes to
partner
Figure 1: Emergent working models for conduct of peer observation process
11
ALT-C short paper #0114: Peer observation for distance learning e-tutors
2.4 Conduct of the observation and feedback processes
There was general agreement amongst participants that the observation process should go beyond
the agreed aims – i.e. that the process of observing a course presents a legitimate opportunity for
personal reflection on one’s own course as well as for wider discussion of the observation aims
under review. For some the observation aims restricted the scope of the comments that they
wanted to discuss with their partner:
“Through the observation I was constantly thinking about my own practice. It was hard to
stay on track when making notes on the aims. The observation aims are important but I
wanted to comment about a lot of other things as well which were picked up in the phone
conversation. My comments didn’t fit with the aims. I was looking for things to say to meet
the aims, when I could say more, if given the freedom.” (MA TEYL tutor)
For others the observation focus was the starting point for a more general discussion of pedagogic
practices and an opportunity to expose differences in pedagogic approach – language and semantics.
This appeared to be related to the depth of interpersonal rapport and relationship between
observer and observee.
“There will be more learning points if there is closer pedagogic proximity between observer
and observee.” (PPM tutor)
As Shortland (2007) and other researchers have shown for face-to-face peer observations, trust is
the key to an effective relationship, underpinning exchanges between observer and observee. The
evidence from this trial suggests that this is equally true for observations conducted online. If this
level of relationship is not in place, the opportunities for mutual learning may be restricted. Evidence
from the triallists suggests that the ensuing feedback discussion may tend to follow more of a preset script around the observation aims, without addressing ‘emergent themes’.
To mitigate the effects of physical distance and to better inform communication between partners,
participants (Models 2 & 3) remarked that a greater level of upfront information about the course
context and tutoring modes under review would have assisted the observer to make sense of online
discussion activities – e.g. how students interact; how the tutor is aiming to interact with them;
expectations of students - what they should do and by what time.
“X’s students have no obligation to go in and post to a certain depth on their site. Without
that knowledge, recommendations on student engagement by observer would have been
very different.” (PPM tutor)
The language used in the delivery of feedback and references to course terminology could also have
been clarified to establish a common understanding. Gaps in understanding can become apparent
when observing a course or discussing feedback with a partner and go unnoticed at the contract
negotiation phase. Consequently participants needed to tease out hidden assumptions at the
beginning of the process.
In the absence of interpersonal understanding, some participants noted a tendency to revert to
‘neutral’ feedback, sticking closely to the pre-set objectives for the observation, without negotiating
challenging or new areas:
12
ALT-C short paper #0114: Peer observation for distance learning e-tutors
“It would have been easier to deliver feedback if a relationship had been established. Instead
I gave neutral feedback, as you would to students, as you don’t know if you will be
misunderstood…I would have felt reluctant to give negative feedback – say negative things in
a direct manner. I would have hidden comments in a more positive statement.” (PPM tutor)
In contrast, a pair of tutors who met face-to-face during the contract negotiation and delivery of
feedback phases (Model 1) reported that they were able to probe areas of weakness, drawing on
the relationship and rapport that they had built up. The observer homed in on a suspected area of
weakness that the observee had wanted to be reviewed as part of the observation process:
“We were able to negotiate criticism – it boils down to personality and trust. You need to be
undefended in receiving criticism. It was easier to negotiate face-to-face. If we had done this
online, we could have been constructively critical, but it would have been a different
experience.” (Lifelong Learning tutor)
Delivery of critical feedback was conveyed through questions or suggestions for alternative modes
(alternative ways of doing things – “have you thought about?”; “on our course we did this…”), but
participants noted that care was needed in arriving at critical remarks, given the risk of false
assumptions based on incomplete information about the course that they were observing.
Observers needed to be aware of what they couldn’t see – i.e. the levels of student-tutor interaction
which were taking place outside the observable learning space by telephone or email.
3. Outcomes and discussion
In drawing conclusions from this study, we must acknowledge the limitations of this trial and the
research evidence that has been generated, which is based on a small sample of self-selecting
participants. The findings, although preliminary in nature, do though support the view that online
peer observation can contribute important learning benefits for e-tutors and serve as a vehicle for
professional development. Learning benefits were recorded by all participants across the trial
programme - irrespective of the working model employed, with all pairs reporting learning points
and beneficial outcomes from the process. The blind matching may indeed have contributed to this
process, by encouraging participants to seek out common ground in their pedagogic approaches and
tutoring techniques, rather than concentrate on teaching content and disciplinary issues.
The results show that learning was not restricted to the predetermined focus & agreed aims, but
was also experienced incidentally through general observation of differences in programmes
delivery and ways of doing things. The process of observation (commenting on another course) was
as valuable and rich in learning benefits (through personal reflection on one’s own practice) as the
output and comments received as an observee.
The key benefits reported by participants may be categorised as follows:

confirmation & reassurance on tutoring practice (what’s working well)
e.g.:
“confirmed that my tutoring style is OK –posts are not too long; student are getting detailed
feedback”
13
ALT-C short paper #0114: Peer observation for distance learning e-tutors
“the balance of style and substance in the course was confirmed”

fresh perspectives on current practice (challenging assumptions; surfacing issues &
weaknesses in current practice; doing things differently and making improvements to
current practice)
e.g. “don’t be scared to set boundaries and ground rules for discussion, signalling the
beginning and ending of discussions”
“introduce video ‘talking heads’ for tutor input to make the course site more engaging create more of a personal stamp on the course..”

comparative viewing of platforms / technologies
e.g. “I learned about technology such as ‘adaptive release’ and the differences between
Moodle and Blackboard”

‘side benefits’: unintended benefits – transferable learning approaches (dealing with
overseas students)
e.g. “helped me to think about how we work with overseas students – useful advice – stepby-step on preparation for writing at Masters level”

‘association with a community of practice’ –networking with other tutors
e.g. “it was very nice professionally to feel part of a group of online tutors -I had not felt that
before, nor felt before that I was part of the University”
However, it is worth noting that the scope and range of learning opportunities were not consistent
across the pairings – different pairings received different learning experiences. In this sense our
findings suggest that a relationship may exist between the choice of working methods and quality of
outcomes in the observation process. Pairings which engaged in face-to-face negotiation and
discussion appeared to develop greater levels of trust, which enabled them to probe tutoring
practice in a more critical way, exploring areas of perceived weakness as well as tackling emergent
themes, moving beyond the agreed objectives. The delivery of critical feedback and attention to
sensitive areas was reported as being easier to negotiate ‘in person’, where a rapport and shared
understanding had been established.
Pairings which worked at a distance (online) tended to adhere more closely to the contract and focus
on predetermined themes with objective comments, with limited evidence of critical feedback in
their exchanges.
Feedback from participants pointed to the contribution that a face-to-face relationship could make
to generate a level of trust between participants for online observation and a common
understanding of the purpose of the peer observation process, ensuring that it is tailored more to
individual needs. Participants remarked that an initial face-to-face meeting can help to set a
common purpose and the tone for the observation process to follow.
14
ALT-C short paper #0114: Peer observation for distance learning e-tutors
This suggests a challenge for the further development of this trial and the conduct of peer
observation at a distance in supporting observers and observees to develop relationships of trust
through online interaction, enabling them to draw the maximum benefits from peer observation to
inform their own professional practice. Further research is required with a larger sample of
participants to verify these findings. As a next step, the University’s Distance Learning Forum will
reflect on the outcomes from the trial with a view to running an extended programme of peer
observation for distance learning e-tutors over the next academic year (2011-12).
References
COOLAID – Collaborative Observation in the On-Line Environment for enhancement across
Institutional Divides (2010). Briefing Paper by: Shirley Bennett, Sue Lee, Patrick Lynch and Lyz
Howard, University of Hull (December 2010). HEA Evidence Net. Available at:
http://www.heacademy.ac.uk/assets/documents/learningandtech/University_of_Hull_Briefing_Rep
ort_26_May_2010.pdf
McMahon, T., Barrett, T. and O’Neill, G. (2007) Using observation of teaching to improve quality:
finding your way through the muddle of competing conceptions, confusion of practice and mutually
exclusive intentions. Teaching in Higher Education 12:4, (pp.499–511).
Shortland, S. (2007) ‘Participation, justice and trust within developmental peer observation of
teaching: a model and research agenda’, The International Journal of Management Education, Vol 6
No 1, pp27-37. Available at:
http://www.heacademy.ac.uk/assets/bmaf/documents/publications/IJME/Vol61/Shortland_Peerobs
.pdf
Swinglehurst, D. (2006) Peer Observation of Teaching in the Online Environment: an action
research approach. Final Report. Centre for Distance Education. Available at:
http://www.cde.london.ac.uk/support/awards/generic2346.htm
Address for correspondence
Richard Walker, Kevin Hall
University of York, Heslington, York, N. Yorks. YO10 5DD
richard.walker@york.ac.uk
kevin.hall@york.ac.uk
15
ALT-C short paper #0114: Peer observation for distance learning e-tutors
APPENDIX
Peer Observation across Distance Learning Programmes (trial)
Section 1a: About You
Your name, department/centre and contact details:
Your course programme and VLE platform:
Your role in relation to this programme, e.g. tutor; programme leader, etc.:
When do you want to be observed (delete as appropriate)?
During which term? Spring/summer/autumn?
During a live/current interactive session? Y/N
During a closed/past interactive session? Y/N
Section 1b: What would you like feedback from the observation to focus on?
Use these questions to guide your thoughts about what you want to achieve by being peer
observed online, then list up to three aims:
 Is there an aspect of your own teaching / support role that you would like to change or
develop?
 What are your goals in your use of technology to support learning? Are they currently being
achieved?
 What aspects of learning, use of technology, use of materials or use of online activities
would you like feedback on?
 Are there particular behaviours within your role as tutor that you would like the observer to
feedback on?
Observation aims

Aim 1

Aim 2

Aim 3
What will the observer need to look at?
E.g. which module/activity/VLE tool (s)/group(s)/
pathway or resource material etc.
Return this form to the trial coordinator at this point (trial only, to facilitate pairing and evaluation)
16
ALT-C short paper #0114: Peer observation for distance learning e-tutors
Section 2a: Observation and feedback arrangements - Observer’s name
Your name, department/centre and contact details:
Section 2b: How will the observation be conducted and feedback shared?
To be agreed directly between you and your observer.

When will the observation take place
(approximate start and finish dates)?

Is any clarification of your observation
aims needed? E.g. to enable your
observer to address them?

Is any clarification of the things that
need to be looked at needed? E.g. does
this involve real time synchronous
discussion, a current ‘live’ asynchronous
discussion or a closed asynchronous
discussion?

How does your observer gain access to
your programme / module /
discussions?

Is there anything else that your
observer needs or wants to know
before the observation begins?

When and how will feedback be
shared? E.g. face to face, by phone, in
writing, a combination, etc.
17
ALT-C short paper #0114: Peer observation for distance learning e-tutors
Section 3: Observer and person being observed
How will feedback / observations be used to inform future teaching?

How will feedback / observations be
used to inform your own teaching?

How will feedback / observations be
used to inform others teaching? E.g.
those who teach on the same
programme or module, those who teach
on other programmes, etc?

How will lessons learned be taken
forward beyond personal actions? E.g.
lessons learned by both observer the
person being observed, examples of
good or innovative practice observed,
ideas for the development of the
technology in use or module design?
18
ALT-C short paper #0114: Peer observation for distance learning e-tutors
Guidance notes
Principles of peer observation (trial)
Peer observation is primarily a developmental technique for the individuals involved. It facilitates
the sharing of impartial and objective feedback based upon actual observations of teaching practice
in order to:



Enable the person being observed to better understand their behaviours and actions whilst
teaching and so make informed decisions about how to behave and act differently in order
to become a better teacher;
Enable the observer to objectively describe and discuss the behaviours and actions of other
teachers and to be informed by these observations. This will include the identification and
sharing of good or innovative practices; and
Enable diverse and innovative teaching methods and practices, including use of technology,
to be opened up to external friendly scrutiny and discussion, in order to ensure that good
and innovative practices are shared.
The person being observed owns the process in order to have confidence in it; hence self selecting
the materials to be observed, the timeframe of the observation and the aims of the observation are
key. Section 1 of the pro-forma encourages the person being observed to think deeply about what
they require feedback on, why they want it and where the evidence is likely to be located.
Ideally both parties should meet prior to the observation to exchange the information set out in
section 2 of the pro-forma, which should be seen as a prompt rather than as a checklist. For the
purposes of the trial, it is proposed that a random pairing of observer and person to be observed will
be adopted in order to inform subsequent evaluation of which combinations work and which do not,
and what other information might need to be shared prior to the observation in order to support
success. This may, for example, lead in future to the sharing of context information where observers
are drawn from different disciplinary areas.
Context information refers to any important information about the group/activity/course that the
observer will need to consider during the observation. This might include:




The balance of online/face-to-face activities
How the online activity/work fits into the overall course (timing / assessment criteria)
Student mix
Particular challenges and past experience
Focus for the observation
The person being observed may wish to reflect on the following questions to guide their thoughts
about what they want to explore through peer observation:

What are your goals in your use of technology and teaching approaches to support students’
learning? Are they currently being achieved?
19
ALT-C short paper #0114: Peer observation for distance learning e-tutors




What aspects of student online learning and use of technology would be most interesting for
you to explore and why?
Is there an aspect of your own online teaching and/or support role that you would like to
change or develop?
Are there particular aspects of your use of the technology, your online materials and/or the
online activity that you would like feedback on? (Note: this might be to understand better
why something works well as much as something that is not working successfully.)
Are there aspects of your role as tutor that you would like the observer to focus on?
Select 2 or 3 specific areas , which might cover one of the following issues:



Materials, instructions & broader design issues: e.g. site structure / materials presentation;
content of online materials – level and clarity; overall course design.
Tutor online activity: e.g. management of the online activity; tutor input into / contribution
within student online discussion (quantity & quality); desirable behaviours; styles of
questioning / prompting / developing student input; etc.
Student engagement in online activity: e.g. amount of student participation; quality of
student interaction; depth of student discussion.
For example, an observee might identify a perceived problem with students’ engagement with a
discussion exercise e.g. exhibiting behaviour such as the ‘yes man’ format or the ‘post and run' or
mini-essay, format within a discussion board activity. A guided observation might help to identify:
-
-
the level and nature of interaction between tutor and student group? E.g. which behaviours
encourage and discourage interaction?
where to begin if the group has already started...”sometimes I wonder if they actually need
me to say anything at all but I feel compelled to make a posting just to highlight that I am
here”
whether there is a student to student problem and, if so, whether and how it may be
addressed.
Don’t forget: The observee should provide advice to the observer on where to look for evidence.
Delivering feedback to the observee
Ideally feedback should be shared face to face, in private, where possible. The observer should seek
to use non-judgemental language, which helps to foster a mutual learning process between
observee and observer.
Note taking during the observation would take the form of “I noticed …. and I wondered / thought /
wanted to ask you…” and should be supported by actual examples.
Both those being observed and those doing the observing should be prepared and plan to practice
what they learn as individuals, but also to share their learning with their immediate team colleagues
and the wider distance learning community, including colleagues who support distance learning such
as the eLearning team.
20
ALT-C short paper #0114: Peer observation for distance learning e-tutors
Post-trial evaluation
Both observers and those being observed will be asked to participate in the post-trial evaluation and
it is assumed that participation in the trial is tacit acceptance of a willingness to do so. No
confidences will be broken in doing so.
Acknowledgement
The pro forma and guidance notes are adapted from the COOLAID resources which were presented
at the HEA evidence-based seminar on peer observation at the University of Hull– 26th May 2010:
http://www.heacademy.ac.uk/assets/documents/accreditation/Hull_COOLAID_presentation_26_M
ay_10.pdf
21