GORP Faculty Report
Dear Professor ____________,
Once again, thank you so much for participating in the pilot project to test the GORP observation
tool here at BU. Below is a summary of the observation data we collected during your class
meetings along with a summary of the results of the pilot project from all the data we collected. A
similar summary was prepared for each instructor that was observed. The data we collected from
your classroom will not be explicitly shared with anyone outside of the GORP team without your
express consent.
The GORP/COPUS protocol (see Lund et. al., attached) is designed to identify what activities the
instructor and students are engaged in during a class meeting in a neutral an objective way using
observers with no special knowledge of pedagogical design. During a class meeting, for each two
minute interval the observer records, via a web application, what kind of activities the instructor
and the students are involved in. The activities that the observers can choose from are given in
Table 1 of the attached paper. Notice that it is possible during each time interval to record more
then one activity, so percentages of activities will add up to more then 100%.
Based on the data reported on in Lund et. al., a cluster analysis was completed to identify specific
types of classroom instruction. The cluster analysis identified both instructor and student activities
that tended to occur simultaneously. For example, clusters that were categorized “Lecture”
included instructors lecturing, real time writing, and students listening. These clusters fell into four
broad categories including: traditional lecture, socratic method, peer instruction, and collaborative
learning.
A summary of the observations is given in the graph below. These represent the average over all of
the class meetings observed.
GW- Group Work
MG- Moving through
groups
FUp- Follow Up
CQ- Clicker Questions
SQ-Student Questions
An-Q-S-Students
Answering Questions
RtW- Real time Writing
Lec- Lecture
Class
100.00
Average
80.00
60.00
40.00
20.00
0.00
GW
MG
FUp
CQ
SQ
An-Q-S
Cluster of observations
RtW
Lec
GORP Faculty Report
A total of 4 class meetings were observed, each observed class on 4/17, 4/22, 04/24, and 04/27.
Three observers were present on 04/22. This allowed for an estimate of inter-rater reliability
(IRR). Inter-rater reliability can be estimated by comparing which items were chosen by each
observer during each two-minute period. For this meeting at which multiple observations were
recorded, the Cohen’s Kappa for observations on instructor behavior (measure of IRR) calculated is
66% (avg. of 3 pairs).
This is lower then we would like, but we think it underestimates the effectiveness of the tool since
the individual observations showed similar values to the averages given above.
Keep in mind that each entry above represents the percentage of 2-minute intervals where the
behavior was observed. Multiple behaviors (for example lecturing and clicker questions) can be
present in a particular time interval. Based on these results using the classifications in Lund et. al.,
your class meetings are consistent with the category of “Group work ” or “ Extensive Peer”
teaching.
Additionally, observers estimated the level of student engagement during each 2-minute interview,
choosing between categories of “low”, “medium”, or “high”. Use of this feature was not consistent
across observers, but based on the observations that were collected, student engagement was
routinely rated between “medium" and “high” and most often “high”.
Based on the observing experiences in your classroom and others we learned several things that
will help improve the future implementation of the tool.
A few of the things we learned include:
• Present and former LA’s are a valuable pool of observers.
• How to improve our training of the observers to increase inter-rater reliability.
• To improve communication with faculty being observed.
• How to categorize various teaching styles and approaches based on the data.
• The tool is imperfect and does not fit all classrooms equally, necessitating the development
of our own protocols when that feature becomes available.
We plan to continue observations in the Fall of 2015, implement improvements as indicated above,
and eventually provide it as a broader service across the university.
This report is of course is a general summary of the data collected. More detailed results from your
individual class observations, including observer comments are available as well. Let me know if
you would like to look at the more detail, and perhaps we can schedule a time to discuss what
questions you would like answered.
We are planning to write a general report on the pilot program. Do you give your consent to release
the summary results of observations of your course displayed above together with the Lund et.al.
classifications? We will anonymize all information.