CONFERENCE PROCEEDINGS
13th Toulon-Verona Conference “Organizational Excellence in Services”
University of Coimbra (Portugal) – September 2-4, 2010
pp. 57-65 – ISBN: 978-972-9344-04-6
USING A PEER AND SELF OBSERVATION APPROACH TO FACILITATE
THE ADOPTION OF ACTIVE LEARNING METHODOLOGIES IN
ENGINEERING LECTURE CLASSES
Bill Williams
Escola Superior de Tecnologia do Barreiro, Setubal Polytechnic Institute,
Barreiro, and CEGIST, Technical University of Lisbon, Portugal
bill.williams@estbarreiro.ips.pt
Isabel Carvalho
Instituto Superior de Engenharia de Lisboa, Lisbon Polytechnic Institute,
Lisbon, Portugal and IDMEC, University of Porto, Portugal
icarvalho@dem.isel.ipl.pt
ABSTRACT
As many studies on quality in higher education have shown the level of activity of students in
lecture classes to be a useful indicator of learning in technology and engineering courses, learner
activity can be employed as a proxy for student learning.
A project to facilitate the introduction of Active Learning techniques in engineering lecture classes
is described. This involves the development of an instrument to assist peer-based faculty
professional development. The simple semi-quantitative instrument (Learner Activity Monitor
Matrix - LAMM) uses in-classroom observation or post-class video observation to allow lecturers
to monitor the degree of student activity before and after the implementation of Active Learning
techniques in their classes and to measure overall learner activity (represented as an activity index)
and participation (as a participation parameter).
The paper compares the use of the LAMM with two other approaches commonly used to measure
student engagement. Empirical data is also presented comparing the time devoted to traditional
lecture techniques in observed classes in the Portuguese context with those of a comparable US
study.
1. INTRODUCTION
The overall aims of the project are centred on the adoption and dissemination in engineering
education at institutional and inter-institutional level in Portugal of good practice with regard to
Active Learning (AL) in lecture classes.
1.1 Background
At institutional level, high failure and drop-out rates for students on engineering courses have been
a matter of concern for many Portuguese higher education institutions as evidenced from external
evaluations carried out by the European University Association and internal quality reports
compiled by institutions.
The majority of engineering instructors have not had specific training in Education Sciences and the
provision of “pedagogical training” is often mooted as a way to tackle perceived skill gaps in this
area. Such training typically takes the form of short-duration workshops run by staff from
13th Toulon-Verona Conference, Coimbra, Portugal, 2 -4 September, 2010
educational rather than engineering fields and in many cases may not lead to significant change in
the faculty practice at the “coalface” of the lecture hall/classroom/laboratory or in course design.
Furthermore, many instructors, although keen to improve their teaching do not feel completely
comfortable with the language and concepts of pedagogy and this may lead them to put their faith in
technological solutions (e.g. e-learning platforms) rather than analyzing the learning process itself
and adopting approaches which have grown out of the considerable body of work related to the
Scholarship of Teaching and Learning in Engineering Education. To date there has been relatively
little work in this field in the context of Portuguese engineering education and most teaching tends
to follow a fairly traditional knowledge transmission model in which the role of the learner is
largely that of passive receptor.
1.2 Significant findings from other research studies
Although there is no published work to date demonstrating the value of the active learning
techniques in lecture classes within the specific context of Portuguese engineering education, given
the existence of a large and credible body of research, including longitudinal and meta-studies,
showing the value of AL techniques in engineering education in other countries, particularly in the
US [1]- [4], we believe one can be confident of the qualitative benefits of employing these
techniques in the Portuguese context.
It is of interest to note that in parallel to the mounting accumulation of evidence found in the above
studies and meta-studies relating to the inherent weaknesses of the traditional lecture approach as
applied to engineering education, there has been a parallel development in the field of physics
education. Harvard physics professor Eric Mazur recounts his increasing disenchantment with the
pure lecture mode when he began to measure student learning by more sophisticated instruments
than conventional examinations and his subsequent success with the implementation of what he
refers to as peer instruction in his book Peer Instruction: A User’s Manual [5]. It may also be
significant that Mazur’s lecture “Confessions of a converted lecturer” has amassed more than
20,000 views between being put up on YouTube in November 2009 and the time of writing (May
2010).
Furthermore our approach encourages faculty to sit in and observe each others classes and although
we have not found studies on the value of this in higher education engineering classes, it is
significant that in a 2007 report carried out by McKinsey which aimed to identify the best school
systems in the world, they concluded that one of the key shared characteristics of the best
performing systems was that “teachers regularly invite each other into each other's classes to
observe and coach” [6]. This study was possible because of the existence of reliable OECD data as
part of the Program for International Student Assessment (PISA). Such data are not available for
higher education systems but it is reasonable to assume that the beneficial effect of this aspect of
teaching community practice would also play a useful role in this context.
1.3 Measuring student activity as a proxy for learning
One of the challenges facing researchers in the field of engineering education research (EER) is that
the vast majority of practitioners in engineering education are engineers and as such their research
training has been very much focused upon a post-positivist model of quantitative research. Borrego
and Elliot argue cogently that EER often calls for a qualitative or a mixed method research
methodology but they note that such approaches are as yet rarely applied in this field [7]. These
types of methodology have been widely used in social science and to a lesser extent in medical
research in recent decades.
13th Toulon-Verona Conference, Coimbra, Portugal, 2 -4 September, 2010
The intention in this work was to improve learning in lecture type engineering classes. As
traditional approaches to measuring learning improvements from intervention (exam results for
example) are problematic and tend to require years of data collection or very large data sample in
order to collect valid results, it was decided to employ a proxy methodology as has been used very
successfully in various other areas of research (e.g. cholesterol levels as a proxy for cardiovascular
health or tree rings for climate history) [8]. Although this methodology is rarely been applied in
EER, we feel that is the most appropriate here so as to provide useful empirical data in a relatively
small-scale project.
Accordingly the proxy measured in this system is student activity/passivity in the lecture classroom
and the semi-quantitative observation measurements recorded allow individual instructors who aim
to turn their lecture classes more active to record an Activity Index, Participation Parameter and
percentage lecture time for their classes over time.
1.4 Research questions
The overall objectives were twofold:
- to motivate engineering faculty to introduce established AL techniques in their lectures (“theory
classes” in the Portuguese system) over a three-year period;
- to cultivate a peer-sharing approach which will encourage the dissemination and development of
this approach at institutional level and beyond.
The research questions that flowed from these objectives can be summarised thus:
i) can a semi-quantitative classroom observation tool be developed which can provide instructors
with useful information, while being simple and flexible to implement;
ii) can such a tool serve as part of a peer-sharing strategy to facilitate the introduction of Active
Learning strategies in Portuguese engineering courses?
2. METHODOLOGY
Active Learning has been defined as any strategy "that involves students in doing things and
thinking about the things they are doing" [6] and this broad definition can be taken to include a very
wide range of teaching and learning activities including collaborative and problem-based learning.
In this work, however, we follow Paulson [4] in using the term more narrowly to refer to a number
of techniques that can be incorporated in the lecture context so as to give students a more active role
in their learning process.
We set out to introduce AL methods via a core group of instructors, mostly new to this approach,
who are teaching students at the ESTBarreiro Engineering College.
Assuming that engineering staff are often more comfortable with quantifiable results and a
pragmatic approach, rather than one involving an immersion in unfamiliar education science theory,
we have been developing a simple semi-quantitative tool, the LAMM, which uses in-classroom
observation or post-class video observation to monitor the degree of student activity before and
after the implementation of Active and Collaborative Learning techniques in their classes. This
13th Toulon-Verona Conference, Coimbra, Portugal, 2 -4 September, 2010
allows an individual lecturer or team to focus on the question of learner activity during class contact
time and develop efficient techniques to increase it.
Thus, the project has involved using self and peer observation of lectures to allow each participating
lecturer in the pilot group (5 in all) to first of all establish a baseline level for the Learner Activity
Index of students in his/her subject curriculum unit initially and then introduce tried and tested AL
techniques with the intention of increasing levels of learner activity and participation. The in-class
activities were adapted from those in two online activity banks [9], [10] containing around 30
different activities found to be useful in engineering courses in the United States, and included the
following: Jig-saw reading texts; Revision puzzles; In-Class Teams; Think-Pair-Share; Minute
paper; Regular uses of students’ names; The "One Minute Paper"; Muddiest (or Clearest) Point;
Affective Response; Clarification Pauses; Wait Time; Discussion; show of hands voting; active
review sessions, and student revision lists.
2.2 Development of the LAMM observation tool
The evolution of the LAMM was described in a previous paper [11]. It is a matrix that aims to allow
an observer to register the behaviour of the majority of students present in the lecture into one of 7
categories, readings being taken every 2 minute. It can both be printed out and filled in by hand
during observation or be in the form of an Excel chart which is completed on a portable computer.
The categories recorded and their relative weightings chosen are: All listening to lecturer or to
another student (1); individual work (2); checking answers (2); pair-work (3); group-work (3);
distracted or other non-classified behaviour (0). The higher weighting for collaborative activities
reflects the evidence that suggests that peer learning results in higher learning gains and retention
[12],[13].
The Activity Index for a 1-hour portion of a lesson is calculated on the basis of these weightings.
An observer classifies the activity of the majority (75%) of learners present in the room at twominute intervals. The observations cover the first 60 minutes of a scheduled lecture class,
irrespective of the actual length of the class (usually 1.5 or 2 hours). An Activity Index is calculated
by assigning a weighting of 1 to passive listening, 2 to individual work and 3 to pair or group work.
Thus thirty recordings are made in each LAMM. Consequently, Activity Index values of 30, 60 e 90
would represent exclusively listening, individual activity and group activity respectively.
In addition to the learner activity index calculation the LAMM has also been adapted to capture data
on learner question and answer which is reflected in a Participation Parameter; this is calculated by
registering spontaneous learner questions/contributions (weighting of 1), group/class response to
lecturer question (0.5) or individually elicited student response (1).
13th Toulon-Verona Conference, Coimbra, Portugal, 2 -4 September, 2010
FIGURE 1. Sample Learner Activity Monitoring Matrix
13th Toulon-Verona Conference, Coimbra, Portugal, 2 -4 September, 2010
3. FINDINGS
3.1 Use of the LAMM
Table 1 shows an example of how a sample participating lecturer can see her Activity Index (AI)
and Participation Parameter (PP) as she is introducing active learning techniques in her lecture
classes over a two year period.
02-06-2209
28-04-2010
13-04-2010
07-04-2010
02-06-2009
14-05-2009
12-05-2009
07-05-2009
07-05-2009
05-05-2009
30-04-2009
28-04-2009
28-04-2009
28-04-2009
26-03-2009
17-03-2009
18-11-2008
18-11-2008
11-11-2008
20-05-2008
17-04-2008
14-04-2008
03-04-2008
AI
AI 30 50 37 57 37 38 37 50 54 30 30 38 55 33 32 38 35 31 52 56 33 31
PP na 3 8 11 7 8 25 29 17 16 11 12 18 21 22 16 27 13 0 0 14 13
Date
TABLE 1. Sample Activity Index values for a participating lecturer
3.2 Comparing data on percentage time devoted to lecturing
To date the LAMM has been used to observe 60 minutes of 120 classes from 10 courses at three
engineering schools in Portugal. The observation dates were normally selected randomly depending
on the availability of observers or video filming facilities Table 1 shows average values for Activity
Index and Participation Parameter values for AL-oriented teachers in comparison with a control
group of traditional instructors who were not part of the project. An AI value of 30 indicates that the
students spent all or almost all of the 60 minutes listening whereas a value of 90 would represent an
hour of collaborative peer activity We note that the AL-oriented classes featured considerably more
activities than the traditional ones as seen in the AI values, while the PP data show that the level of
student participation was also considerably higher as represented by the number of questions and
answers during the class.
LAMM Results
Participation
Instructors
Activity index
parameter
AL-oriented (n = 92)
45,39
17,1
Traditional (n = 15)
30,2
9,5
TABLE 2
Assuming for simplicity that the class time recorded in column 1 of the LAMM to represent
“lecturing”, Table 3 shows a comparison between the % time engaged in lecturing for both ALoriented and traditional instructors in our study.
13th Toulon-Verona Conference, Coimbra, Portugal, 2 -4 September, 2010
LAMM Results
Instructors
AL-oriented (n = 92)
Traditional (n = 15)
% lecture time
62
93
TABLE 3
This tallies with findings by Cox and Cordray who used the VOS observation system to study
instructors using the “How People Learn” (HPL) framework and compare them with traditional
instructors on 28 bioengineering courses in the US [14]. They concluded that “although courses are
designed to be innovative, the dominant pedagogical practice is still lecture”.
3.3 Peer sharing of pedagogical practice at institutional level
An important benefit of the approach described here is that it encourages faculty members to sit in
and observe each others classes and this has been cited as a significant learning experience by all
participating staff in interviews carried out. This is very much in keeping with the findings of the
McKinsey report mentioned earlier.
The national workshop with Professors Richard Felder and Rebecca Brent in the first year of the
project appears to have been significant in the development of the group in that it gave visibility and
legitimacy to work that hitherto had involved a small group of instructors. It also helped give
participants language to discuss their experience with others and confidences to speak in public
about pedagogy – an area those most engineering instructors see as being outside their sphere of
expertise.
The project is now in its third and final year and in addition to the development of the LAMM itself,
it has led to discipline specialists who had not previously been actively involved in EER being
responsible for 5 research papers and 1 poster presentation at international conferences and two
papers and 2 poster presentations at national conferences.
Internally at Setubal Polytechnic Institute there has also been significant activity with team
members running two series of well-attended workshops dealing with classroom pedagogy.
3.4 Comparison of the LAMM approach with existing observation systems
The VOS Observation System was developed in the US by the Vanderbilt-Northwestern-TexasHarvard/MIT Research Center for Bioengineering Education and Technology based on work done
in K12 classes employing the “How People Learn” system. This is a powerful system which has
been validated in a large number of courses and produces very fine-grained data when implemented
by suitably trained staff [14, 15, 16].
Over the last 10 years a considerable number of studies have been published describing the use of
Audience Response Systems (Clickers) as a management tool for engaging students, particularly in
large lecture classes [17].
Table 4 compares the different observation systems. The LAMM approach, being relatively simple
to implement at department level and facilitating peer observation and discussion may be a good
choice for engineering faculties who want to introduce the Active Learning approach and encourage
staff professional development.
13th Toulon-Verona Conference, Coimbra, Portugal, 2 -4 September, 2010
Logistical
requirements
Data type
Data Components
Underlying
Theoretical
framework
Peer observation
and community
building
VOS
Trained observers
using hand-held IT
equipment
Quantitative
“How People
Learn”
Not usually
Audience
Response
Systems
Proprietary
equipment (IR or
RF based); data
captured
automatically - no
observer needed
Peer observer (or
trained student
assistant); video
camera optional
Quantitative
Classroom
Interaction; Student
Engagement;
Narrative Notes;
Global Rating.
Learner response to
simple questions.
Active Learning
Not usually
Activity Index,
Participation
Parameter.
Active Learning
Yes
LAMM
Semiquantitative
TABLE 4 Comparing the three observation systems
4. LIMITATIONS
It should be said that it is not our aim to attempt to demonstrate unequivocally that AL techniques
are as beneficial in the Portuguese engineering education context as they have been shown to be
internationally, because the generation of valid, credible data on this aspect would imply a
timescale, number of participating students and scale of project we are not in a position to undertake
at this stage.
5. CONCLUSIONS
The work presented here indicates that the use of the LAMM presents a useful approach to staff
pedagogical development in engineering education institutions as it provides instructors with a way
to monitor what they and their students are doing and can facilitate the implementation of Active
Learning techniques in lecture classes thus representing an important contribution to educational
quality.
The LAMM system is easy to implement, distinguishes between traditional and AL-oriented
pedagogical practice and contributes to peer-sharing of pedagogical practice.
Acknowledgements
Financial support has been provided by the Portuguese Fundação para a Ciência e a Tecnologia
(FCT) of the Portuguese Ministry of Science and Technology and Higher Education (PTDC / CED /
69529 /2006).
References
[1] Springer, L., Stanne, M.E. and Donovan, S. (1998). Effects of Cooperative Learning on Undergraduates in Science, Mathematics, Engineering,
and Technology: A Meta-analysis, retrieved 30/1/2008 from http://www.wcer.wisc.edu/archive/cl1/cl/resource/scismet.pdf.
[2] Felder, R.M., Felder, G.N. and Deitz, E.J. (1998). A Longitudinal Study of Engineering Student Performance and Retention. V. Comparisons with
Traditionally-Taught Students. J. Engineering. Education, 87(4), 469-480.
[3] Prince, M. 2004. "Does Active Learning Work? A Review of the Research", Journal of Engineering Education , 93(3): 223-231; retrieved
30/1/2009 from
http://www4.ncsu.edu/unity/lockers/users/f/felder/public/Papers/Prince_AL.pdf
[4] Paulson, DR. Active Learning and Cooperative Learning in the Organic Chemistry Lecture Class. J. Chem. Educ. 1999, vol. 76, 1136.
13th Toulon-Verona Conference, Coimbra, Portugal, 2 -4 September, 2010
[5] E. Mazur, Peer Instruction: A User’s Manual, Prentice Hall, Upper Saddle River, NJ, 1997
[6] Barber, M. & Mourshed, M., “How the World's Best Performing Schools Come Out on Top”,
McKinsey and Company, September 2007, retrieved 25/5/2010 from
http://www.mckinsey.com/App_Media/Reports/SSO/Worlds_School_Systems_Final.pdf
[7] Borrego, M., Douglas, E. P., & Amelink, C. T. (2009). Quantitative, Qualitative, and Mixed Research Methods in Engineering Education. Journal
of Engineering Education, 98(1), 53-66.
[8] Snow, A. L., Cook, K. F., Lin, P., Morgan, R.O., and Magaziner, J., “Proxies and Other External Raters: Methodological Considerations” Health
Serv Res. 2005 October; 40(5 Pt 2): 1676–1693.
[9] Paulson R.M. and Faust, J.L. (2008). Active Learning for the College Classroom, retrieved 30/1/2008 from
http://www.calstatela.edu/dept/chem/chem2/Active/
[10] Felder, R.M. and Brent, R. (2006). Active Learning, retrieved 30/2/2007 from
http://www.uwf.edu/cutla/workshops/Active%20Handout.pdf.
[11] Carvalho, I.S., Williams, B., “Facilitating the adoption and dissemination of Active Learning Methodologies in Engineering Education using a
Classroom Observation approach” , Proceedings of the 37th Annual Conference of the European Society for Engineering Education (SEFI),
Rotterdam, The Netherlands, July 1-4, 2009 ISBN : 978-2-87352-001-4, http://www.sefi.be/wp-content/abstracts2009/Carvalho.pdf
[12]Hake, R. R. (1998). Interactive-engagement versus traditional methods: a six-thousand student survey of mechanics test data for introductory
physics courses. Am. J. Phys. 66(1), 64–74.Pollock, S. J. (2006). Transferring transformations: learning gains, student attitudes, and the impacts of
multiple instructors in large lecture courses. AIP Conf. Proc. 818(1), 141–144.
[13] MacManaway, L. A. (1970). Teaching methods in higher education—innovation and research. Universities Quart. 24(3), 321–329.
[14] Cox, M., Operationalization of Innovative and Traditional Pedagogical Practices within Undergraduate Bioengineering Courses, International
Journal for the Scholarship of Teaching and Learning, Vol. 3, No. 1 (January 2009), http://www.georgiasouthern.edu/ijsotl
[15]Harris, A. H., and Cox, M. F. (2003). Developing an Observation System to Capture Instructional Differences in Engineering Classrooms, Journal
of Engineering Education, October 2003.
[16] Cox, M. F., Cordray, D.S. (2008). Assessing Pedagogy in Bioengineering Classrooms: Quantifying Elements of the "How People Learn" Model
Using the VaNTH Observation System (VOS), Journal of Engineering Education, October 2008.
[17] Caldwell, J. (2007). Clickers in the Large Classroom: Current Research and Best-Practice Tips. CBE-Life Sciences Education, 6 (1): 9-20,
Spring 2007.