Learning Behavior Surveys
Dr. Rudolph Mitchell
Associate Director for Assessment and Evaluation
Teaching and Learning Laboratory
Massachusetts Institute of Technology
Perhaps the most compelling challenge in educational research is to determine those
aspects of the educational environment that contribute to student learning. Factors to be
explored include obvious ones, as, for example, pedagogical methods or tasks students
undertake to demonstrate they have mastered content or skills. But the impact of
relatively newer pedagogical practices, such as problem-based learning, or recently
developed educational technologies also needs to be understood. Questions that
educational practitioners and researchers seek to answer include how best to implement
these practices or use these technologies, under which conditions they are most effective,
and which students might benefit most from them.
We have developed three surveys—the Small Group Survey, the Heat Exchanger Survey,
and the WebLab Survey—to assess the effectiveness of several different pedagogical
innovations with which we have experimented. We group these surveys under the
umbrella term Learning Behavior Surveys because in each case we have asked
students to identify the extent to which a new pedagogical practice or technology has (or
has not) impacted their learning. Some educational researchers maintain that student
reports of their own learning are valid sources of data that can stand in for more direct
measurements of performance.
Background
The Learning Behavior Surveys are an outgrowth of the Cognitive Behavior Survey to
Assess Medical Student Learning and the Cognitive Behavior Survey: Residency Level
developed by Dr. Mitchell (1994, 1995, 2005) and his co-authors. These surveys profiles
cognitive, metacognitive, and experiential aspects of medical students’ and medical
residents’ learning. Specifically, they measure:
. . . the degree of memorization, conceptualization, and reflection
present . . . ; the degree to which [students and] residents function
as critical thinkers and independent learners; and how well they
relate to the educational experience (Mitchell, 2005, p. 2).
Mitchell and his co-developers sought to better understand how medical school students
and residents learn because they hypothesized that the ways in which these students
behaved in relation to their learning would have an impact on how well they performed. .
This line of research follows the work of Kolb (1981, 1984, 2001), Schon (1983), Biggs
(2001), Sternberg and Grigorenko (2001), and others who postulate that students have
Dr. Mitchell may be reached at rudy@mit.edu.
May 2007
individual learning styles that are “habitual patterns of preferred ways of doing something
(e.g., thinking, learning, teaching) that are consistent over long periods of times or across
many areas of activity” (Sternberg and Grigorenko). It is the students’ approach to
learning that Mitchell’s early surveys, as well as the Learning Behavior Surveys, are
designed to measure.
Generation and Use of the Surveys
Each of the three surveys was developed from extensive interviews conducted with
students in the classes in which the pedagogical or technological innovation had been
introduced. Surveys were then piloted and scales were developed.
The Small Group Survey explores several aspects of small group learning: cognitive
behavior, student interactions, and effectiveness of the learning experience. The survey
consists of 36 statements for which students rate their agreement by means of a sevenpoint rating scale. A value of "1" indicates strong disagreement while a value of "7"
indicates strong agreement.
The survey includes two scales: cognitive behavior comparison scale and meaningful
learning scale. The cognitive behavior comparison scale measures how well small group
work stimulates students toward positive cognitive behavior (e.g., deeper understanding,
increased attentiveness) in relation to traditional methods. The meaningful learning scale
measures how meaningful students find the small group learning experience. The
coefficient alphas for the two scales are .91 and .95 respectively.
Examples of questions from the cognitive behavior comparison scale of the Small Group Survey
28. Traditional recitations or lectures provide a more effective learning experience than the small group sessions.
29. The small group sessions encourage me to think more deeply about the subject matter than traditional
recitations or lectures.
Examples of questions from the meaningful learning scale of the Small Group Survey
17. As we worked through problems, my small group focused on understanding the underlying concepts.
18. Working in a small group gave me an opportunity to express what I knew.
The Heat Exchanger Survey consists of 41 statements or descriptors that profile aspects
of the experience students have had with a remote laboratory. By means of a seven-point
scale, students express their agreement with each statement or descriptor. A "1" indicates
strong disagreement while a "7" indicates strong agreement.
The survey groups items into five areas which represent different perspectives of the
learning experience: team profile, graphical interface, understanding, learning behavior,
and learning experience. To provide an additional dimension to the data analysis, scales
were developed based on the five groupings: teamwork, graphical interface, understanding, conceptual learning, and learning experience.
Examples of questions from the Heat Exchanger Survey
2. How well did the heat exchanger experiment affect your understanding
in the following areas?
d. Appropriate assumptions to make in modeling a heat exchanger.
e. How the theory of heat transfer within a heat exchanger differs from the process in an actual heat exchanger.
The WebLab Survey profiles student experience with WebLab in regard to the interface,
learning experience, learning behavior, understanding, favorability, and effort. Students
are asked to express their agreement or disagreement with each statement on a sevenpoint scale. A “7” indicates strong agreement while a “1” indicates strong disagreement.
Examples of questions from the WebLab Survey
How well do the following phrases describe the mental operation used while
gathering data when running the WebLab Experiments?
Visualizing how devices work
Reasoning about the behavior of devices
Thinking about relationship among the variables
Impact, Engagement and Application
The importance of the data produced from Learning Behavior Surveys is that they
provide insights into the students’ learning as they experience it themselves. These
surveys allow instructors and researchers to gain a better understanding of what
contributes to that learning, as well as what does not. They can then use these results in
an iterative process to maintain those aspects of the intervention that are a factor in
student learning, and strengthen or eliminate those that are not.
References
Biggs, J. (2001). “Enhancing Learning: A Matter of Style or Approach?” In Sternberg,
R. J. and Zhang, L., eds., Perspectives on Thinking, Learning, and Cognitive Styles.
Mahway, N.J.: Lawrence Erlbaum.
Entwistle, N., McCure, V., and Walker, P. (2001). “Conceptions, Styles, and Approaches
within Higher Education: Implications.” In Sternberg, R. J. and Zhang, L., eds.,
Perspectives on Thinking, Learning, and Cognitive Styles. Mahway, N.J.: Lawrence
Erlbaum.
Kolb, D. A. (1981). “Learning Styles and Disciplinary Differences.” In Chickering, A.,
and Associates, eds., The Modern American College. San Francisco: Jossey-Bass
Publishers.
Kolb, D. A. (1984). Experiential Learning. Englewood Cliffs, N. J.: Prentice Hall.
Kolb, D. A. and Boyatzis, R. E. (2001). “Experiential Learning Theory: Previous
Research and New Directions.” In Sternberg, R. J. and Zhang, L., eds., Perspectives on
Thinking, Learning, and Cognitive Styles. Mahway, N.J.: Lawrence Erlbaum.
Mitchell, R. and Liu, P. (1994). “The Development of the Cognitive Behavior Survey to
Assess Medical Student Learning,” Teaching and Learning in Medicine, 6:161-7.
Mitchell, R. and Liu, P. (1995). “A Study of Resident Learning Behavior,” Teaching and
Learning in Medicine, 7:233-240.
Mitchell, R., et al. (2005). “The Development of the Cognitive Behavior Survey:
Residency Level (rCBS) to Measure Resident Learning and Related Cognitive
Behaviors.”
Schon, D. A. (1983). The Reflective Practitioner. New York: Basic Books.
Sternberg, R. J. and Grigorenko, E. L. (2001). “A Capsule History of Theory and
Research on Styles.” In Sternberg, R. J. and Zhang, L., eds., Perspectives on Thinking,
Learning, and Cognitive Styles. Mahway, N.J.: Lawrence Erlbaum.
Sternberg, R. J. and Zhang, L. (2001). Perspectives on Thinking, Learning, and
Cognitive Styles. Mahway, N.J.: Lawrence Erlbaum.