THE LEARNING MODEL AT BYU-IDAHO: EVIDENCE FROM
COURSE EVALUATIONS
By
SHAWN V. JENSEN
A dissertation submitted in partial fulfillment of
the requirements for the degree of
DOCTOR OF EDUCATION
WASHINGTON STATE UNIVERSITY
Department of Educational Leadership and
Counseling Psychology
DECEMBER 2010
To the Faculty of Washington State University:
The members of the Committee appointed to examine the dissertation of SHAWN
V. JENSEN find it satisfactory and recommend that it be accepted.
____________________________________
Paul Pitre, Ph.D., Chair
____________________________________
Paul Goldman, Ph.D.
____________________________________
Michael Pavel, Ph.D.
ii
ACKNOWLEDGEMENT
Throughout this four year journey in pursuit of a terminal degree, I have
recognized how blessed I have been to have the support and guidance of kind and loving
individuals. First and foremost, I acknowledge the grace and tender mercies of my
Savior and God. I have felt his direction in my life. I know that he loves me and wants
me to be successful in all virtuous endeavors I embark in. Second, I acknowledge the
blessing it is to have a living prophet. For the words of President Gordon B. Hinckley to
“get as much education as possible” has motivated me to keep going when it seemed too
difficult to continue. Third, my family has been ever constant in their support and
encouragement. Hilary, my sweet wife, never complained about all of the evenings she
spent as a “single mom” while I was away taking classes, or at summer institutes, or
working on my dissertation. She has been the best blessing of my life. I also
acknowledge the sacrifice of my five children as they have given up hours of play time
with their daddy over the past four years to allow me time to finish. Last but not least, I
acknowledge my committee; Dr. Pitre, Dr. Goldman, and Dr. Pavel. I appreciate all of
their direction and guidance throughout this process. Without the support and help of
these kind individuals, this dissertation would not be possible.
iii
THE LEARNING MODEL AT BYU-IDAHO: EVIDENCE FROM
COURSE EVALUATIONS
Abstract
by Shawn V. Jensen, Ed.D.
Washington State University
December 2010
Chair: Paul Pitre
The purpose of this study is to examine the relationship between the Learning
Model (LM) compared with end-of-course ratings and student perceived learning per
existing course evaluation data at Brigham Young University-Idaho. The LM is based on
a broadly accepted pedagogy associated with improving student learning by way of
increasing student engagement. Students are asked to take more responsibility for their
own education and learning by using LM principles and applying related process steps.
The three main process steps of the LM include: Prepare, Teach-One-Another, and
Ponder/Prove. This study represents an attempt to measure the effects of implementation
of the LM on course evaluations. The change in course ratings and student perceived
learning scores were measured. The correlation between LM scores and perceived
learning scores were likewise evaluated. LM scores were derived from the new
evaluation instrument beginning Winter 2009 semester. These scores reflect the level of
student and faculty participation of LM process steps for a given course. A total of eight
semesters were used in this study. A comparison of means analysis was used to compare
iv
the changes between course evaluation score from the Fall 2007 semester to the Winter
2010 semester. Also a correlation matrix was employed to measure the relationship
between LM scores and student perceived learning scores from the time the new
evaluation instrument was implemented (Winter 2009) to the Winter 2010 semester.
Once data were collected, disaggregated, and analyzed as mentioned, further
investigation of breakout variables based on class size, faculty status, and types of
courses were compared to one another. The findings suggest that course ratings
significantly increased from the time the LM was introduced in the Fall 2007 to the
Winter 2010 semester. Interestingly, student perceived learning scores did not seem to
improve significantly over the same time period for the 33 courses identified in this
study. Finally, conclusions and implications were inferred regarding the significance of
applying the three LM process steps on course ratings and student perceived learning.
The implications of these findings can have far reaching consequences as other
institutions consider implementing and/or measuring a campus wide learning model.
v
TABLE OF CONTENTS
Page
ACKNOWLEDGEMENTS .......................................................................................iii
ABSTRACT...............................................................................................................iv
LIST OF TABLES .....................................................................................................ix
LIST OF FIGURES ...................................................................................................x
CHAPTER
1. INTRODUCTION .........................................................................................1
Background ........................................................................................1
Research Questions ............................................................................2
The Learning Model ..........................................................................2
Prepare ..................................................................................5
Teach-One-Another ..............................................................9
Ponder/Prove .........................................................................10
Application .........................................................................................11
The New Evaluation Instrument ........................................................12
2. REVIEW OF LITERATURE ........................................................................18
A Change from the Traditional ..........................................................18
A New Model for Learning................................................................24
Preparing for an Active Learning Environment.....................25
Active Learning Pedagogy .....................................................26
Critical Reflection ..................................................................30
Course Evaluations ............................................................................31
vi
Measuring Effective Instruction ............................................33
3. METHODOLOGY ........................................................................................35
The Two Evaluation Instrument ........................................................37
LM Scores ..............................................................................38
Research Questions and Methods ......................................................41
Research Question 1 ..............................................................42
Research Question 2 ..............................................................46
Research Question 3 ..............................................................49
Limitations .........................................................................................51
4. FINDINGS…. ................................................................................................53
Research Question 1: A Change in End-of-Course Scores ................54
Research Question 2: A Change in Perceived Learning Scores ........65
Research Question 3: The Relationship between LM and
Perceived Learning Scores .................................................................72
The Effects of Class Size .......................................................80
The Effects of Semester Enrollment ......................................81
The Effects of Instructor Employment Status ........................83
Summary of Findings .........................................................................85
5. CONCLUSIONS............................................................................................86
Results of Findings ............................................................................86
The Change in Course Ratings Over Time ............................87
The Change in Perceived Learning Over Time .....................89
vii
The Relationship between LM Scores and Perceived
Learning Scores .....................................................................92
Possible Implications .........................................................................96
Suggestions for Future Research .......................................................98
REFERENCES ..........................................................................................................100
APPENDIX I - INITIAL RESEARCH FINDINGS BY SCOTT BERGSTROM ....109
APPENDIX II - OLD EVALUATION INSTRUMENT ...........................................113
APPENDIX III - NEW EVALUATION INSTRUMENT.........................................116
APPENDIX IV – CORRELATION MATRIX FOR COURSE RATINGS ..............119
APPENDIX V – TWO SAMPLE T-TEST MEASURING SIGNIFICANCE ..........120
APPENDIX VI – CORRELATION MATRIX FOR PERCEIVED LEARNING ....121
APPENDIX VII – PARAMETER ESTIMATES COMPARING PERCEIVED
LEARNING SCORES WITH EMPLOYMENT STATUS .......................................122
APPENDIX VIII – MEAN LM SCORES BY SEMESTER BASED ON
CLASS SIZE .............................................................................................................. 123
APPENDIX IX – LM SCORES COMPARED TO PERCEIVED LEARNING
SCORES BY CLASS SIZE .......................................................................................124
viii
LIST OF TABLES
1. Course Ratings of Qualifying Courses for T1Semesters ...............................56
2. Course Ratings of Qualifying Courses for T2 Semesters ..............................59
3. ANOVA: End-of-Course Ratings for T1& T2 Semesters .............................61
4. Perceived Learning Scores for T1 by Semester .............................................67
5. Perceive Learning Scores for T2 by Semester ...............................................69
6. ANOVA: Student Perceived Learning Scores for T1& T2 Semesters ..........71
7. Mean Perceived Learning Score by Semester Based on Instructor’s
Employment Status ........................................................................................75
8. Mean LM Scores by Semester Based on Instructor’s Employment Status ...77
9. Mean Perceived Learning Scores by Semesters Based on Class Size ...........79
10. Correlation Matrix of LM Scores and Perceived Learning Scores by
Class Size .......................................................................................................81
11. Correlation Matrix of LM Scores and Perceived Learning Scores
by Semester ....................................................................................................82
12. Correlation Matrix of LM Scores and Perceived Learning Scores by
Instructor Employment Status........................................................................84
ix
LIST OF FIGURES
1. The traditional teaching process of higher education ....................................7
2. BYU-I’s LM process......................................................................................8
3. The three process steps of the LM .................................................................8
4. Questions Regarding “Prepare” from both Evaluation Instruments ..............14
5. Questions Related to “Teach-One-Another” from both
Evaluation Instruments ..................................................................................15
6. Question Regarding “Ponder/Prove” from both Evaluation Instrument........16
7. LM Question Set from the New Course Evaluation Instrument ....................39
8. LM Score Totals ............................................................................................41
x
CHAPTER 1: INTRODUCTION
Background
Over the last decade Brigham Young University-Idaho (BYU-I) has seen many
changes. Perhaps the most significant change occurring in June 2000 when the Board of
Trustees announced an expansion of the school from a two year junior college (Rick’s
Junior College) to a four year university (BYU-I) (see General History of BYU-Idaho).
This announcement brought about opportunities to implement, enhance, and introduce
new programs, curriculum, and courses. In August 2001 the school began its first
semester as a four year university.
More recently, another significant event occurred on June 6, 2005 when Dr. Kim
B. Clark, former dean of Harvard Business School, was announced as the 15th President
of BYU-Idaho (Rydalch, 2005). From day one President Clark’s mantra for BYU-I has
been, “rethinking education” (Clark, 2009). In his inaugural address, President Clark
encouraged a more progressive and active learning approach to teaching when he stated:
The challenge before us is to create even more powerful and effective learning
experiences in which students learn by faith…. Students need opportunities to
take action…where prepared students, exercising faith, step out beyond the light
they already possess, to speak, to contribute, and to teach one another. (As cited
by Gilbert, et al, 2007, p. 98)
1
Research Questions
With the implementation of the LM at BYU-I—which denotes the need for
students to take more responsibility for their learning—and with an understanding of the
three process steps of the LM, one is left to ask the following questions:
1. How have end-of-course ratings changed from the Fall 2007 semester (pre-LM
implementation) to the Winter 2010 semester, as the LM was introduced on
campus?
2. How have student perceived learning scores changed based on course evaluation
data collected from Fall 2007 to the Winter 2010 semesters?
3. Is there a relationship between LM scores and student perceived learning based on
the new evaluation instrument (beginning Winter 2009 semester)?
The Learning Model
In the summer of 2007, President Clark introduced to the faculty and students a
new model of teaching and learning that was simply termed, the Learning Model (LM)
(Learning Model, 2008). The LM was introduced as a pedagogical approach to
classroom teaching and learning. This new model reflected President Clark’s philosophy
of active student engagement and by implementation of said model, faculty were invited
and encouraged to rethink and evaluate their instructional pedagogy. Active student
engagement plays an essential role in the effective use of the LM. By creating a more
participatory learning environment students are able to be more active and engaged in
their own learning. Faculty members were encouraged to evaluate their teaching
methodology and evaluate how they could facilitate a more active learning environment
2
in their course(s). The LM was implemented to foster student growth and development,
both academically and spiritually through active student engagement (Learning Model,
2008). As President Clark stated:
BYU-Idaho has embarked on a journey of growth and progress that began with
the decision to transform Ricks College into a four-year university. From that
change, BYU-Idaho has implemented unique academic offerings, creative
calendaring, and programs focused on the development of students. (Clark, 2009,
p. 1)
In this study, the researcher will compare the effects of using the newly
implemented Learning Model (LM) on course ratings, instructor ratings, and student
perceived learning scores as found on course evaluations at BYU-I. This LM consists of
three process steps which students are encourage to incorporate and apply as a key
component to learning, and are explained in further detail below. The manner in which
the LM has affected course ratings, instructor ratings, and student perceived learning
scores over a two and ½ year period as found on end-of-course evaluations is the focus of
this study.
It is important to point out however, that learning models are typically a
compilation of proven practices in teaching and learning (Thurgood & Gilbert, 2008).
Likewise, BYU-I’s LM is not proprietary to BYU-I. In general, educational theorists and
cognitive psychologists have endorsed a more active learning environment for students
for some time now (Thurgood & Gilbert, 2008). Also, active learning methodology has
been a focus of study for many researchers throughout the recent years. For example,
methods for engaging students that have been studied include peer instruction (Mazur,
3
1997), cooperative learning (Slavin, 1995), and mutual peer tutoring (King, et al, 1998),
just to name a few. BYU-I’s LM also endorses a more active learning environment for
students, but the implementation of this LM is unique in that instructors and students
campus wide are encouraged to apply the three process steps of this model in their
learning and teaching, respectively. The level at which students perceive the LM being
implemented into their respective classes may be reflected in end-of-course evaluation
scores.
Obviously, the LM is not structured to be a prescriptive “fix all” to passive
learning. The ways in which an instructor designs his/her course(s) to facilitate student
activity and learning, as it relates to the LM, can vary considerably (Gilbert, 2008). In
fact, instructors are encouraged to incorporate LM strategies that best fit the subject they
teach (Gilbert, 2008). For example, a physics course may have different options to
choose from, in terms of motivating students to actively participate, then does an English
course or computer science course. In each case, the instructor has the latitude to
consider which instructional tool will be most effective in actively conveying the
information to the students. Thus, instructors have a wide range of autonomy in choosing
which teaching technique to incorporate that best fits their teaching style as it relates to
the three process steps of the LM.
As the LM was introduced, both instructor and student were encouraged to
become familiar with the principles and process steps related to the model (Clark, 2007).
New freshman entering their first semester at BYU-I are introduced to the processes and
principles outlined in the LM in almost all of their required freshman level courses
(Learning Model, 2008). The University’s website contains information regarding this
4
model which can easily be accessed. The following was taken from the above mentioned
website and gives a brief definition of the LM:
The Learning Model was created to deepen the learning experiences of students at
BYU-Idaho. The Learning Model enables students to take greater responsibility
for their own learning and for teaching one another. The model is structured
around three process steps. (Learning Model, 2008)
The three process steps of the LM are designed to help students become more
responsible for their own learning, as mentioned above, and include: Prepare, Teaching
One Another, and Ponder/Prove. In other words, the LM endorses student preparation
before attending class (i.e., reading the assigned chapters or literature) to deepen class
discussion, during class time through actively participating in discussions, and after class
by reflecting on the material learned to help extend their understanding of the information
being presented in class.
Prepare. Beichner et al. (2007) states that in order to create an active learning
environment in the classroom the foremost element is student preparation. In this study,
the LM used at BYU-I included a preparation step. In fact, preparation is the first process
step in the LM and contains three parts (Learning Model, 2007). There is first an
ongoing, underlying spiritual preparation for each student. That spiritual preparation is
critical to learning by faith and being guided and taught by the Holy Spirit. It requires
daily attention to prayer and scripture study, and a commitment to obedience (Learning
Model, 2007).
The second part of preparation is individual. Here the student prepares to learn
and to teach by organizing required readings or assignments; by reading and studying and
5
writing or solving problems; and by thinking about questions the material raises, and how
to answer those questions (Learning Model, 2007).
The third part of preparation involves group activities. For example, the process
assumes that students have been organized into small learning teams or study groups (5-7
members). These groups “meet” for a short period (e.g., 30 minutes) on a regular basis
(before class—daily or multiple times a week) either online or face-to-face to discuss
assignments, to try out ideas, and to test understanding. The groups may also be required
to complete a group assignment or project. With this kind of preparation, students come
to class ready to learn and to teach (Learning Model, 2007).
The Prepare step is not solely the responsibility of the student. The instructor
must organize, assign, and hold students accountable for being prepared. Information
regarding the Prepare process step of the LM can be found on the BYU-I’s teaching and
learning website and reads as follows:
Student preparation work is designed, guided and aided by the instructor, but the
impetus for actually doing the work is on the student. Students rely on the help of
the instructor to show them how to successfully prepare. The instructor defines
the questions framing the assignment, provides suport materials like worksheets,
reading questions or the like, and the instructor defines both the way in which
students are expected to engage the new material and how it will be assessed.
(BYU-I, 2009)
The LM places more responsibility on the student to prepare themselves before
class (Learning Model, 2007). By doing so, the student can absorb the new material
based on their preferred learning style and at their own pace. Bonwell & Eison (1991)
6
suggest that “cognitive research has shown that a significant number of individuals have
learning styles best served by pedagogical techniques other than lecturing” (p. 46). In
addition, “several studies have shown that students prefer strategies promoting active
learning to traditional lectures” (Bonwell & Eison, 1991, p. 45). Gilbert (2008) defines
the traditional teaching process as following this sequence; students attend a class lecture,
followed by homework assignment to review the material covered in class, followed by
another lecture (see fig. 1).
Lecture
Homework
Lecture
Homework
Lecture
Homework
Figure 1. The traditional teaching process of higher education as defined by Gilbert et al
(2008).
The BYU-Idaho LM (2009) suggests that students prepare before class so they
can better benefit from and contribute to in-class learning and teaching experiences (see
fig. 2). This model places more responsibility on the student to come prepared to
participate in classroom discussion. In fact, the preparation element is critical to
classroom participation.
7
Preparation
Class
Preparation
Class
Preparation
Class
Figure 2. BYU-I’s LM process.
This approach places a responsibility for students to prepare before class to allow
for richer conversations and higher level comprehension (Gilbert et al, 2008). With this
approach “homework” becomes preparation work for future class discussion. More time
can then be used during class to facilitate a more active and engaged classroom
environment. Using this method of instruction, students study before class to prepare in
order to better participate during class. Requiring more of students early in the process
allows the instructor to spend more time deepening understanding rather than dispensing
information (BYU-I, 2009). Prepare is a key element of the LM. When students fail to
prepare for class the level of active participation during class is hampered. Each process
step of the LM is connected and leads to the next (see figure 3).
Prepare
Teach One
Another
Figure 3. The three process steps of the LM.
8
Ponder/
Prove
Student preparation cannot be overlooked nor neglected for the LM to be
effective (Gilbert, 2008). As shown from the figure above, each of the three process
steps of the LM lead to the next step with the third process step, ponder/prove,
connecting back to prepare.
Teach One Another. Preparing before class to actively contribute during class
time naturally lends itself to a more participatory learning environment, given that the
instructor encourages peer-to-peer discussion (Gilbert et al, 2008). The second process
step of BYU-I’s LM is called “Teach One Another”. This step includes group projects,
peer evaluations, teams, presentations, and any other means to facilitate a more active
learning environment. These techniques and teaching strategies are, in essence, what
makes up the Teaching One Another process step of the LM. Student are encouraged to
come to class prepared to participate and contribute to classroom learning via any of the
above mentioned techniques and strategies. BYU-Idaho Boardmember, Richard G. Scott
(2005) stated the importance of having an engaged and active learning environment in the
classroom when he stated:
Never, and I mean never, give a lecture where there is no student participation. A
“talking head” is the weakest form of classroom instruction…. Assure that there
is abundant participation because that use of agency by a student authorizes the
Holy Spirit to instruct. It also helps the student retain your message. As students
verbalize truths, they are confirmed in their souls and strengthened in their
personal testimonies. (Scott, 2005)
9
In addition, President Clark has taught that “overtime it will become apparent that
the most powerful way for students to learn is for them to teach… they will teach to
learn” (BYU-I, 2009).
Ponder/Prove. The third process step of the LM includes two elements;
pondering and then proving what is being taught. The “ponder” element includes taking
time to reflect on what is being learned to help solidify and relate the new concepts,
ideas, or perspectives being taught. This can be accomplished in a variety of ways.
Reflecting individually through journals, notes, blogs, and/or by simply telling another
person what you have learned can help solidify new learning. Group discussion and
reflection can also play a role in helping individuals ponder ideas and concepts being
taught (Learning Model, 2009). Research has demonstrated, for example, that if an
instructor allows students to ponder what is being taught and allow students time to
consolidate their notes by pausing three times for two minutes each during an hour long
lecture, they will learn and retain significantly more information (Ruhl, Hughes, and
Schloss 1987).
“Prove” includes, but is not limited to, the assessment and evaluation element of a
course (Learning Model, 2009). The Ponder/Prove process of the LM might include
tests, exams, and even thought provoking questions, where students must apply what they
have learned in order to solve a problem or in answering a question. These process steps
are grounded in current literature surrounding active learning and student engagement.
Research has consistently shown that traditional lecture methods, where professors
lecture and students listen and write, are less effective compared with active learning
10
methodology (Prince & Felder, 2006; Prince, 2004; Bonwell & Eison, 1991). Students
must be given an opportunity to actively prove their learning from time to time.
Active learning and student-centered instruction encourages students to be
actively engaged in reading, writing, discussions, or problem solving activities (Prince &
Felder, 2006; Draper & Brown, 2004; Chickering & Gamson, 1987). These types of
learning experiences must engage students in higher order thinking tasks, such as
analysis, synthesis, and evaluation. The LM was introduced as a means of encouraging
more participation in these proven teaching methods and techniques.
Application
Effectively applying the three process steps of the LM by both students and
instructors may require a change in either the instructor’s teaching methodology or the
students’ perception of what is expected of them, and sometimes both changes are
requisite (Felder & Brent, 1996). As instructors actively apply these process steps of the
LM in their courses some students might struggle with the added responsibilities placed
on them to be active participants in the classroom. Woods (1994) observes that students
forced to take major responsibility for their own learning go through some or all of the
steps psychologists associate with trauma—from shock and denial to resistance and then
finally integration. Students may not be used to taking more responsibility for their own
learning and students may push back when an instructor takes this approach. This type of
instruction may impose steep learning curves on both instructors and students, and initial
instructor awkwardness and student hostility are both common and natural (Felder &
Brent, 1996). Students who are accustom to the traditional method of teaching—which
11
includes note taking during a lecture followed by some form of independent
homework—can have a difficult time initially adjusting to a more participatory or active
model for learning. Felder and Brent (1996) gave the following example:
When confronted with a need to take more responsibility for their own learning,
[students] may grouse that they are paying tuition-or their parents are paying
taxes-to be taught, not to teach themselves…They may gripe loudly and bitterly
about other team members not pulling their weight or about having to waste time
explaining everything to slower teammates (p. 7).
As instructors begin to hear this type of resistance some may feel awkward and
want to return to the more familiar lecture methodology. What’s more, Felder and Brent
(1994) have found that an instructor who uses a more participative and active teaching
methodology (i.e., as in the three process steps of BYU-I’s LM) can initially experience a
drop in their course-end ratings.
The New Evaluation Instrument
BYU-I has recently adapted their end-of-course evaluation instrument (Winter
Semester, 2009) to reflect the three process steps of the LM, which emphasizes the need
for students to take more of a responsibility for their own learning. The means by which
to track how effective students are in taking more responsibility for their learning is
through questions asked on the end-of-course evaluations regarding LM process steps.
This new evaluation instrument has questions built-in that directly relate to the level at
which LM process steps are being implemented, based on students’ perceptions
12
(Bergstrom, 2009). Students are asked to rate how effectively they and their instructor(s)
used the three process steps of the LM throughout the course (see Appendix III).
The changes between the two end-of-course evaluation instruments are stark. For
example, in the old evaluation instrument questions were general to the course and/or to
the instructor’s teaching abilities. However, in the new evaluation instrument students
are asked questions related to their own preparation, participation, and personal reflection
in addition to how well they perceived the LM was implemented by their instructor and
peers. Using a side-by-side comparison of the two end-of-course evaluation instruments,
one can detect the difference in the question sets as they relate to students taking
responsibility in each of the three process step of the LM. The change in end-of-course
evaluation instruments is important to keep in mind throughout this study as the
methodology is affected as a result of the differences between the two evaluation
instruments. Therefore, questions that address a similar concept (i.e., the Prepare process
step of the LM) are compared and contrasted using the following tables. The first sideby-side comparison shows the difference in questions students are asked to answer as
they relate to the Prepare process step of the LM (see Figure 4).
13
Old Evaluation Instrument
Assigned homework is not just busywork.
New Evaluation Instrument
I was prepared for each class.
Text(s) and other materials have helped
The instructor held me accountable for
me understand course content.
coming to each class prepared.
The approximate number of hours per
week that I have spent in outside
preparation for this class is…
The approximate number of hours per
week that I have spent in outside
preparation for this class is…
I arrived at class on time
Figure 4. Questions Regarding “Prepare” from both Evaluation Instruments
Notice the two questions directly related to student accountability found in the
new evaluation instrument, “I was prepared for each class” and “I arrived at class on
time”. Essentially, this means that course and instructor end-of-course evaluation scores
are being influenced by student participation in the LM.
A significant change is noticed in the amount of questions asked regarding the
second process step of the LM—the Teach One Another step—between the two end-ofcourse evaluation instruments in question. The Teach One Another process step was
heavily emphasized on the new evaluation instrument, as shown in the side-by-side
comparison in Figure 5. One will quickly notice that no questions were found in the old
evaluation instrument that relate to the Teach One Another process step of the LM while
the new evaluation instrument contains six related questions. This change in course
evaluations highlights the added focus and attention administrators at BYU-Idaho are
14
giving to the LM, particularly the Teach One Another process step. These added
questions focused on peer-to-peer teaching may be used as a motivational tool to
encourage student and faculty to participate in developing and maintaining a more
participatory learning environment.
Old Evaluation Instrument
New Evaluation Instrument
I was an active participant in online or
face-to-face class discussion.
I sought opportunities to share my learning
with others outside of class.
I feel that I made important contributions
to the learning and growth of fellow
classmates.
The course provided opportunities to learn
No related questions were found.
from and teach other students.
Group work, if assigned, was beneficial
and meaningful.
Students were actively involved in this
class through discussions, group work, &
teaching.
The instructor provided appropriate
opportunities to participate in the class.
Figure 5. Questions Related to “Teach-One-Another” from both Evaluation Instruments
15
An increase in questions asked regarding the third process step of the LM—the
Ponder/Prove step—was also noticed when comparing the new evaluation instrument
with the old. The evaluation instrument that was in place previous to the implementation
of the LM contained one question related to pondering and/or proving new learning,
whereas the new evaluation instrument contained three. Likewise, the new evaluation
instrument contains questions that emphasis student taking more responsibility for their
own learning as it relates to the third process step of the LM (see Figure 6).
Old Evaluation Instrument
New Evaluation Instrument
The instructor provided opportunities to
reflect upon my learning and experiences
Exams are good measures of my
in the class.
knowledge, understanding, or ability to
I sought opportunities to reflect on what I
perform.
had learned in the class.
The course as a whole has produced new
knowledge, skills, and awareness in me.
Figure 6. Question Regarding “Ponder/Prove” from both Evaluation Instrument
This comparison between the old evaluation instruments with the new evaluation
instrument shows the evolution of the question sets and how the new evaluation
instrument has a strong LM emphasis (see Appendix II). These differences may
contribute to a change in an instructor’s course rating as students are asked to consider
additional or new elements when rating their instructor(s). For example, an instructor
16
might have had a class rating of 5.0 (out of 7 possible) based on the old evaluation
instrument, but on the new evaluation instrument the same instructor’s course rating
could now be positively or negatively affected by student perceptions of how effectively
they and their instructor applied the LM process steps throughout the course. This added
emphasis on students taking more of a role for their own learning and being held
accountable for such, surely may have an effect on end-of-course evaluations, but to what
extend? This is the purpose of this study, to explore the implications of introducing a
campus wide learning model on course ratings and student perceived learning as found
on end-of-course evaluations.
17
CHAPTER 2: REVIEW OF LITERATURE
The focus of this study is based on the implementation of the three process steps
inherent in the LM and the effects of said implementation on end-of-course student
evaluations. The researcher has chosen to focus on literature that outlines the rationale
behind the need for a change in methodology. Additionally, literature that relates to the
three process steps of the LM—Prepare, Teach One Another, and Ponder/Prove have
been examined. Research collection and analysis of data using course evaluations are
also a significant aspect to this study as the finding are based on students’ assessments of
courses they have taken. Therefore, the subsequent review of literature is divided into
three main sections; A Change from the Traditional, A New Model for Learning, and
Course Evaluations.
The first section, A Change from the Traditional, includes a review of literature
that has been published regarding the traditional teaching methodology in higher
education (specifically lectures) and reviews relevant studies regarding the significance
of a more active learning approach. The second section, A New Model for Learning,
consists of a compilation of studies related to the three process steps embedded in the
LM; i.e., prepare, teach-one-another, and ponder/prove. The third section, Course
Evaluation, considers the viability of using this type of method to evaluate effective
teaching and reviews literature that supports its use.
A Change from the Traditional
Traditionally, higher education courses have been taught using some form of a
lecturing method, mostly to accommodate for larger audiences. Recent studies have
18
shown that lecturing is not always the best methodology for disseminating information
from instructor to the students (Goyak, 2009; Johnson et al, 1998; Felder & Brent 1996).
One author, Woods (1994), describes lecture based instruction as a waste of everyone’s
time. He states that it is neither teaching nor learning and suggests that it is more a
practice in stenography. “Instructors recite their course notes and transcribe them onto the
board, the students do their best to transcribe as much as they can into their notebooks,
and the information flowing from one set of notes to the other does not pass through
anyone's brain” (Felder & Brent, 1996, p. 44). Instructors have the best intentions of
helping their students understand the material they are presenting but many times the
learning is minimal.
One problem with this form of teaching is that the instructors are largely
responsible for the dissemination and explanation of course content instead of placing
more responsibility to the students for their own learning (Crouch & Mazur, 2001;
Bonwell & Eisen, 1991). Most often instructors teach based on how they were taught.
However, if instructors were to shift responsibility of learning more to students then
learning would be transposed based on the students’ preferred learning methodology.
American educational philosophers Dewey (1916), Piaget (1928), and Vygotsky (1978)
each argues that learning is more powerful when learners are actively engaged in the
knowledge creation process. “Many proponents of active learning suggest that the
effectiveness of this approach has to do with student attention span during lecture”
(Prince, 2004). Wankat (2002) cites numerous studies that suggest “student attention
span during lecture is roughly fifteen minutes.” Earlier, Hartley and Davies (1978) found
that after fifteen minutes of lecturing the number of students paying attention drops
19
quickly and their ability to retain the information likewise decreases with each passing
minute. They further found that immediately following a lecture, students retained 70
percent of information given during the first ten minutes of the lecture and only 20
percent of information was retained from the last ten minutes of the lecture. These
finding suggest that breaking up the lecture may be beneficial because students’ minds
start to wander and activities provide the opportunity to start fresh again, keeping
students engaged (Prince, 2004).
In the mid-1980s, Halloun and Hestenes (1985) conducted a series of studies
measuring knowledge gains of conceptual material for physics students. They
demonstrated that physics students performed poorly on pre- and post-tests when even
though their grades suggested otherwise. Students could memorize the formulas, but did
not seem to understand the underlying concepts (Halloun & Hestenes, 1985). This led to
several different studies experimenting with peer instructional techniques including
concept tests designed by Eric Mazur (2001) and others. Internal departmental studies
began to show dramatic improvements in conceptual understanding (Mazur, 2001). Large
sample studies were next to be studied, of which the Hake (1998) study was the most
significant.
There has been continued growth in regards to the empirical literature that
encourages and promotes a more active learning environment (Gilbert et al, 2007). For
example, Karabenick and Collins-Eaglin (1997) conducted a study of more than 1,000
students across nearly 60 different classes and found that students in courses that
emphasized collaborative learning were more likely to develop higher-order learning
skills, such as comprehension and critical thinking. Another study by Laws et al (1999)
20
shows how a more active learning environment can greatly enhance student
understanding of basic physics concepts compared with traditional teaching techniques.
In addition, Alison King and her colleagues (1998) found that students who used
structured questioning techniques to teach and learn from their peers had a better
understanding of the information than did students who were taught by traditional
methods. Richard Hake (1998) studied more than 6,000 students in 62 introductory
physics courses and found that “students who used peer instruction techniques showed
almost two standard deviations greater average normalized gain on pre- and post-tests
than students who were taught through traditional methods” (as cited by Gilbert et al,
2007, p. 102). This is a significant finding for those advocates of active learning, in that
it shows evidence of student learning when compared with traditional lecturing
techniques.
The use of these techniques in the classroom is vital because of their powerful
impact upon students' learning. Research has demonstrated, for example, that if a faculty
member allows students to consolidate their notes by pausing three times for two minutes
each during a lecture, students will learn significantly more information (Kellogg, 2008;
Boyle, 2001; Ruhl, Hughes, and Schloss 1987). The reason why students learn more
information is due to the fact that they are given time to actively process the new
information and compartmentalize new ideas more effectively as they are being taught
(Kellogg, 2008). Other research studies evaluating students' achievement have
demonstrated that many strategies promoting active learning are comparable to lectures
in promoting the mastery of content but superior to lectures in promoting the
development of students' skills in thinking and writing (Smith et al., 2005; Prince, 2004;
21
Bonwell & Eison, 1991). However, students have a variety of learning styles and no
instructional approach can be optimal for everyone (Grasha, 1994; Felder, 1993; Claxton
& Murrell, 1987). Therefore, the importance to differentiate instruction becomes that
much more important as it relates to student learning.
Felder and Brent (1996) discuss some of the broad approaches to differentiating
instruction that could lead to more student engagement:
•
Substituting active learning experiences for lectures.
•
Holding students responsible for material that has not been explicitly
discussed in class.
•
Assigning open-ended problems and problems requiring critical or
creative thinking that cannot be solved by following text examples.
•
Involving students in simulations and role-plays.
•
Assigning a variety of unconventional writing exercises.
•
Using self-paced and/or cooperative (team-based) learning.
Other active learning pedagogies include cooperative learning, debates, drama,
role playing and simulation, and peer-to-peer teaching. In short, the published literature
on alternatives to traditional classroom presentations provides many different approaches
faculty could use as instructional tools (Bonwell & Eison, 1991).
Student engagement and active participation has shown to be helpful in promoting
long-term retention of information, motivating students toward further learning, allowing
students to apply information in new settings, and in developing students' thinking skills
(McKeachie et al., 1986). Research has suggested, however, that to achieve these goals
faculty must be knowledgeable of alternative techniques and strategies for questioning
22
and discussion (Hyman, 1980) and must create a supportive intellectual and emotional
environment that encourages students to take risks (Lowman, 1984). In other words,
faculty must be willing to try new methodology to engage students.
So why doesn’t every higher educational instructor strive to improve student
engagement in terms that were previously outlined? Bonwell and Eison (1991) speak of
some common barriers to instructional change. Included in their list are the powerful
influence of educational tradition, faculty self-perceptions and self-definition of roles, the
discomfort and anxiety that change creates, and the limited incentives for faculty to
change. However, there are certain specific obstacles that are associated with the use of
active learning. These include limited class time, a possible increase in preparation time,
the potential difficulty of using active learning in large classes, and a lack of needed
materials, equipment, or resources (Bonwell & Eison, 1991).
Perhaps the most significant barrier for instructors not being willing to modify
their instruction to fit a more participatory learning environment involves risk. They may
feel the risk that students will not respond to the active learning requirements that
demand higher-order thinking and open participation of students. In other words,
instructors who are trying to change their teaching methodology to a more active learning
environment take a risk that students will resist the change and not respond to the added
responsibilities associated with active learning. Some instructors may feel a loss of
control as they implement active learning strategies like peer-to-peer instruction. What’s
more, some instructors may even feel a lack of necessary skills required in fostering an
active learning environment. However, each of these barriers can be overcome with
23
careful planning, direction, and preparation (Bonwell & Eison, 1991). The LM at BYU-I
is designed to help faculty and students overcome some of these barriers (Gilbert, 2008).
A New Model for Learning
This section of the literature review is focused on the relevant literature
surrounding the process steps of the BYU-I’s LM—Prepare, Teach-One-Another, and
Ponder/Prove. For the most part, the LM is based on student engagement and active
participation. Facilitating an active learning environment is possibly the most effective
way to increase student engagement (Kuh, 2001). All three process steps focus on ways
of improving activity and engagement of students to increase their learning. The
literature review sections that relates to the LM are termed; Preparing for an Active
Learning Environment, Active Learning Pedagogy, and Critical Reflection.
Empirical literature is building in each area of the LM. One example of this
would be active learning, which has been studied by many authors, and is the basis of the
Teach One Another process step of the LM (Borman et al., 1993; Brockway, Mcneely
and Niffennegger, 1992; Dev, 1990; Hunt et al., 1985; Mallor, Near, & Sorcinelli, 1981).
The natural question that follows would be how one could implement this style of
teaching into their regular classroom instruction. Penner (1984) gives an answer to this
question when he suggests that the modifications of traditional lectures are ways to
incorporate this form of active learning in the classroom. The following literature will
outline the most relevant methodology that supports modification of lectures to make
room for active student engagement.
24
Preparing for an active learning environment. The LM’s first process step is
“Prepare”. This step is foundational to the integrity and efficacy of the LM. Beichner et
al. (2007) agree that student preparation is foremost in facilitating a cooperative learning
environment where students freely collaborate, question, and teach one another. Most
Universities preach the unwritten law of “two for one” as their academic mantra, meaning
“undergraduate students should spend at least two hours preparing for every class
hour…Unfortunately, most students spend only about half that amount of time” (Kuh,
2003, p. 27). Additionally, only a fifth of the first-year students through seniors
“frequently” come to class with any preparation. Meaning, that only 20% of the student
body come to class prepared to ask questions, discuss meanings, and/or contribute to the
learning environment by sharing experiences, stories, ideas, and thoughts that all have to
potential of increasing the level of learning for others in the classroom. What’s more,
these students say “their institutions give little emphasis to studying and spending time on
academic” preparation before class (Kuh, 2003, p. 27).
One pedagogical approach used to encourage student preparation is by inviting
students to begin learning about a topic before it is presented in class by reading
information that pertain to the topic. “Students are asked to focus on the key ideas and
express them in their own words before completing an assignment based on the reading.”
(Beichner et al, 2007, p. 5). Another methodology used to encourage pre-class
preparation is the use of online reading quizzes. These quizzes are designed to encourage
students to read the material before it is covered in class. In this way, students come to
class already prepared to participate in discussions and other activities (Beichner et al,
2007).
25
Active learning pedagogy. The second process step in The LM is termed TeachOne-Another. This type of teaching methodology is not new. In fact, “first-century
Roman rhetorician, stated that students could benefit by teaching one another” (Goyak,
2009, p. 31). Likewise, Johann Amos Comenius, a prolific reformer of pedagogy of the
early 1600s, believed that peer-to-peer teaching was beneficial to student learning
(Johnson et al., 1998). The concept of teaching one another is not new but it has taken
many different names and forms throughout the years. Forms of teaching one another
have been termed as active learning, peer-to-peer instruction, collaborative learning,
cooperative learning, inductive learning, and student-centered learning (Felder & Brent,
2006).
Active learning incorporates activities that engage students in doing something
besides listening to a lecture and taking notes to help them learn and apply course
material. Students may be involved in talking and listening to one another, or writing,
reading, and reflecting individually (Felder & Brent, 2006). “Active learning strategies
reach all types of learners in the visual, auditory, read/write, kinesthetic, and tactile
schemes. With active learning strategies, visual learners are targeted by the presence of
models and demonstrations” (Lujan & DiCarlo, 2006, p. 18). Auditory learners are
reached through discussion during peer instruction (Cortright et al, 2005; Rao & DiCarlo,
2000), debates (Scannapieco, 1997), games (Bailey et al, 1999; Mierson, 1999; Moy et al,
2000; Patil et al, 1993), and answering questions. Manipulating models (Rodenbaugh et
al, 1999; Silverthorn, 1999; Chan et al, 1991) and role playing (Kuipers & Clemens,
1998) satisfy kinesthetic and tactile learners. It is generally thought that students have
better retention and understanding of knowledge when taught by active as opposed to
26
passive methods (Matlin, 2003; Rao & DiCarlo, 2001; McKeachie, 1994; McDermott,
1993; Michael, 1993).
Lymna's (1981) Think-Pair-Share and Mazur's (1997) Peer Instruction are “peer
instruction activities that provide opportunities for students to be actively engaged in the
reasoning and application of concepts” (as cited by Lujan & DiCarlo, 2006, p. 20).
Studies have “documented that pausing two to three times during a 50-min class to allow
peer instruction of concepts enhanced the students' level of understanding and ability to
synthesize and integrate material” (Lujan & DiCarlo, 2006, p. 20). Specifically, peer
instruction enhanced the mastery of original material and meaningful learning (Lujan &
DiCarlo, 2006).
Collaborative learning is a subset of active learning in which students interact
with one another while they learn and apply course material (Felder & Brent, 2006).
Collaborative testing is an effective cooperative learning teaching strategy (Cortright et
al, 2003; Rao et al, 2002). To conduct a collaborative testing experience, students first
complete a quiz individually. Once the quiz is completed individually, students complete
the same quiz in groups. Specifically, students complete the quiz individually for the first
50 minutes immediately after completing the quiz individually, students are assigned to a
group of two or three and work as a team to answer the original questions; 80% of the
final quiz score is based on individual results, and 20% of the final quiz score is based on
group results. This collaborative group testing has been shown to increase student
learning (Rao et al, 2002) as well as student retention of previously learned material
(Cortright et al, 2003). For this study, the second process step of the LM ties in naturally
to collaborative groups and other cooperative learning techniques.
27
Cooperative learning is a form of collaborative learning in which students work
together on structured assignments or projects under conditions that assure positive
interdependence, individual accountability, periodic face-to-face interaction, appropriate
development and use of interpersonal skills, and regular self-assessment of group
functioning (Felder & Brent, 2006). When students learn with others, they have the
emotional and intellectual support that allows them to go beyond their present knowledge
and skills and accomplish shared goals (Silberman, 1996). In the cooperative learning
setting, students become more engaged in learning by discussing material, facilitating
understanding and encouraging hard work (Lujan & DiCarlo, 2006). In addition,
cooperative learning has positive effects on race relations, self-esteem, and a willingness
to cooperate in other settings (Slavin, 1983).
Sapon-Shevin and Schniedewind (1992) promoted the concept and purpose of
cooperative learning in the following statement:
Cooperative learning [is] more than a teaching strategy, more than an instructional
technique. Cooperative learning is an entirely different way of viewing the
educational process of schools, reshaping them into communities of caring in
which individual students take responsibility for the learning of their classmates
and respect and encourage each other’s diversity. Cooperative learning has the
potential to completely transform all aspects of your classroom and of your school
so as to promote the sharing of power, responsibility, and decision-making
throughout. (p. 16)
Cooperative learning argues that it is not enough merely to work with others or
just to work in groups within the class. Cooperative learning demands that all students
28
within a class actively and meaningfully engage in a discovery process of learning
(Johnson & Johnson, 1999).
Goyak (2009) suggests that the reward structure of a classroom can also be very
different from a traditional setting when viewed in light of cooperative learning
principles. He further states that an interpersonal reward structure highlights rewards by
how they are linked to students. “In a negative reward interdependence system,
competition in the classroom is emphasized. Competition in this framework refers to one
student’s success necessitating another student’s failure. Conversely, in a positive reward
interdependence system, cooperation within groups is emphasized” (Goyak, 2009, p. 9).
Goyak (2009) explains that one student’s success allows all group members to experience
success in the classroom.
Inductive learning has been referred to as problem-based learning, project-based
learning, guided inquiry, or discovery learning (Felder & Brent, 2006). These new
approaches to instruction present challenges to both teachers and students (Marx et al,
2004). For teachers using instructional methods based on recitation and direct instruction,
inductive teaching strategies challenges them to develop new content knowledge,
pedagogical techniques, approaches to assessment, and classroom management
(Blumenfeld, Marx, & Soloway, 1994; Edelson, Gordin, & Pea, 1999; Marx, Blumenfeld,
Krajcik, & Soloway, 1997)
Felder and Brent (1996) studied student-centered instruction (SCI) and found that
SCI differs from traditional instruction. In traditional instruction, the teacher's primary
functions are lecturing, designing assignments and tests, and grading; in SCI, the teacher
still has these functions but also provides students with opportunities to learn
29
independently and from one another and coaches them in the skills they need to do so
effectively. More of a focus is centered on peer-to-peer instruction and student’s
participation. In recent decades, the education literature has described a wide variety of
student-centered instructional methods and offered countless demonstrations that
properly implemented SCI leads to increased motivation to learn, greater retention of
knowledge, deeper understanding, and more positive attitudes toward the subject being
taught (Bonwell and Eisen 1991; Johnson Johnson and Smith 1991a,b; McKeachie 1986;
Meyers and Jones 1993).
Critical reflection. Critical reflection is a term used in academia that relates to
the third process step of the LM, the ponder/prove process step. Critical reflection,
sometimes grouped with transformative learning, involves the critical examination and
self-reflection of held personal beliefs to assess their validity, bias, and limitations in a
given context (Wolf, 2009). Mezirow (1990) proposes that critical “reflection can take
place within both instrumental and communicative learning realms” (p. 7).
Ponder/Prove, occurs after class and may involve individual or group activities. In
this step students ponder what they have learned in class. They keep a learning journal
and write down impressions and insights they have received. Students pursue unanswered
questions and discuss the class with their study group. This process step naturally
transitions back to the initial process step of preparing before class (BYU-I, 2007).
The process step is not only functional and helpful as a way of increasing student
learning, it is also a good practice for instructors. Further, each faculty member should
engage in self-reflection, exploring his or her personal willingness to experiment with
alternative approaches to instruction (Bonwell & Eison, 1991).
30
Course Evaluations
Course evaluations have been used for many years in higher education to measure
student perceptions regarding the course design and the instructors’ strengths and
weaknesses. Studies have also been conducted on the implementation, usefulness,
design, and application of course evaluations for a number of years. In the mid-eighties,
Marsh (1987) conducted a thorough review of the massive literature on the use of student
evaluations and their relation to teaching effectiveness. He identified nine dimensions
critical to effective instruction at the university level; workload, teachers' explanations,
empathy (interest in students), openness, and the quality of assessment procedures
(including quality of feedback) (March, 1987). He also found that lecturers rated
themselves higher than students did regarding good teaching at the university level.
A decade earlier, Feldman's (1976) developed a rubric of 19 categories of
instructional effectiveness that included; stimulation of student interest, teacher
sensitivity to class level and progress, clarity of course requirements, understandable
explanations, respect for students, and encouraging independent thought.
Another studies based on evaluation of course improvement rather than individual
instructor ratings have emerged. Entwistle & Tait (1990), for example, found that those
factors that best improve a course include; the provision of clear goals, appropriate
workload and level of difficulty, assignments providing choice, quality of explanations,
level of material and the pace at which it is presented, enthusiasm, and empathy with
students' needs.
Literature on student evaluation such as those of Marsh (1987), Feldman (1978),
and Roe & Macdonald (1983), together with the authoritative summaries of key findings
31
of validity studies (e.g. McKeachie, 1983; Centra, 1980) attest to the usefulness and
accuracy of student evaluation of instruction in comparison with other measures such as
peer evaluations (Ramsden, 1991, p. 135). By the time students find themselves in a
higher educational classroom, most have seen a great deal of teaching and many
understand which teaching approaches best maximize their own learning. As Ramsden
(1991) puts it, “nonexperts in a subject are uniquely qualified to judge whether the
instruction they receive is helping them to learn” (p. 137). Marsh (1987) reports students
are rarely misled into confusing good entertainment or performance with effective
teaching. Some instructors are good performers or entertainers, yet fail to teach
effectively the concepts and objectives of the course. Student evaluations are the means
whereby they evaluate effective teaching, and rarely are students mislead into thinking an
entertaining performer is always an effective teacher (Ramsden, 1991). However, an
effective teacher can also be a great entertainer. A fair and effective evaluation based on
performance and designed to encourage improvement in both the teacher being evaluated
and the college or university is a basic need of a good teacher evaluation system (Stronge,
1997). A comprehensive teacher evaluation system should be outcome oriented in that it
contributes to the personal goals of the teacher, to the mission of the program, to the
school, and to the educational institution, and should provide a fair measure of
performance (Stronge, 1997).
Studies have shown a positive relation between student evaluation and student
achievement, supporting the validity of student ratings. The many sources of potential
bias in student evaluations are minimized if exacting controls over methods of
administration are imposed and if students are asked only about those aspects of teaching
32
which they are qualified to comment upon. The general consensus is that there is no other
single measure of teaching performance which is as potentially valid (Ramsden, 1991).
Measuring effective instruction. Studies have shown aggregate-level
associations between course quality and perceived student learning using the instrument
of course evaluations (Entwistle & Tait, 1990; Ramsden & Entwistle, 1981; Ramsden et
al., 1989). These findings point to the usefulness and reliability of student evaluations in
comparison to other measures such as peer evaluations (Marsh, 1987; Feldman, 1978;
Roe & Macdonald, 1983). Ramsden (1991) also stated that because students see a great
deal of teaching, they are in an unrivalled position to comment on its quality. Moreover,
it shows that non-experts in a subject (i.e., students) are uniquely qualified to judge
whether the instruction they receive is helping them to learn. After all, students know if
they are benefiting from and learning as a result of a particular method or style of
teaching.
In summary, there is much literature available that supports the concepts
presented in the LM. Preparation before class is essential to creating a more active
learning environment in the classroom. Incorporated into the Teach One Another step of
the LM are a variety of teaching pedagogy and strategies that could be employed.
Cooperative learning, peer-to-peer instruction, and group work—just to name a few—
have been found to be effective instructional teaching strategies when correctly
implemented. Likewise, the third process step of the LM—the Ponder/Prove process
step—also has scholarly roots tied to it. In addition, literature also is prevalent in the area
of using end-of-course evaluations to measure effective teaching. This study will
33
investigate the implications of implementation of the three process steps of the LM on
end-of-course evaluations.
34
CHAPTER 3: METHODOLOGY
As mentioned previously, the purpose of this study is to examine the effects of the
LM on course ratings and student perceived learning as found on end-of-course
evaluations. The methodology used to collect the desired data for this study was
predominately through the compilation and interpretation of end-of-course student
evaluations. Course evaluations are archived at BYU-I and are considered public
information; they may also be viewed upon request. This chapter first details the
methods used to collect and compile data from end-of-course evaluations for both the old
evaluation instrument and the new evaluation instrument, both used to measure course
ratings and student perceived learning. The second section of this chapter specifies the
methodology and procedures used to address each of the three research questions
governing this study.
The methodology used in this study was determined based on the three research
questions, as previously outlined. The first research question asks if a change has
occurred in end-of-course ratings from the Fall 2007 semester (pre-LM implementation)
to the Winter 2010 semester, as the LM was introduced on campus? The purpose in
asking and addressing this question was to learn how end-of-course ratings have changed,
if at all, from the semester just prior to when the LM was introduced (Fall 2007) to the
latest semester in which data were collected for this study (Winter 2010). Remember, the
LM was first introduced during the Fall 2007 semester, but was not fully endorsed and
encouraged as a campus wide model for teaching and learning until the Winter 2008
semester. The purpose in asking research question one was to learn if the implementation
of the LM at BYU-I has had any significant affect on course ratings over time, from a
35
general perspective. Now admittedly, many variables can be contributed to the inflation
or deflation of end-of-course ratings from one semester to another. This chapter will
discuss such variables and will also address the methods used to control for confounding
variables associated with each research question.
Research question two states, “How have student perceived learning scores changed
based on course evaluation data collected from Fall 2007 to Winter 2010 semesters?”
The methodology used in examining this research question is similar to the methodology
used to address research question one, as mentioned above, with the exception of student
perceived learning scores being the focus. In other words, research question two
examines the changes that may have occurred in perceived student learning scores over
time.
Both the old evaluation instrument along with the new evaluation instrument
(introduced Winter 2009 semester) asks the same, identical question regarding student
perceived learning. Even though the two course evaluation instruments contain the same
question verbatim, regarding perceived learning, the questions preceding it are different.
The differences in question sets from the old evaluation instrument to the new evaluation
instrument could bias the way students answer the perceived learning question. Methods
to control for the difference in the two evaluation instruments are discussed further in this
chapter.
The third research question governing this study asks, “Is there a relationship between
LM scores and student perceived learning based on the new evaluation instrument
(beginning Winter 2009 semester)?” This question asks how strongly correlated are LM
scores with perceived learning? Remember, LM scores are derived from the new course
36
evaluation instrument which was introduced as a means of collecting data Winter 2009
semester. The LM had already been introduced campus wide for one year when the
evaluation instrument changed to reflect the three process steps of the LM. In this new
evaluation instrument, questions are asked and scores given based on students’
perceptions of the level at which the LM was applied throughout the course. LM scores
are then compiled based on this information. This third research question addresses the
relationship between LM scores and student perceived learning scores. Thus a
correlation methodology was employed to help answer this question. Also included in
this chapter are the control variables used to limit outside influences that may have biased
the study.
The Two Evaluation Instruments
For this study, it is important to understand that course evaluation data spans two
different evaluation instruments (see Appendix II and III). The differences of these two
evaluation instruments will be evaluated in this section. The purpose in doing so is
critical to the integrity of this study as research questions one and two involve a
comparison of data that were generated from both instruments. For research question one
and two, both evaluation instruments were employed as a means of tracking changes in
course ratings and student perceived learning scores. The differences between the two
evaluation instruments are important to consider when comparisons are being made.
As mentioned previously, the new course evaluation instrument was first used to
measure course ratings and student perceived learning for the Winter semester of 2009.
The evaluation instrument was changed in an attempt to emphasize the LM and to
37
highlight the importance of students taking personal responsibility for their own learning.
As a result, questions were changed and added to reflect this initiative, as explained in
Chapter 1. This is important to consider as a methodological parameter as it has played a
role in determining how the variables will be studied.
LM scores. Questions that were added to the new evaluation instrument were
specific to the way either the student or instructor had applied aspects of the LM
throughout the course. Questions were either rewritten or new questions were added that
specifically addressed the level of implementation of the three process steps of the LM—
Prepare, Teach One Another, and Ponder/Prove (see Appendix III). As a result, LM
scores were computed. These results are made available for the course instructor. These
scores reflect the level at which the LM was applied in a course as students perceived it
to be. For example, if the collective student population for a specific course rated both
themselves and the instructor as moderately implementing the Prepare aspect of the LM
throughout the semester the instructor would receive around a 70% LM score for the
course. LM scores are critical to this study as they will be used in determining the
relationship between LM scores and perceived learning scores. Specific LM scores are
only found in the new evaluation instrument, employed Winter 2009. For this reason,
research question three, which addresses the relationship between the two variables, only
uses data from the Winter 2009 semester to the Winter 2010 semester.
The questions that make up the LM scores are listed in Figure 7. Each question
makes up the overall LM score for that particular LM process step, as shown. For
example, there are four questions that relate to the prepare process step of the LM. The
38
prepare LM score for a course is based on how students answer each of the four questions
respectively.
Prepare
I was prepared for each
class.
I arrived at class on
time.
Teach-One-Another
Ponder/Prove
I was an active participant in
I sought opportunities to
online or face-to-face class
reflect on what I had
discussions.
learned in the class.
I sought opportunities to share
my learning with others outside
of class.
The course as a whole
has produced new
knowledge, skills, and
awareness in me.
I worked hard to meet
I feel that I made important
the requirements of this
contributions to the learning and
class.
growth of fellow classmates.
The instructor held me
The course provided
accountable for coming
opportunities to learn from and
to each class prepared.
teach other students
The instructor provided
opportunities to reflect
upon my learning and
experiences in the class.
Group work, if assigned, was
beneficial and meaningful.
Students were actively involved
in this class through discussions,
group work, and teaching.
The instructor provided
appropriate opportunities to be
an active participant in the class.
Figure 7. LM Question Set from the New Course Evaluation Instrument
39
For the third research question, LM scores serve as the independent variable. As
mentioned earlier, these scores are derived from the questions students are asked to
evaluate based on their own performance in, and instructor’s level of, implementing the
three LM process steps throughout a course. In other words, students are asked to rate
the level at which they perceived the presence of the LM both in their effort and
throughout the course as designed by the instructor. They may choose from the
following categories: Very Strongly Disagree (VSD), Strongly Disagree (SD), Disagree
(D), Agree (A), Strongly Agree (SA), and Very Strongly Agree (VSA). Points are
assigned to their choice with the following scale: 1=VSD, 2=SD…7=VSA (see Figure 8).
The way the LM scores are compiled is by taking the average score for each process step
based on student responses. For example, suppose ten students enrolled for a particular
course one semester, and for each of the Prepare questions found on the end-of-course
evaluation they each answered VSA. In this case, 7 points would be added to each of the
four questions pertaining to the Prepare element of the LM for this particular course.
Thus, the average would be 28 points out of a maximum of 28 points. The end-of-course
evaluation LM score for Prepare would be 100% since each student selected VSA for this
element.
See the following figure for a breakdown of all three LM process steps:
40
Number of
Questions
4
Maximum Points
From Evaluation
7
Teach-One-Another
7
7
49 Points
Ponder/Prove
3
7
21 Points
Process Step of the LM
Prepare
Total Points Possible
28 Points
Figure 8. LM Score Totals
Another important aspect of course evaluations is the manner in which they are
administered at BYU-I. Courses are evaluated based on instructor’s employment status.
At BYU-I, faculty are hired as adjunct (ADJ), one-year appointments (1YR), or PreContinuing Faculty Status (CFS). Faculty, once in the CFS track, can work towards
continuing faculty status (VET) or in other words a “tenured” faculty member. This
process takes 4 years to successfully earning VET status. To earn this status, one factor
is largely dependent on end-of-course ratings. ADJ, 1YR, and CFS are evaluated each
semester. Later in this chapter the methodology used to control for differences in
employment status will be addressed for each research question.
Research Questions and Methods
Each research question has a slightly different methodological approach and will
therefore be addressed one at a time. It is helpful to remember that questions one and two
of this study compare the differences between course evaluation scores and perceived
learning scores, respectively, over a specific time period. The semesters included for this
analysis are the Fall 2007 semester through to the Winter 2010 semester, a total of eight
41
individual semesters. For both previously mentioned questions the purpose in comparing
the two variables to an earlier point in time is to learn if a change has occurred in course
ratings or perceived learning scores as the LM has been implemented campus wide. The
methods and procedures used to collect data and to control for confounding variables are
outlined below.
Again, the purpose in asking the first research question is to establish whether end-ofcourse ratings have been affected during the time period when the LM was introduced
across campus. The second research question asks how student perceived learning scores
have been affected, if at all, by the implementation of the LM. The third research
question delves deeper into the strength of correlation between the LM scores derived
from course evaluations and perceived learning scores. Again, each question has its own
set of variables and methodology in gathering and analyzing the data. Each of the three
research question in this study will address the following: the purpose in asking each
research question, the variables measured and controlled for, the data collection
methodology employed to aid in answering each respective question, and the analysis
procedure used for each research question.
Research Question 1. The first research question asks, “How have end-of-course
ratings changed from the Fall 2007 semester (pre-LM implementation) to the Winter
2010 semester, as the LM was introduced on campus?” The purpose in asking this
question is to find out if the LM has had an effect on end-of-course ratings over time, as
the LM has been implemented by the University to further student learning and to
encourage a more participatory learning environment. Once again, the LM was
introduced to students and faculty beginning Winter 2008 semester. From that point on,
42
did course ratings increase, decrease, or stay the same? One might conclude that due to
the added emphasis on students taking more responsibility for their own learning, course
ratings might be adversely affected by students who prefer a more traditional approach in
receiving instruction (Felder & Brent, 1996). Thus begs the question, does application
of the LM positively or negatively affect end-of-course ratings?
As mentioned above, questions were added to the new evaluation instrument in
the Winter semester of 2009. The change in the evaluation instrument has impacted the
methodology used to address research question one. To control for the difference in
evaluation instruments the researcher collected data regarding course ratings for the four
semesters preceding Winter 2009 semester and data found in the four semesters after the
new evaluation instrument was employed. This was done to control for the differences in
the two evaluation instruments. Therefore, the four semesters previous to the Winter
2009 semester were evaluated independent of the four semesters following this time.
Again, the first four semesters, beginning with the Fall 2007 semester, represents the time
period where the old evaluation instrument was employed to survey students. This first
time period will be referred to as T1 from this point on. The second time period,
beginning with the Winter 2009 semester, represents the four semesters following the
change in the course evaluation instrument, as previously explained. This time period
will be referred to as T2.
In addition, a second variable addressed in this study was for new instructors; or
in other words, the natural change that occurs when an instructor teaches a course more
than one time. In general, instructors have a natural propensity, to change, adapt, and
adjust the content and approach of a course each time they teach it (Kagan, 1992). This
43
can be especially true when an instructor teaches a course for the first or second time. No
doubt these adjustments, no matter how subtle they may be, could affect end-of-course
ratings. In other words, instructors experience learning curves in their first few attempts
at teaching a course, which can affect end-of-course ratings. To control for this variable
the researcher collected data from only those faculty members who had taught each
respective course more than three times previous to their teaching it the Fall of 2007
semester. Comparing data in this manner will reduce the chance of inflation of course
evaluations due the learning curve that can be present with instructors teaching a course
in their first few attempts (Kagan, 1992).
Another consideration in addressing research question one is whether to compare
end-of-course evaluation scores for specific instructors who taught the same course in
specific semesters or to include data from all instructors over a broader time period. In
an attempt to control for conflicting variables, the researcher has chosen to look
specifically at instructors who taught the same course during the specific semesters in
question (i.e., from the Fall 2007 semester to the Winter 2010 semester). In this way, the
same instructor teaching the same course Fall of 2007 is compared with himself/herself
teaching the same course Winter of 2008, etc…. The reasoning behind this
methodological approach being that the change in scores will more closely reflect the
change in the use of the LM by the course instructor rather than other influences.
Another variable that was considered being a contributing factor in end-of-course
scores was class size. For instance, if a class increased in student count from the Fall
semester compared to the Winter semester or vice versa, the change may have affected
course ratings for that particular course. Class size can and does have an effect on
44
learning outcomes as shown in a study by Krueger (1999). Krueger measured gains on
standardized tests for more than 11,600 students in Tennessee. In his study Krueger
categorized students into small and regularly sized classrooms based on the number of
students in each. He found a significant gain in test scores for those students in a smaller
classroom compared with those in regular sized classroom.
To control for class size the researcher has disaggregated the data (based on
Krueger’s 1999 study) to the following class sizes; small, medium, large, and very large.
For this study, classes that had less than 10 students enrolled were categorized as a small
class, classes with 10-20 students were categorized as a medium class, classes with 20-30
students enrolled were categorized as a large class, and classes with more than 30
students enrolled were categorized as a very large class. For research question one and
two, only those instructors who consistently had the same amount of students throughout
the eight semesters were included in the study.
The statistical tool chosen to analyze the change in end-of-course ratings between
semesters was through a comparison of means and ANOVA. With the use of ANOVA
the difference in means of course ratings were measured and compared one with another.
A paired, two sample t-test was a second statistical tool that was also employed for this
research question. In addition, a significance value (P-value) was also calculated to
determine if the change in end-of-course ratings were considered to be statistically
significant. The significance level was pre-determined to be .05, as it was calculated
from the two-sample two-tailed t-test. Meaning, if a change in scores had occurred
between the two semesters and was calculated to be less than the .05 percentile, it was
45
considered to be a significant change. This analysis tool was used to measure the
difference in end-of-course ratings for both T1 and T2, respectively.
In summary, the focus of research question one relates to the change in course
ratings as the LM has been implemented. As mentioned, the variables that will be
controlled for are; the change to the evaluation instruments, learning curve of new
instructors, inconsistencies in different instructors teaching a particular course, and class
size. To control for the confounding variables associated with this question, the
researcher collected data from those instructors who have taught the same course each of
the eight semesters, from the Fall 2007 to the Winter 2010 semesters. Instructors who
had not taught the course at least three times previous to the Fall 2007 semester were
eliminated. Due to the change in evaluation instruments, the first four semesters will be
analyzed independent of the last four semesters. In other words, T1 represent the period
when the old evaluation instrument was being used to track course ratings. T1 was
examined independent of T2, representative of the period when the new evaluation
instrument was employed to measure course ratings. Class size was also factored in,
meaning only those instructors who taught small, medium, large, or very large classes
consistently over the four semester time frame were included. As a result of controlling
for these variables, the data set was reduced from 850 courses down to 33 courses. The
following chapter will address the findings for research question one as outlined above.
Research Question 2. The second research question is much like the first in that
data was collected in an attempt to learn how the implementation of the LM affected, if at
all, student perceived learning. In this case, the second question specifically addresses
the change in student perceived learning over time. The question is stated in this way,
46
“How have student perceived learning scores changed based on course evaluation data
collected from Fall 2007 to the Winter 2010 semesters?” The purpose in asking this
question is to learn if students, in general, perceive that they have learned more in their
courses—compared to other college courses they have taken—as the LM has been
introduced at the University level.
In both the new course evaluation instrument and the old evaluation instrument
the perceived learning question remained the same; “Compared to other college courses
you have taken, would you say that you have learned….” The students then select either
“a great deal less”, “a little less”, “about the same”, “a little more”, or “a great deal more”
based on their experience. On both evaluation instruments students are asked to rate their
learning as it compares with other classes taken. This information was used to determine
the change, if any, in student’s perceived learning between the Fall 2007 and the Winter
2010 semesters. The student perceived learning score is derived from the above
mentioned question found on both course evaluation instruments (see Appendix II & III).
If a student answers that they have learned about the same amount when compared with
other college courses taken then the instructor would receive a 0 score, per that students’
answer. All students perceived learning scores are averaged for each course. In other
words, if an instructor received a 1.49 student perceived learning rating for a particular
course this would mean that the average student perceived learning for that course was
between “a little more” and “a great deal more” as compared with other college courses
taken.
As mentioned, both evaluation instruments contain the same question regarding
student perceived learning. Yet, the questions leading up to this one are different on the
47
pre-LM instrument compared with the new evaluation instrument, and could affect how
the latter is answered. More specifically, the new course evaluation includes questions
related to the level at which students were responsible for their own learning; “I was
prepared for each class”, for example. These questions may influence the way students
answer the perceived learning question found at the end of the evaluation. To control for
this variable, the researcher collected student perceived learning for each semester,
beginning with the Fall 2007 semester and each semester thereafter, up to the Winter
2010 semester. Attention was given to the difference in student perceived learning scores
between semesters. However, the first four semesters were compared one to another
independent of the last four semesters, and vice versa. This was done to reflect the
semesters when the old evaluation instrument was used compared with the semesters
when the new evaluation instrument was used to survey students.
Similar to the methodology employed for the previous research question, the
researcher compared only those semesters together in which the evaluation instrument
was unchanged. Again, T1 semesters were compared together, as were T2 semesters.
Thus, eliminating the possible effects associated with the use of two different course
evaluations instrument to measure student perceived learning scores. The same variables
were controlled for to measure the change in perceive learning scores as were used to
measure the change in course ratings. This resulted in the same data set of 33 courses
evaluated regarding changes in perceived learning scores over the two time periods—T1
and T2.
A comparison of means and ANOVA were also employed as the statistical
method for analyzing the change in mean scores for student perceived learning. The
48
differences between the means were explained. Likewise, a p-value was employed to
measure the level of significance between the differences. A small p-value indicates that
the probability of the difference in end-of-course perceived learning scores changing due
to chance unlikely. A t-test was then employed to measure the level of significance
between the difference of means for T1 semesters, and again for T2 semesters. The
significance level was pre-determined to be .05 or less. In other words, the change from
the time the LM was first introduced to the time the course evaluation was changed will
be analyzed independent of the time the new course evaluation was introduced to the
most recent semester.
Research Question 3. The third research question states, “Is there a relationship
between LM scores and student perceived learning based on the new evaluation
instrument (beginning Winter semester 2009)?” Another way to ask this question would
be to say, “Does evidence of LM application in a course help predict student perceived
learning scores for a particular course?” The purpose in asking this question is to identify
the relationship of the LM on perceived learning scores. Do scores drop, increase, or stay
the same as the level of implementation of the LM changes, for example. Perhaps, the
most telling information that can be gained in asking this question is whether or not
students perceive themselves learning more when LM process steps are employed by an
instructor. In other words, do students perceive themselves as learning more information
when an instructor employs the three process steps of the LM? This is the purpose of
research question three, to learn the relationship between LM scores and perceived
learning scores.
49
Perceived learning scores can be influenced by a variety of variables, as mentioned
previously. However, for this question LM scores are used as the independent variable
whereas perceived learning scores serve as the dependent variable. In other words, the
researcher is interested in learning how the level of implementation of the LM correlates
with students’ perceived learning for a particular course. Furthermore, the researcher has
broken down the data to more specific variables. These variables include the relationship
between LM scores and perceived learning scores as they relate to; (1) class size (number
of students enrolled in the course), (2) semester in which the student was enrolled, and
(3) instructor employment status (i.e., adjunct, one-year appointment, Pre-CFS (nontenured), and CFS (tenured)).
As mentioned previously, LM scores are derived from the new evaluation instrument
and are based on student perceptions of the level at which the three process steps of the
LM were infused throughout the course. The data set for this question comes from
student evaluations of individual courses during T2. The researcher disaggregated the
information from course evaluations to draw out the previously mentioned variables.
All data sets found in T2 (the last four semesters of this study) were incorporated in
the findings. Remember research question three explores the relationship between LM
scores and perceived learning scores given particular variables such as; class size,
semesters taken, and instructor’s employment status. LM scores are a product of the new
evaluation instrument and are therefore only available for the last four semesters of this
study. When the evaluation instrument was changed the LM had already been introduced
to students and faculty for one year (three semesters). All perceived learning scores and
50
their associated LM scores for T2 were used to measure the strength of correlation
between these variables.
To find this relationship between LM scores and perceived learning scores the
researcher first compared mean perceived learning scores by semester and employment
status for all data points in T2. The relationship between the independent variable with
the dependent variables, as previously explained, was evaluated using a correlation
matrix. Data points were collected and analyzed using SPSS as the statistical analysis
tool. R-values and P-values were calculated to show the strength of correlation and the
level of significance for each set of variables. Results and findings are found in the
following chapter.
Limitations
Limitations of this study include the consistency of data and regularity of course
evaluations collected for instructors. Untenured faculty members including adjunct, oneyear appointments, and Pre-CFS are usually evaluated each semester for their first four
years of teaching. Tenured faculty members who have earned CFS are only evaluated
once every three years, unless requested by themselves or by one of their supervisors.
Due to this method for evaluating faculty members, the numbers of qualifying courses for
this study are limited.
Another limitation of this study relates to the fact that the faculty is ever
changing. Faculty members have been hired at different times and at different capacities
(i.e., Full-time, adjunct, or temporary status). As a result of faculty being ever changing,
data will likely reflect this pattern. BYU-Idaho has experienced tremendous growth since
51
the announcement of Ricks Jr. College becoming a four year university. Each year, an
increase in the number of new faculty members hired has gone up, to keep up with the
student growth. This again has affected the methodology for this study as controls have
been put in place, as previously mentioned to control for new instructors.
Faculty members often change their teaching methodology and strategies from
semester to semester based on previous findings, student’s needs, workload, and a variety
of other reasons. Determining the actual reasons for increased end-of-course ratings,
instructor ratings, and perceived learning can be a difficult task. Even though many of
the variable that could affect the viability of this study have been controlled, one
significant limitation must be recognized; students differences from one semester to
another. Students’ attitudes, learning styles, and personalities can change significantly
semester-by-semester. This limitation will not be accounted for in this study; only the
collective perceived answers given by students as it relates to end-of-course evaluations
have been evaluated and compared.
Lastly, but perhaps most significant, is the fact that the student population
changes each semester. New students enter each semester with new learning styles and a
variety of experiences. The demographics of each semester can also change dramatically
from semester to semester. Likewise, class sizes fluctuate between semesters.
As this is an Ex Post Facto study, some of the limitations mentioned are not
possible to measure or control for. However, those variables that are controllable have
been considered and methodology to control for such variables has been previously
mentioned.
52
FINDINGS
In this chapter, data pertaining to each research question were analyzed and
findings were discussed. This study was conducted in an attempt to determine the
significance of implementing and applying the three process steps of the LM (Prepare,
Teach One Another, and Ponder/Prove) relative to course ratings and perceived learning
scores, as measured from course evaluations. Findings associated with each research
question are addressed in this chapter.
Research questions one and two are similar in methodology, in that the researcher
took a large data set and disaggregated it down to a smaller fraction to control for
confounding variables. As a result, 33 courses were used in determining the change in
course ratings and perceived learning scores over time. Remember, research question
one asks, from the Fall 2007 semester up to the Winter 2010 semester, how have course
ratings changed over the eight semesters time period? In the second research question,
the change in student perceived learning for the same time frame was the focus. Both
research question one and two employed a comparison of means, ANOVA, and a two
sample t-test to measure the differences in mean scores.
Research question three looks at a variety of variables that may contribute to the
level of correlation between LM scores and student perceived learning scores. As
mentioned in Chapter 3, the three variables that are measured relative to the third
research question are; class size, semester when courses were taken, and instructor
employment status. By disaggregating the data to class size, semesters, and instructor
employment status the degree at which these variables affect the strength of correlation
53
between LM and student perceived learning scores were evaluated and analyzed
respectively.
Research Question 1: A Change in End-of-Course Scores
The first research question in this study asks if a change was noticed in course
ratings beginning Fall 2007, representing the semester preceding the implementation of
the LM. In other words, the LM was fully introduced to students and faculty before the
second semester (Winter 2008) of this study. Remember, there are a total of eight
semesters examined throughout the study. The first semester represents the time before
the LM was fully implemented, while the remaining seven semesters are reflective of the
period when the LM was introduced and encouraged as a learning and teaching
methodology for student and faculty campus wide.
As mentioned previously, the evaluation instrument used to measure course
ratings was changed beginning Winter semester, 2009. For this reason, course ratings for
T1 will be compared one with another. Likewise, end-of-course ratings for T2 will be
evaluated. Remember, T1 represents the semesters (Fall 2007 to Fall 2008) where the old
evaluation instrument was used to collect data from student evaluations. T2 represents
the semesters (Winter 2009 to Winter 2010) in which the new evaluation instrument was
used for the same purpose. The methodology used to measure the mean differences in
course ratings were primarily through a comparison of means using ANOVA and a
paired, two-sample mean t-test, as previously explained.
Have course ratings change over time? For question one, the researcher collected
end-of-course ratings for all courses from the Fall 2007 semester to the Winter 2010
54
semesters. The data were further disaggregated to only those instructors who had taught
the same course in each semester, consecutively. The researcher found 33 courses that
met the criteria. Course ratings for the 33 courses were compared and analyzed.
A correlation matrix was used to evaluate the mean differences in course ratings
for each semester (see Appendix IV). This revealed that each semester’s course ratings
are closely correlated. In other words, no semesters revealed a significant deviation from
the other semesters.
Furthermore, a breakdown of each individual course by semester was given in a
comparison of means table. Table 1 shows those qualifying courses for this study; each
course is categorized by department. Notice the difference in individual course ratings
for each individual course over the eight semesters. For the most part, increases in course
ratings between semesters were realized over time. When comparing course ratings for
Fall 2007 with Fall 2008 course ratings 21 out of the 33 courses showed gains, one
course stayed equal, and 11 courses showed a drop in courses ratings.
55
Table 1.
Course Ratings of Qualifying Courses for T1Semesters
Departments
Accounting
Agribusiness
Animal Sciences
Arch & Const
Art
Biology
Communications
Computer Science
English
Exercise and
Sports Sciences
Foreign Languages
History
Home and Family
Mathematics
Music
Psychology
Sociology
Inst.
ID
146
146
267
266
727
888
471
471
225
228
743
931
889
717
297
203
767
722
722
722
722
367
202
470
836
508
508
111
823
534
245
245
245
Fall
2007
4.37
4.61
5.68
5.60
6.33
5.00
5.24
5.75
5.63
6.18
5.95
6.22
5.00
4.37
6.17
5.34
4.50
6.14
6.30
6.05
5.81
5.53
5.07
5.95
4.38
5.58
5.33
7.00
4.79
4.76
5.57
4.68
5.46
Course
ACCTG 202
ACCTG 312
AGRON 122
AS 347
CONST 430
ART 130
B 461
B 478
B 466
COMM 150
COMM 230
COMM 130
COMM 100
COMM 316
COMM 102
CS 124
ENG 316
ESS 160
ESS 161
ESS 162
ESS 264
SPAN 101
HIST 201
CHILD 210
CHILD 335
HFED 100
HFED 350
MATH 100A
MUSIC 209
PSYCH 302
SOC 111
SOC 340
SOC 360
56
Winter
2008
4.71
4.29
5.70
6.29
6.09
5.19
5.53
6.10
6.00
6.26
6.47
6.20
4.94
4.67
5.77
5.59
4.87
6.29
6.47
5.93
5.91
5.38
4.94
5.47
4.67
5.74
5.24
6.29
4.54
5.43
5.66
4.43
4.91
Spring
2008
5.15
4.78
6.00
6.80
5.86
5.68
5.90
6.10
6.67
5.86
6.29
6.00
4.58
4.75
5.96
5.82
5.11
6.10
6.20
6.10
5.54
4.89
4.48
5.77
5.41
5.86
5.90
5.50
4.50
4.87
5.06
5.20
5.40
Fall
2008
4.60
4.61
5.54
6.43
5.83
5.22
5.91
6.36
6.42
6.59
5.94
6.52
4.75
4.52
6.23
6.22
5.55
6.18
6.25
6.18
6.25
5.00
5.00
5.80
6.00
6.12
5.38
5.95
4.62
5.12
5.50
4.95
6.00
Mean
Score
4.71
4.57
5.73
6.28
6.03
5.27
5.64
6.08
6.18
6.22
6.16
6.23
4.82
4.58
6.03
5.74
5.01
6.18
6.31
6.07
5.88
5.20
4.87
5.75
5.12
5.82
5.46
6.18
4.61
5.04
5.44
4.82
5.44
Also, notice the mean scores for the 33 courses in question. The two
lowest mean course ratings were ACCTG 312 and COMM 316. The two highest mean
course ratings were ESS 161 and AS 347. Interestingly, the two courses that scored
lowest on course ratings were found to be in the Accounting and Communication
departments, respectively. While the two highest mean course ratings were from the
Exercise and Sports Sciences and Animal Sciences departments. The courses that scored
lowest are typically taken to fulfill a general education requirement, whereas the courses
that scored the highest normally are taken to satisfy a major course requirement.
Other observations noticed as a result of comparing course ratings for T1were the
quantity of courses that showed consistent gains over the four semester period. There
were six courses that showed consistent gains in course ratings from one semester to the
other over time. Those six courses include B461, B478, CS 124, ENG 316, CHILD 335,
and HFED 100. There was only one course found of the 33 total courses in which a
consistent drop in course ratings was noticed over the four semesters, CONST 430. In
other words, six of the 33 courses showed consistent gains in course ratings over the four
semester period, whereas only one course showed consistent losses in course ratings for
the same period.
Once again, T1 represents the first four semesters of this study. In these
semesters the old evaluation instrument was used to collect course rating data. The first
semester (Fall 2007) is representative of the last semester preceding the implementation
of the LM. The next three semesters in T1 represent the first three semesters when the
LM was introduced and application of said model encouraged campus wide.
57
Course rating data for T2 was also collected and organized in a similar manner.
T2 represents the time period when the new evaluation instrument was employed to
gather end-of-course data. Remember, T2 includes the last 4 semesters of this study.
Due to the fact that the two evaluation instruments are somewhat different as it relates to
the question sets, independent analyses where conducted for each time period. Table 2
shows course ratings for each of the 33 courses during T2.
58
Table 2
Course Ratings of Qualifying Courses for T2 Semesters
Departments
Accounting
Agribusiness
Animal Sciences
Arch & Const
Art
Biology
Communications
Computer Science
English
Exercise and Sports
Sciences
Foreign Languages
History
Home and Family
Mathematics
Music
Psychology
Sociology
Inst
ID
146
146
267
266
727
888
471
471
225
228
743
931
889
717
297
203
767
722
722
722
722
367
202
470
836
508
508
111
823
534
245
245
245
Course
ACCTG 202
ACCTG 312
AGRON 122
AS 347
CONST 430
ART 130
B 461
B 478
B 466
COMM 150
COMM 230
COMM 130
COMM 100
COMM 316
COMM 102
CS 124
ENG 316
ESS 160
ESS 161
ESS 162
ESS 264
SPAN 101
HIST 201
CHILD 210
CHILD 335
HFED 100
HFED 350
MATH 100A
MUSIC 209
PSYCH 302
SOC 111
SOC 340
SOC 360
Winter Spring
2009
2009
4.63
4.54
4.25
4.39
5.68
5.29
6.67
6.45
6.07
6.20
6.11
5.24
6.13
5.71
6.63
6.00
5.92
6.00
6.35
6.19
5.67
6.17
6.45
6.08
5.38
4.53
4.76
5.30
6.29
6.06
6.08
6.00
5.16
5.02
6.18
6.31
6.69
6.54
6.26
6.25
6.10
6.38
5.24
6.00
5.33
5.03
6.18
6.07
5.29
4.92
6.07
5.86
5.74
5.93
6.57
5.50
4.52
5.35
4.77
5.56
5.47
5.67
5.16
4.84
5.00
5.93
59
Fall
2009
4.81
4.59
5.96
6.60
6.11
6.00
6.15
6.62
5.62
6.23
6.00
6.02
4.81
4.50
6.20
5.39
5.63
5.83
6.58
6.53
6.70
5.80
5.00
6.13
6.00
6.19
5.89
6.00
5.32
5.75
5.73
5.27
5.41
Winter Mean
2010 Score
4.87
4.71
5.38
4.65
5.70
5.66
6.50
6.56
6.24
6.15
5.43
5.70
5.81
5.95
6.05
6.32
5.75
5.82
6.24
6.26
5.54
5.84
6.21
6.19
5.60
5.08
5.60
5.04
6.26
6.20
6.07
5.89
5.36
5.29
5.77
6.02
6.57
6.60
6.13
6.29
6.55
6.43
5.59
5.66
4.43
4.95
6.10
6.12
6.00
5.55
6.06
6.05
5.60
5.79
6.36
6.11
5.15
5.08
5.48
5.39
5.31
5.54
5.12
5.10
5.08
5.35
During T2, it seems that a leveling off occurred for each of the 33 courses in
question. Only one course (ACCTG 312) showed consistent gains in course ratings for
each of the four semesters in T2. Likewise, there was only one course (HIST 201) that
showed consistent drops in each consecutive semester during the same time period.
Interestingly, ACCTG 312—the one course that showed gains in each of the four
semesters during T2—had the lowest mean score for T1. This being said, the two lowest
mean scores in T2 were found to be ACCTG 202 and ACCTG 312. Even with consistent
gains in ACCTG 312, this course went from having the lowest mean course ratings of the
33 courses being evaluated to next to lowest mean rating. What’s most curious about this
course is the jump in course rating for Winter 2010 semester. What was different about
this semester that caused the change? Was it more use of the LM? Research questions
two and three will examine further these inquiries.
The two highest course ratings were again found for ESS 161 and AS 347. In
fact, mean course rating scores for both ESS 161 and AS 347 for T1 and T2 respectively,
show an increase from one period to another. ESS161 showed a gain in mean score from
T1 to T2 of .29 points. Likewise, AS 347 course showed gains from 6.28 to 6.56 during
the same time, a difference of .28. The high ratings and evidence of continued improved
scores are cause for further investigation, as this study relates to the affects of LM
application on course ratings and student perceived learning. Are LM scores associated
with higher course ratings? Again, question three will address this relationship.
Once individual courses were examined using a comparison of means analysis the
researcher used a one-way ANOVA test to measure the differences in the combined
means for T1 and T2. In other words, an average course rating of all 33 courses were
60
computed for each semester. This was done in an attempt to compare the differences in
mean course ratings for the entire 33 courses by semester. The researcher wanted to learn
if course ratings improved over time when the 33 courses were grouped into one body.
Again, T1 represents the semesters previous to the change in the course evaluation
instrument (from Fall 2007 to Fall 2008 semesters). T2 represents the semesters when
the new course evaluation was employed (from Winter 2009 to Winter 2010 semesters).
Table 3 shows the results of the one-way ANOVA test for T1 and T2. Notice the
differences between the means for each semester.
Table 3
ANOVA: End-of-Course Ratings for T1& T2 Semesters
Evaluation Instrument
Standard
Semester
N
Course Rating
Employed
Deviation
Fall 2007
33
5.46
.45
Old Evaluation
Winter 2008
33
5.51
.41
Instrument (T1)
Spring 2008
33
5.57
.37
Fall 2008
33
5.68
.41
Winter 2009
33
5.72
.46
New Evaluation
Spring 2009
33
5.67
.35
Instrument (T2)
Fall 2009
33
5.79
.34
Winter 2010
33
5.75
.26
*Indicates a significance level at or less than .05 between semesters
61
In five out of the seven opportunities for course ratings to change an increase in
average course ratings was realized. Only in two semesters did the average course rating
drop for the composite body of courses. Although the average means for each semester
were different, the significance level was measured at .55 between each semester. This
indicates that no significant changes in end-of-course ratings were found between
semesters during the T1 time period. Or in other words, no significance was found for
the change in average course ratings from one semester to another (i.e., from Fall 2007
semester to the Winter 2008 semester). Again, for T2 no significant differences between
semesters were found. The P-value for end-of-course ratings for T2 was measured at .86,
clearly showing no significance.
However, notice the increase in course ratings from Fall 2007 semester to Winter
2010 semester for the 33 courses in question. Gradually, the end of course ratings
increased from one semester to another over the time period when the LM was introduced
and implemented campus wide. Remember, the LM was informally introduced just prior
to the Fall 2007 semester. It wasn’t until the next semester, Winter 2008, when the LM
was fully encouraged to be implemented by students and faculty throughout the campus.
The three semesters following the Fall 2007 semester, representing the time period when
the old evaluation instrument was employed, realized a consistent increase in course
ratings.
From the Fall 2007 semester to Fall 2008 semester, average end-of-course ratings
improved from 5.46 to 5.68, more than .2 points increase. However, the improvements in
end-of-course ratings for T2 were not as profound as in T1. The findings reveal a
average course rating for Winter 2009 of 5.72 and a mean course rating for Winter 2010
62
of 5.75, a little more than .03 point difference. A two-sample t-test was used to measure
the level of change and to calculate the significance level of the change for T1 and T2.
The hypothesized change was selected to be zero, meaning that no changes in course
ratings were anticipated.
The results of this t-test showed that the change in course ratings for T1 was
significant (see Appendix V). Remember, the significance level of this study was set at
the .05 level. For T1, the findings reveal that the increase in end-of-course ratings for T1
measured .019. A similar t-test was conducted for T2. Taking the beginning semester
(Winter 2009) and comparing it to the most recent data available for this study (Winter
2010), revealed a P-value of .66 (see Appendix V).
The significance value for T2 is very large at .66, revealing no significance in
the differences between mean end-of-course ratings for this time frame. The findings
show that during the T1 semesters a steady increase was apparent for the 33 courses
being measured. But, in T2 semesters the change in end-of-course ratings showed no
significance between Winter 2009 and Winter 2010 semesters. The change in course
ratings for T1 was found to be significant, whereas the change in course ratings for T2
was not significant.
The difference between end-of-course ratings for Fall 2008 semester and Winter
2009 semester was also evaluated to see if the change in course evaluation instrument had
any significant affect on course ratings. The Fall 2008 semester showed a mean course
rating for the 33 courses under investigation measuring 5.68 points. Whereas, the Winter
2009 semester showed a mean course rating of 5.72 for the same courses. The
differences of mean course ratings between Fall 2008 semester and Winter 2009 semester
63
measured less than .04 points, not a noticeable change. Likewise, using a two sample ttest the significance value between these two semesters measured .57, revealing no
significance (see Appendix V). Remember, the Fall 2008 semester represents the last
semester in which the old evaluation instrument was used to measure end-of-course
ratings. Also, the Winter 2009 semester represents the first semester in which the new
evaluation tool was employed to gather end-of-course ratings, among other data. This
lack of significance between the Fall 2008 and Winter 2009 semester is surprising. The
change in evaluation instruments did not seem to have much of an effect on course
ratings between the two semesters in question.
As a result of finding no difference between course ratings between T1 and T2 a
comparison of means for the beginning semester of this study with the last semester of
this study was conducted. The difference in mean course ratings for the Fall 2007 and
the Fall 2010 semesters were evaluated using the same two sample t-test. The same 33
courses were used in this evaluation to keep with methodological criteria previously
established for this research question. The purpose in comparing the Fall 2007 semester
with the Winter 2010 semester was to learn the significance of the change in course
ratings from the semester before the LM was implemented to the latest semester. In other
words, for the average mean course ratings of the 33 courses in question, what changes
have occurred over the last eight semesters and was this change significant?
When course ratings from the first to the last semester are compared a significant
difference in mean scores is realized. The findings show a significant change between
composite course ratings over time. The significance level for this two-tailed t-test was
measured at the .003 level, meaning that there is roughly a 99.997% chance that the
64
difference between the two end-of-course mean ratings did not occur by chance alone
(see Appendix V). The mean rating for Fall 2007 semester was measured to be 5.46,
compare that with the mean score for Winter 2010 of 5.75. With an n of 33, this
difference becomes significant between the two semesters.
What does this tell us? For one, it shows that some factor has caused or aided this
increase in scores from one semester to another. However, it doesn’t help us understand
what that factor was. Many variables may have played a role in causing this increase in
course ratings and only one of those contributing factors might be the implementation of
the LM. Research Question three deals with data that may help us understand to what
level the LM may have been a contributing factor in raising end-of-course scores for
these instructors.
Research Question 2: A Change in Perceived Learning Scores
The second research question governing this study asks how student perceived
learning scores have changed during the time the LM was introduced and implemented
campus wide. Perceived learning scores are evaluated and calculated with a similar
methodology used to evaluate the previous research question regarding course ratings.
The purpose in this evaluation is to learn if students’ perceived learning scores have
increased, decreased, or if the scores stayed the same in relation to the time the LM was
implemented campus wide (between the Fall 2007 and Winter 2010 semesters).
The data used to find the differences between perceived learning scores for T1
and T2 were compiled using the same methodology as in the first research question.
Instructors were identified who taught the same courses for each of the eight semesters,
65
spanning the Fall 2007 semester through to the Winter 2010 semester. The differences in
class sizes between semesters were controlled for and beginning instructors (those who
have taught the course less than 3 times previous to the Fall 2007 semester) were
eliminated from the data set. In controlling for such variables, the remaining qualifying
data set revealed 33 courses that met all criteria previously address in Chapter 3. Again,
the 33 courses used to answer question two, regarding the change in student perceived
learning over time, are the same courses used in research question one.
In the previous section, the findings show that course ratings increased during T1,
T2, and from Fall 2007 to Winter 2010 semesters, as reported above. But, how has
students’ perceived learning changed, if at all, during T1, T2, and between the Fall 2007
to the Winter 2010 semesters? This section will attempt to compile and compute data
that will help in answering this question.
A correlation matrix was used to quickly compare perceived learning scores for
each of the eight semesters of this study (see Appendix VI). In each case, the correlation
values of perceived learning scores for each semester were found to be significantly
correlated. In other words, none of the semesters showed extreme differences in
perceived learning scores. Each of the perceived learning scores for the eight semesters
was found to be homogeneous one with another.
Individual perceived learning scores were compiled using a comparison of means
table. Table 4 shows the results of individual perceived learning scores for each of the 33
courses for T1. Remember, the scores may range from negative two to positive two.
66
Table 4.
Perceived Learning Scores for T1 by Semester
Departments
Accounting
Agribusiness
Animal Sciences
Arch & Const
Art
Biology
Communications
Computer Science
English
Exercise and Sports
Sciences
Foreign Languages
History
Home and Family
Mathematics
Music
Psychology
Sociology
Inst
ID
146
146
267
266
727
888
471
471
225
228
743
931
889
717
297
203
767
722
722
722
722
367
202
470
836
508
508
111
823
534
245
245
245
Course
ACCTG 202
ACCTG 312
AGRON 122
AS 347
CONST 430
ART 130
B 461
B 478
B 466
COMM 150
COMM 230
COMM 130
COMM 100
COMM 316
COMM 102
CS 124
ENG 316
ESS 160
ESS 161
ESS 162
ESS 264
SPAN 101
HIST 201
CHILD 210
CHILD 335
HFED 100
HFED 350
MATH 100A
MUSIC 209
PSYCH 302
SOC 111
SOC 340
SOC 360
67
Fall
2007
0.64
0.19
1.21
1.89
1.38
1.43
0.81
1.30
1.26
1.38
1.38
1.78
0.62
0.37
1.06
1.11
0.17
0.91
0.77
0.57
0.94
1.41
0.93
0.74
1.00
1.00
0.44
2.00
0.59
0.75
1.00
1.08
1.39
Winter Spring Fall Mean
2008
2008
2008 Score
0.71
1.32 0.73
.85
-0.14
0.68 0.10
.21
1.00
1.55 0.86 1.16
1.71
2.00 2.00 1.90
1.35
1.20 1.17 1.28
1.21
1.64 1.33 1.40
1.20
1.50 1.55 1.27
1.60
1.90 1.68 1.62
1.50
1.89 1.63 1.57
1.07
1.05 1.51 1.25
1.50
1.53 1.13 1.39
1.58
1.53 1.48 1.59
0.58
0.00 0.07
.32
0.25
0.39
-.16
.21
0.87
1.36 1.00 1.07
1.17
0.30 1.50 1.02
0.35
0.44 0.47
.36
1.00
0.71 1.04
.92
0.94
0.48 0.50
.67
0.60
0.60 0.73
.63
1.09
0.92 1.25 1.05
0.79
1.58 1.42 1.30
0.97
0.67 0.46
.76
0.75
1.07 0.91
.87
1.20
1.35 1.50 1.26
1.29
1.37 1.13 1.20
0.48
0.95 0.81
.67
1.75
2.00 1.60 1.84
0.57
1.17 0.50
.71
1.00
1.13 0.97
.96
1.04
0.56 0.88
.87
0.90
1.29 1.23 1.13
1.09
1.38 1.35 1.30
The findings show that the two courses with the lowest mean perceived learning
scores for T1 are the same courses that had the lowest course ratings for the same time
period. Both ACCTG 312 and COMM 316 scored a .21 for perceived learning. This
shows that a correlation between course ratings and perceived learning scores may exist.
Likewise, AS 347 had the highest mean perceived learning score and course rating for
T1.
What about for T2? Table 5 shows the result of perceived learning scores by
semester for T2. Again, AS 347 showed the highest perceived learning score for T2.
Also, ACCTG 312 and COMM 316 accounted for two out of the three lowest perceived
learning scores for T2.
68
Table 5.
Perceive Learning Scores for T2 by Semester
Departments
Accounting
Agribusiness
Animal Sciences
Arch & Const
Art
Biology
Communications
Computer Science
English
Exercise and Sports
Sciences
Foreign Languages
History
Home and Family
Mathematics
Music
Psychology
Sociology
Inst
ID
146
146
267
266
727
888
471
471
225
228
743
931
889
717
297
203
767
722
722
722
722
367
202
470
836
508
508
111
823
534
245
245
245
Winter Spring Fall Winter
Course
2009
2009 2009 2010
ACCTG 202
0.53
0.90 0.73
0.69
ACCTG 312
0.19
0.25 0.24
0.71
AGRON 122
1.03
0.80 1.00
1.09
AS 347
1.89
1.80 1.80
1.80
CONST 430
1.34
1.47 1.33
1.39
ART 130
1.68
1.38 1.74
1.21
B 461
1.69
1.62 1.75
1.38
B 478
2.00
1.57 1.92
1.52
B 466
1.62
1.65 1.08
1.30
COMM 150
1.50
1.29 1.40
1.42
COMM 230
1.20
1.08 1.42
1.15
COMM 130
1.63
1.62 1.43
1.62
COMM 100
0.58
0.15 0.43
1.10
COMM 316
-0.16
0.65
-.17
0.35
COMM 102
0.38
1.08 0.24
1.03
CS 124
1.49
1.61 1.44
1.53
ENG 316
0.61
0.71 0.81
0.49
ESS 160
1.17
0.93 0.78
0.54
ESS 161
0.86
0.34 0.83
0.67
ESS 162
0.40
0.50 0.27
0.13
ESS 264
1.20
1.46 1.60
1.64
SPAN 101
1.13
1.12 1.16
1.18
HIST 201
0.95
0.47 0.78
0.43
CHILD 210
1.24
1.24 1.17
0.56
CHILD 335
1.07
0.92 1.79
1.67
HFED 100
1.05
0.32 0.68
0.49
HFED 350
0.97
1.17 1.02
0.76
1.33 1.00
1.91
MATH 100A 1.67
MUSIC 209
0.03
0.65 0.77
0.55
PSYCH 302
0.62
1.12 1.14
1.20
SOC 111
0.74
1.10 1.10
0.85
SOC 340
1.03
1.17 1.20
1.18
SOC 360
0.75
1.31 0.71
0.46
69
Mean
Score
.71
.35
.98
1.82
1.38
1.50
1.61
1.75
1.41
1.40
1.21
1.58
.57
.17
.68
1.52
.66
.86
.68
.33
1.48
1.15
.66
1.05
1.36
.64
.98
1.48
.50
1.02
.95
1.15
.81
Preliminary evidence shows that a correlation may be present between course
ratings and perceived learning scores. In other words, if perceived learning scores are
high then course ratings will likely be high as well. This intuitive result seems plausible
based on experience. When a student enjoys a course they may have a tendency to rate
the course high. Also, when a student feels he/she has learn a significant amount in a
course they may tend to rate highly all aspects of the course. Could application of LM
process steps be the reason students perceive themselves as learning more in a given
course? Again, research question three will address this in detail.
Individual mean scores for perceived learning were averaged for each semester
for both T1 and T2. This was done to study the average fluctuation of perceived learning
scores by semesters. Table 6 shows the mean perceived learning scores for T1 and T2 by
semesters. As previously defined, perceived learning scores are based on a scale from a
negative two to a positive two. A negative two indicates that all students participating in
the end-of-course evaluation felt they had learned a “great deal less” when compared
with other courses. A combined evaluation score of negative one indicates that students
felt they had learned “a little less” in this particular course as it compares to other courses
they have taken. A zero score indicates students perceived that they had learned “about
the same” as compared to other courses they have taken. A positive one perceived
learning score suggests students felt as if they had learned “a little more” than other
courses taken. Lastly, a score of positive two would mean all students felt they had each
learned “a great deal more” when compared to other college courses taken.
70
Table 6
ANOVA: Student Perceived Learning Scores for T1& T2 Semesters
Evaluation Instrument
Semester
Perceived Learning
Standard
Score
Deviation
N
Employed
Fall 2007
33
1.01
.20
Old Evaluation
Winter 2008
33
0.99
.18
Instrument (T1)
Spring 2008
33
1.13
.27
Fall 2008
33
1.03
.26
Winter 2009
33
1.03
.28
New Evaluation
Spring 2009
33
1.05
.20
Instrument (T2)
Fall 2009
33
1.04
.25
Winter 2010
33
1.02
.22
*Indicates a significance level at or less than .05 between semesters
Notice that the mean scores for student perceived learning is close to positive one,
plus or minus a tenth of a point. What this suggests is that for the 33 courses being
evaluated the average student perceptions of their learning was “a little more” than other
college courses taken. However, no noticeable pattern of an increase in perceived
learning scores are apparent in this initial ANOVA for T1 & T2.
The differences between the Fall 2007 semester and the Fall 2008 semester is
very slight. There seemed to be a spike in perceived learning scores during the Spring
2008 semester, but then scores seem to level off around 1.04. No significance was found
between the differences in perceived learning scores for either T1 or T2. Likewise, the
71
differences in mean scores between Fall 2007 and Winter 2010 were such that almost no
difference was noticed. The significance level was found to be .904, a value close to one,
indicating little to no change.
In summary, the findings show that no significant change in student perceived
learning was noticed for the 33 courses measured. The average mean perceived learning
score indicated that, in general, student felt that they had learned “a little more” in the
courses being studied compared to other college course they had taken. Interestingly, the
findings for research question one indicated that a significant change in course ratings
was realized for this same population being studied. Yet, research question two revealed
no significant change in student perceived learning during the same time period. It seems
that course ratings have been affected by some outside factor, but student perceived
learning has remained unchanged, regardless of the change in course evaluation
instruments. To what degree has the LM contributed to the change in these variables?
The next section will explore the strength of correlation between LM scores and student
perceived learning scores.
Research Question 3: The Relationship between LM and Perceived Learning Scores
The purpose in asking the third research question is to learn how closely LM
scores correlate with student perceived learning scores. If a correlation exists between
these variables one could assume that the independent variable could help predict the
outcome of the dependent variable. In other words, if evidence shows a strong
correlation between variables, then LM scores could be used to predict the general
outcome of student perceived learning scores. Furthermore, one could extrapolate from
72
this assumption that when instructors are effective in implementing LM process steps in a
course an increase in student perceived learning will be the outcome, generally speaking.
For this study, LM scores act as the independent variable and student perceived learning
represent the dependant variables measured.
In addition, by measuring the strength of correlation between LM scores and
student perceived learning scores the implications of applying LM process steps begin to
be revealed. The finding for this section can help answer questions like the following:
“Do student perceived learning scores improve as LM scores improve?” or “When LM
scores are high in a given class do students perceive to learn more in these classes
compared to course where a low LM score was recorded?”
The first step in learning the strength of correlation between LM scores and
perceived learning scores was to compare the means for each variable; this was
completed in the previous two sections. For this research question all data points found
in T2 were used, totally 5,696 courses over the four semester span. These data points
were disaggregated to control for class size, semesters in which the course was taught,
and the instructor’s employment status.
Table 7 shows mean perceived learning scores for each semester based on
instructor’s employment status. Notice the increase in mean perceived learning scores
for CFS instructors when compared to 1YR, ADJ, and VET for each of the four
semesters represented. On average, CFS instructor’s mean perceived learning scores
were found to be at least .15 points higher than 1YR instructor’s mean perceived learning
score and .21 points higher than VET instructor’s scores. Also, notice the standard
deviation for CFS is lower than the other three variables with an average of .46 points.
73
Less variation from the mean perceived learning score was found for CFS instructor’s
than for 1YR, ADJ, and VET. VET instructor’s had the highest deviation from the mean
perceived learning score with an average deviation of .59 points.
74
Table 7.
Mean Perceived Learning Score by Semester Based on Instructor’s Employment Status
Semester
Fall
2009
Spring
2009
Winter
2009
Winter
2010
Mean
Instructor’s Employment
Status
1 Year Appointment
Adjunct
Continued Faculty Status
Veteran
Total
1 Year Appointment
Adjunct
Continued Faculty Status
Veteran
Total
1 Year Appointment
Adjunct
Continued Faculty Status
Veteran
Total
1 Year Appointment
Adjunct
Continued Faculty Status
Veteran
Total
1 Year Appointment
Adjunct
Continued Faculty Status
Veteran
Total
Mean
.93
.83
1.09
.87
.94
.91
.94
1.07
.86
.96
.88
.89
1.10
.89
.96
.99
.85
1.07
.86
.93
.93
.88
1.08
.87
.95
N
122
311
488
564
1485
111
233
458
402
1204
102
405
533
578
1618
128
306
404
551
1389
463
1255
1883
2095
5696
Std. Deviation
.53
.54
.45
.58
.54
.51
.53
.46
.63
.55
.53
.53
.48
.59
.55
.45
.52
.47
.55
.52
.50
.53
.46
.59
.54
Next, the researcher likewise compared mean LM scores by semester based on
instructor’s employment status (see Table 8). Similar results were noticed with CFS
instructors having the highest mean LM score per semester over 1YR, ADJ, and VET
instructors. The difference between average LM scores was not as stark of a contrast
75
when compared to perceived learning scores. On average, only .03 points separated the
top score from the bottom score. A similar pattern was noticed in standard deviation, as
CFS instructors showed lowest deviation from the mean of .06 points and VET
instructors had the highest standard deviation score of .10 points.
76
Table 8.
Mean LM Scores by Semester Based on Instructor’s Employment Status
Semester
Fall
2009
Spring
2009
Winter
2009
Winter
2010
Mean
Instructor Employment
Status
1 Year Appointment
Adjunct
Continued Faculty Status
Veteran
Total
1 Year Appointment
Adjunct
Continued Faculty Status
Veteran
Total
1 Year Appointment
Adjunct
Continued Faculty Status
Veteran
Total
1 Year Appointment
Adjunct
Continued Faculty Status
Veteran
Total
1 Year Appointment
Adjunct
Continued Faculty Status
Veteran
Total
Mean
.83
.83
.85
.82
.83
.84
.84
.85
.81
.83
.82
.83
.85
.82
.83
.84
.83
.85
.83
.84
.83
.83
.85
.82
.83
N
122
311
488
564
1485
111
233
458
402
1204
102
405
533
578
1618
128
306
404
551
1389
463
1255
1883
2095
5696
Std. Deviation
.07
.09
.06
.10
.08
.06
.07
.07
.12
.09
.06
.09
.06
.10
.09
.05
.10
.07
.07
.08
.06
.09
.06
.10
.08
A similar pattern was noticed for both perceived learning scores and LM scores
when controlling for instructor’s employment status. For each semester, CFS instructors
77
scored the highest for both perceived learning scores and LM scores. Likewise, VET
instructors had the lowest score for most of the semesters. Standard deviation scores
were low for CFS and high for VET instructors. Compared with VET instructors, in
general CFS instructors will have a perceived learning score that is .101 points higher
(see Appendix VII). Does this trend hold true when perceived learning scores and LM
scores are disaggregated by class size? Does class size affect this trend?
Perceived learning scores were categorized by semester and class size to compare
the differences in means (see Table 9). The results of the comparison show a difference
in mean perceived learning scores between small classes and very large classes. Small
sized classes have the highest perceived learning score, whereas very large sized classes
tend to have the lowest perceived learning scores. However, perceived learning scores
tend to have the highest deviation from the mean and very large classes have the lowest
variance from the mean score, generally speaking.
78
Table 9.
Mean Perceived Learning Scores by Semesters Based on Class Size
Semester
Fall 2009
Spring
2009
Winter
2009
Winter
2010
Mean
Class Size
Mean
N
Std. Deviation
0-10
1.04
102
.74
11-20
21-30
Over 30
Total
0-10
11-20
21-30
Over 30
Total
0-10
11-20
21-30
Over 30
Total
0-10
11-20
21-30
Over 30
Total
0-10
11-20
21-30
Over 30
Total
1.0
.96
.87
.94
1.13
1.05
.97
.88
.96
1.12
.96
.97
.89
.96
1.02
1.01
.94
.87
.93
1.10
1.02
.96
.88
.95
263
417
703
1485
126
238
308
532
1204
240
332
449
597
1618
97
268
435
589
1389
565
1101
1609
2421
5696
.60
.51
.48
.54
.67
.50
.53
.53
.55
.74
.59
.47
.47
.55
.65
.64
.52
.42
.52
.71
.59
.51
.48
.54
LM mean scores were also compared by semester based on class size (see
Appendix VIII). No differences in mean LM scores were noticed between large,
medium, and small classes, each scoring on average .84 points. Very large class sizes
showed a slight drop in mean LM score with an average score over the four semesters of
.82 points. For the most part, there were virtually no differences in LM scores from one
79
semester to another. In other words, mean LM scores for class size based on the semester
year in which the course was taught showed little to no difference between categories or
semesters.
Preliminary evidence shows that perceived learning scores tend to be affected by
class size. Yet LM scores show little to no effects from the change in class size from one
semester to the next. Likewise, LM scores tend to be similar regardless of a small,
medium, or large class. There seems to be some variability for very large classes,
however. Very large classes show a drop in LM scores for most of the semesters in
which it was taken.
The last section for question three addresses the relationship between the LM and
student perceived learning. This portion of the third research question is most revealing
and telling about the effectiveness of the LM and the importance of its use in the
classroom. Implications could be significant if students feel they learn more when the
LM process steps are applied throughout a course.
Again, this section is broken down into three areas of study in an attempt to
evaluate the strength of correlation between the LM and student perceived learning, given
different variables. First an overview of the average correlating values will be looked at,
followed by a breakdown of variables as previously mentioned.
The effects of class size. Class size can have an effect on both LM scores and
student perceived learning scores, as show previously. The following section will look at
the correlating effects of class size and evaluate how the difference in class sizes might
affect the strength of correlation between the two variables. Table 10 shows how average
LM scores, perceived learning scores, and class size correlate one with another.
80
Table 10.
Correlation Matrix of LM Scores and Perceived Learning Scores by Class Size
Average LM
Perceived
Class Size
Scores
Learning Scores
Average LM Scores
1
Perceived Learning Scores
.518
1
Class Size
-.091
-.142
1
Each of the correlating values was found to be significant at the .05 level. In
other words, evidence suggests when LM scores increase perceived learning scores do
likewise, generally speaking. Another significant finding was noted in this correlation
matrix regarding class size. Notice the negative correlation between class sizes and both
LM scores and perceived learning scores, respectively. This shows that when class sizes
are increased, LM scores and perceived learning scores decrease. Remember, from
earlier findings very large sized classes tend to have the lowest LM scores and perceived
learning scores when compared to other sized classes. This begs the question, could
using the LM in very large sized courses produce greater change in perceived learning
scores compared with other sized classes?
The effects of semester enrolled. This next section focuses on the effects course
fulfillment status may have on LM scores and student perceived learning scores. As
mentioned from previous sections, semesters are divided into four separate categorizes;
81
Winter 2009, Spring 2009, Fall 2009, and Winter 2010. Table 11 shows the correlation
between LM scores and student perceived learning scores by semesters.
Table 11.
Correlation Matrix of LM Scores and Perceived Learning Scores by Semester
Semester
Winter 2009
Spring 2009
Fall 2009
Winter 2010
Variables
LM Scores / Perceived Learning
Score
LM Scores / Perceived Learning
Score
LM Scores / Perceived Learning
Score
LM Scores / Perceived Learning
Score
LM Scores /
Perceived Learning
Scores
.61
N
1485
.56
1204
.62
1618
.64
1389
To summarize this section of the study regarding the semester when students
enrolled, the pattern of a positive correlation between LM scores and student perceived
learning scores are consistent throughout. Similarly to previous sections of this study, a
correlation seems to be present between the two variables when compared one to another.
Courses taken during the Winter 2010 semester showed the strongest correlation between
LM scores and student perceived learning scores. In other words, the LM scores of those
students enrolled in a course during the Winter 2010 semester could be used to predict
perceived learning score more accurately than the other three semesters. However, this is
not conclusive. Not every winter semester will bring similar result. Likewise, in each
semester, the correlation values are positive and fairly strong. In other words, the
82
semester in which a course was taken seems to have little to no affect on the strength of
correlation between LM scores and perceived learning scores.
The effects of instructor employment status. The last portion of this section,
regarding the relationship between LM scores and perceived learning scores, looks at the
effects that instructor employment status might have on the strength of correlation
between these two variables. In other terms, will the strength of correlation between LM
scores and perceived learning scores be affected by whether the instructor is tenured
(VET), working towards tenure (CFS), adjunct (ADJ) or hired on as a temporary oneyear appointment (1YR)?
The first to be considered are instructors who are working towards tenure (CFS).
It is assumed that most of instructors in this category are working towards and are hoping
to receive tenure status. One requisite in receiving tenure at BYU-Idaho are good student
evaluations. Likewise, instructors who use good teaching strategies, like those inherent
in the LM, will have more of the advantage receiving tenure status. Table 12 represents
the relationship between LM scores and student perceived learning scores for each
respective instructor. Notice again the strong correlation between the two variables in a
positive direction. This value was calculated to be at the .000 significance level, meaning
that the probability of this correlation between LM scores and perceived learning scores
happening by chance is less than .001%. Therefore, not only does this tell us that a
positive correlation exists between these two variables, but that the correlation itself is
significant.
83
Table 12.
Correlation Matrix of LM Scores and Perceived Learning Scores by Instructor
Employment Status
Instructor
Employment
Status
1YR
ADJ
CFS
VET
Variables
LM Scores / Perceived Learning
Score
LM Scores / Perceived Learning
Score
LM Scores / Perceived Learning
Score
LM Scores / Perceived Learning
Score
LM Scores /
Perceived Learning
Scores
.76
N
463
.67
1255
.60
1883
.55
2095
It is interesting to see that the strongest correlation between LM scores and
student perceived learning was found for 1YR instructors. The lowest correlation
between the two variables was found to be with VET instructors. In other words,
instructors on a one year appointment have a better chance of predicting perceived
learning scores based on LM score results over courses taught by VET instructors. This
might be due to teacher reputations. A VET instructor may already be known to
incoming students. This previous bias may play a role in the difference in strength of
correlation between these variables.
The findings for employment status seem to be consistent in showing a positive
correlation between LM scores and student perceived learning scores. Some correlations
are stronger than others, but all seem to be positively correlated, meaning LM scores are
good predictors of student perceived learning, regardless of class size, semester enrolled,
or instructor employment status.
84
Summary of Findings
The findings for each of the three research questions reveal differing results. For
research question one, findings show a significant increase in end-of-course ratings from
the Fall 2007 semester and the Fall 2008 semester. Likewise, a significant increase in
end-of-course ratings from the Fall 2007 and Winter 2010 semesters were measured for
the 33 courses studied. Yet, no significant difference in mean end-of-course ratings were
noticed between the Winter 2009 and Winter 2010 semesters, the semesters reflective of
the new course evaluation instrument.
For question two, no significant change in perceived learning scores was revealed
for the 33 courses under investigation. Somewhat surprising was the fact that little to no
difference in student perceived learning scores was measured between the Fall 2007 and
the Winter 2010 semesters. Overall, student perceived learning scores stayed relatively
constant at a rating of around 1.04.
The third research question was a study of the relationship between LM scores
and student perceived learning scores for semesters when the new course evaluation
instrument was used to collect student data. Furthermore, the findings were
disaggregated into class size, semester enrolled in course, and instructor employment
status. In all cases, a significant correlation was found between these variables.
The question remains, what do these findings mean? Chapter 5 will focus on the
possible implications and future impact these findings may have on curriculum
development, student study habits, course design, among other related areas.
85
CONCLUSIONS
The purpose in conducting this study was to determine if the implementation of
LM principles and process steps has affected (either negatively or positively) course
scores and/or student perceived learning scores as found on end-of-course evaluations.
Furthermore, the researcher inquired to know what effects class size, semesters enrolled,
and instructor employment status had, if at all, on the strength of correlation between LM
student perceived learning, respectively. Data was collected, based on course evaluations
from the Fall 2007 through Winter 2010 semester. The data were disaggregated to
control for the different variables, as previously mentioned, and statistical analyses were
employed to reveal the findings. This Chapter will summarize the revealed findings and
draw conclusions related to the implementation of the LM and the associated affects on
course scores and student perceived learning. Also, possible implications of these
findings along with further research possibilities and suggestions are explored.
Results of Findings
The purpose in asking research question one was to learn if end-of-course scores
had changed beginning with the Fall 2007 semester to the Winter 2010 semesters. The
difference between the two semesters is roughly two and ½ years. Shortly following the
Fall 2007 semester the LM was formally introduced to students and faculty as a campuswide model to follow. The researcher wanted to learn if the implementation of the LM
has had an effect on course scores over time.
The second research question was asked in an attempt to learn if student
perceived learning scores changed from the Fall 2007 to the Winter 2010 semesters. The
86
claim for implementing the LM at BYU-Idaho was to increase student learning and
understanding of the material being presented in class. If students perceived themselves
as learning more from one semester to another then perhaps something might be
contributing to this change. This was the intent in asking the second research question.
The third research question looks specifically at the semesters after the course
evaluation instrument was changed to measure the level at which the three process steps
of the LM were being employed by the instructor and students. This question was asked
in an attempt to evaluate the strength of correlation between LM scores and student
perceived learning scores. Furthermore, data were disaggregated down to subcategories
of class size, semester when taken, and instructor employment status. These
subcategories were evaluated to see what effects they might have on the strength of
correlation between LM scores and perceived learning scores.
Change in course ratings over time. For research question one, the researcher
first looked at course ratings of those classes that were taught by the same instructor for
the Fall 2007 semester to the Winter 2010 semester. The researcher used a comparison of
means and ANOVA to measure the differences in mean course ratings for the 33 courses
identified as matching the criteria previously determined for this question. A two-paired
t-test was used to measure the significance between mean scores for T1, T2, and the Fall
2007 and Winter 2010 semesters. This analysis revealed a significant difference between
the mean course ratings for the Fall 2007 semester compared with the Winter 2010
semester. This finding suggests that the change in mean course ratings from Fall 2007 to
the Winter 2010 semester did not happen by chance. In other words, during the two and
½ year time frame between the Fall 2007 and Winter 2010 semesters something occurred,
87
or a variety of things occurred, to bring about the change in end-of-course scores from
one semester to the other.
Remember, the course evaluation instrument was changed in the Winter 2009
semester. For this reason, the collection of data was divided into two parts, T1 and T2.
T1 represents the semesters in which the old evaluation instrument was used to collect
student data (Fall 2007 – Fall 2008), and T2 represented the semesters in which the new
course evaluation instrument was employed for the same reason (Winter 2009 – Winter
2010). The findings revealed a significant difference in mean course ratings for T1, but
not for T2. Also shown, was the fact that each consecutive semester’s mean course rating
improved for both T1and T2 time periods. With the implementation of the LM occurring
after the Fall 2007 semester, one may question “how has the LM process steps attributed
to this increase in course ratings?” Definite conclusions cannot be made at this point
regarding the cause of the increasing in course ratings during T1 through to the end of
T2. However, evidence shows that course ratings have been increasing, generally
speaking, from the Fall 2007 to the Winter 2010 semesters.
The change in mean course ratings for the Fall 2008 and the Winter 2009
semester was also measured. The purpose in doing so was to learn what affects the new
course evaluation instrument may have had on end-of-course ratings for the 33 courses
being evaluated. The finding revealed no significant change in course ratings between
the two semesters. Likewise, the increase in course ratings from the Fall 2008 to the
Winter 2009 semester was equal to the increases of previous semesters. What this
suggests is that the change in evaluation instrument seemed to have had little to no affect
on course ratings for the 33 courses studied. With this in mind, the researcher expanded
88
the study to span both T1 and T2. In other words, with no significant change in course
ratings between the Fall 2008 and Winter 2009 semesters, the researcher conducted a ttest on the mean differences in course ratings between the Fall 2007 and the Winter 2010
semesters.
The differences between mean course ratings between Fall 2007 and Winter 2010
had a positive difference, meaning that the average scores increased from one semester to
the other. On average, the scores did not drop between the two semesters. In fact, the
average mean scores for this group of instructors increased significantly from the
beginning semester to the end semester in this study. This is important in learning that
the implementation of the LM did not have a negative effect on course scores, generally
speaking.
In conclusion, for the 33 courses evaluated course ratings seemed to have
improved between the first semester and the last semester of this study. Still the question
remains: Could the LM be a reason for the increase? One of the probable causes of this
increase may be due to the application of LM process steps by instructors, but that
conclusion cannot be made at this time. Research question three addresses the strength of
correlation between these two variables in question, and will be addressed later in this
section.
The change in perceived learning over time. Findings from research question
one shows a definite change in course ratings from the semester before the
implementation of the LM to the Winter 2010 semester, representing the most recent data
available. Likewise, perceived learning scores were evaluated using the same
methodology as was used to address research question one. However, results for
89
perceived learning ratings for the 33 courses evaluated revealed a much different
outcome than was found for course ratings. No significant changes in perceived learning
scores were noticed for the 33 courses evaluated. T1 and T2 showed no significant
difference in perceived learning scores. In fact, the correlating value approached 1,
meaning the perceived learning scores between semesters were nearly the same.
Likewise, when the study was expanded to include the beginning semester (Fall 2007) to
the last semester in this study (Winter 2010) a similar result was realized. For the 33
courses measured, no significant differences in perceived learning scores were found.
Perhaps the reason for the relative similarity in perceived learning scores between
semesters over time is due to the structure of the question. The question used on both
evaluation instruments to measure perceived learning scores asks, “Compared to other
college courses you have taken, would you say that you have learned…” then students
select one of the following; “much less”, “a little less”, “about the same”, “a little more”,
or “much more”. The findings show that the mean perceived learning score is close to
one, which correlates with the response, “a little more”. In other words, for the 33
courses evaluated, students felt that, “compared to other college courses [they] have taken
… [they] have learned a little more.” One issue with this question is that students are
basing their answers on other college course they have taken. Does this give a real
representation of what students have learned in the particular class being evaluated or is it
merely a reflection of how much more or less a student has learned in a particular course
compared with other college courses taken? What if a student is evaluating a course in
which he/she has no experience in? Would the perceived learning score be higher in this
course compared with a course in which the student has a fundamental background
90
knowledge. For example, a student who is a native Chinese speaker may feel she learns
less in an Introduction to Chinese course compared with a non-Chinese speaking student,
simply due to past experiences. Certainly, perceived learning scores would be affected in
this scenario. A more accurate measure of perceived learning scores would be to
measure students’ competency level at the beginning of a course and again at the
conclusion of the same course.
Another interesting finding noticed when comparing mean perceived learning
scores with mean course ratings per semester was the fact that both scores seemed to
follow a similar trend. What was noticed was those courses which had the highest
perceived learning scores also had the highest course ratings. Likewise, those courses
that scored lowest in perceived learning also has the lowest course ratings. This
preliminary evidence shows that course ratings and perceived learning scores are
correlated. In other words, when course ratings go up it is likely that perceived learning
scores would also improve. Research question three would measure the strength of this
correlation.
In conclusion, the findings show no significant change in perceived learning
scores for the 33 courses evaluated during T1 or T2. However, the fact that perceived
learning scores measured close to a one for each respective semester suggests that in
general they felt they had learned “a little more” in these courses compared to other
courses they had taken. Does this suggest that these courses are at a higher level than
other courses? There are still too many variables to say what factors help predict student
perceived learning scores. More research is needed related to the variables that affect
91
student perceived learning scores. However, research question three of this study
examined the effects of LM scores on this variable and others.
The relationship between the LM scores and perceived learning scores. The
third research question looks particularly at the relationship or correlation between LM
scores and student perceived learning scores. The purpose in evaluating the correlation
between these two variables is to learn if an instructor whom implements the process
steps of the LM would realize a higher student perceived learning score. The findings
point to the fact that there is a correlation between LM scores and student perceived
learning scores.
In an attempt to measure the strength of correlation between each factor, LM
scores and perceived learning scores for T2 were disaggregated to class sizes, semesters
when the courses were taught, and instructor employment status. In each case, the
findings showed a positive correlation between LM and student perceived learning
scores. This suggests that the higher the LM score is for an instructor the higher their
student perceived learning score will be. The correlation between LM scores and
perceived learning scores remained strong regardless of class size, semesters when course
was taken, or instructor employment status.
Furthermore, when data were disaggregated to class size a difference in perceived
learning scores were realized between small and very large classes. Small classes
showed a composite mean perceived learning score of 1.10. Compare this score to the
composite mean perceived learning score for very large sized classes, measuring .88, and
one can quickly notice a significant difference between the two scores. In other words,
students perceive themselves as learning more in small sized classes than they do in very
92
large sized classes, generally speaking. This is congruent with Krueger’s (1999) study of
gains in test scores for a large population of students, as mentioned in Chapter 3.
Also noted was the difference in LM scores between small and very large sized
classes. Small sized classes had a higher LM score than did courses that were considered
to be very large in size. This difference between small and very large classes tells us one
of two things. Either, those instructors who are teaching small classes are implementing
and practicing the LM more effectively than instructors of very large classes, or
instructors of very large classes simply are not using the LM in their respective classes.
A correlation matrix was employed to measure the strength of correlation and
direction between LM scores, perceived learning scores, and class size. Evidence was
found suggesting that LM scores and perceived learning scores are positively correlated.
The strength of this correlation was found to be .518, a fairly strong result. In other
words, when LM scores increase student perceived learning scores will likely increase as
well. This is a significant finding considering initial literature suggests that instructors
are resistant to changing their teaching methodology for fear of lower course ratings and
student perceived learning scores (Bergstrom, 2009). This is evidence of just the
opposite. When the LM is practiced more in a course, chances are course ratings and
student perceived learning will improve.
Also noted from the correlation matrix measuring LM scores, perceived learning
scores, and class size, were the effects class size had on LM scores and perceived
learning scores. A negative correlation was found between these variables, meaning
evidence points to the fact that when class size increases LM scores decrease. Likewise,
when class size increases perceived learning scores decrease as well. In other words, the
93
bigger the class size the lower the LM score and perceived learning score will be,
generally speaking.
However, the strength of correlation for class sizes showed growth between LM
scores and perceived learning scores as the size of classes increase (see Appendix IX).
The larger the class size the stronger the correlation between the two variables. In other
words, the confidence in predicting perceived learning scores based on LM scores would
be higher for very large sized classes compared to small sized classes, generally
speaking. In fact, very large sized classes revealed a very strong correlation between LM
scores and perceived learning scores of .71. What this suggests is that in courses of very
large size, instructors who have a high LM score will likely have high student perceived
scores as well. Since this correlation between the two variables is so strong, one must ask
if instructors of very large sized classes would experience the most improvement in
course ratings and student perceived learning by simply using more of the LM process
step in their teaching methodology. This inquiry demands further study and investigation
to give a definite answer to this question.
Interestingly, when data were disaggregated to semester, no significant changes
were found between LM scores and perceived learning scores. In other words, it is
irrelevant when a course is taken as it relates to LM scores and perceived learning scores.
The time of year when a course was taken showed no differences between LM scores and
perceived learning scores, respectively. Fall, Winter, and Spring semesters all had
similar scores.
Conversely, significant changes in LM scores and perceived learning scores were
found when data were disaggregated to employment status. CFS instructors showed the
94
highest mean LM and perceived learning scores, whereas VET instructors showed the
lowest respective scores. Remember, CFS instructors are those who are hired fulltime
but not yet tenured. BYU-Idaho’s tenured track system is a four year process. VET
instructors represent that portion of the faculty who are tenured and have been employed
at BYU-Idaho for a minimum of four years. It’s not surprising to learn that those
instructors who are working towards tenured status are implementing the LM at a higher
level than those instructors who are not. However, it is somewhat discouraging to find
that VET instructors have the lowest perceived learning scores compared to the rest of the
faculty population. One might assume that an instructor who is tenured and has been
teaching for at least four years would have at least comparable student perceived learning
scores with their colleagues. But, evidence shows that VET instructors have significantly
lower LM and perceived learning scores compared with CFS instructors.
One factor that may be contributing to the differences in LM and perceived
learning scores between VET and CFS instructors could be due to variance in scores.
CFS instructors showed the lowest deviation from mean for both LM and perceived
learning scores. Conversely, VET instructors were found to have the highest deviation
from the mean for both variables. What this reveals is the fact that more CFS instructors
are receiving scores that are close to the mean score than VET instructor. In other words,
more VET instructors are receiving a wider range of scores than are CFS instructors.
More VET instructors are receiving very low LM and perceived learning scores
compared to CFS instructors.
However, when comparing the strength of correlation between LM and perceived
learning scores for each of the four categories of employment status, 1YR instructors
95
received the strongest correlation value of .76. VET instructors measured the lowest
correlation strength of .55. Both of these correlation values are strong, suggesting that
LM scores are a good predictor of perceived learning scores. The difference in
correlation strength between each employment category is interesting. Perhaps one
reason for the difference in correlation strength is due to reputation? Notice, those
instructors who are hired as a one-year appointment are relatively unknown to student
prior to their teaching assignment. Compare them to VET instructors who have been
teaching for at least four years and have likely built some type of reputation with
students. An instructor who is unknown to the student and is without prior reputation
seems to have the strongest chance of improving student perceived learning scores by
implementing more of the LM. However, more research is needed in this area to make a
conclusive statement.
To conclude, evidence suggests that when the LM is practiced at a higher level of
implementation in the classroom student perceived learning scores will likely improve,
generally speaking. Also, regardless of class size, the semester when the course is taken,
or the employment status of an instructor, when LM scores are high student perceived
learning scores will likely also be high.
Possible Implications
Scott Bergstrom, BYU-I’s Institutional Researcher, writes, “We are aware that
some instructors are concerned that their course evaluations might be adversely affected
by incorporating elements of the Learning Model into their teaching.” Then he adds,
“The new course evaluation instrument, which has a strong Learning Model orientation,
96
gives us some preliminary information that might allay this concern” (Bergstrom, 2009).
Bergstrom found that in courses where students percieved the highest occurance of LM
applications, instructors likewise received equally high marks. Similarly, this expanded
study of more than 5,600 courses also corroborate Bergstroms findsings. This suggests
that instructors who effectively impliment the LM process steps may not see a drop in
their course ratings nor in student perceived learning scores.
The implications of these findings may help alleviate concerns regarding
instructors changing teaching techniques to mirror LM practices. In turn, if instructors
realize the positive effects of implementing LM practices in their teaching curriculum,
student learning will be enhanced and the level of learning will increase campus wide.
In this study, the researcher compared data previously collected from the old
course evaluations with data collected from the new course evaluation instrument,
presently being used on campus. What was discovered was that, in general, the
implementation of the LM has had no ill effects on course ratings or student perceived
learning. What’s more, the findings point to evidence that shows an increase in course
ratings and student perceived learning when the level of implementation of the LM is
increased. The implications of such a study could be useful for administrators,
instructors, and students alike as they consider what type of teaching methodology could
best facillitate learning in the classroom.
On a broader scale, researching the significance of implimenting a model for
teaching and learning in the arena of higher education could have far reaching
implications. Results from this study could be used in developing an accountability
model for students, much like how the end-of-course evaluations are designed in this
97
study. Institutions with a desire to emphasis the importance of students taking more
responsibility for their own learning could also profit from this study.
Fears regarding large scale implimentation of a learning model at a University
with similar demographic may be alliviated as a result of these findings. What’s more,
the findings in this study show a strong correlation between teaching practices that
encourage active participation and engagement of students and perceived student learning
scores. Meaning, students learn more when they actively prepare for and participate in
class. The finding is not new, but the way to encourage students and faculty to apply
good teaching practices is. The process steps inherent in the LM at BYU-Idaho
incorporates elements of sound, effective teaching strategies. Another University or
college could, in effect, incorporate the idea of a model for learning and teaching—
founded in effective teaching strategies. This study highlights one method of tracking
good teaching through the use of course evaluations. Having a LM score reflective of the
level of active participation in and implementation of effective teaching can be benefitial
to other orgainizations.
Suggestions for Future Research
This study has revealed many avenues for future study. As the study progressed
over time, additional questions arose and caused the researcher to take note. Some of the
possible future research ideas involve specific techniques while other questions are
general in nature. This study was the first attempt at measuring the effects of the LM at
BYU-Idaho. A more detailed look into specific teaching techniques, instructor skills, and
student demography, along with many other variables are still waiting to be explored.
98
One possible research idea might include a qualitative analysis of an effective
instructor and an ineffective instructor and determining the differences between the two.
Also, one might focus their study on an instructor that has a low LM score but has high
course ratings, instructor ratings, and high student perceived learning scores and compare
their teaching style to an instructor who scores low on their course ratings, instructor
ratings, and student perceived learning scores although their LM score is high.
Another possible action research idea would be to introduce this LM to an
instructor who’s teaching technique is more of a traditional style and measure whether
their end-of course ratings, instructor ratings, and student perceived learning scores
increase as the instructor increases the level of implementation of the LM. This could
also be extended to a group of instructors or even to another University.
Again, there are many avenues that one could pursue in conducting further
research in this area of study. The findings of this study are significant enough that they
do demand further study and investigation. After all, the purpose in education is to
further knowledge and increase learning. This study suggest that there is evidence in
doing so, which should cause all true educators to sit up and take notice.
99
REFERENCES
Bailey, C.M., Hsu, C.T., & DiCarlo, S.E. (1999). Educational puzzles for understanding
gastrointestinal physiology. Advanced Physiology Education, 21, 1–18
Beichner, R.J., Saul, J.M., Abbott, D.S., Morse, J.J., Deardorff, D.L., Allain, R.J.,
Bonham, S.W., Dancy, M.H., & Risley, J.S. (2007). The Student-Centered
Activities for Large Enrollment Undergraduate Programs (SCALE-UP) Project.
In Research-based reform of introductory physics, ed. E. Redish. Available online
by searching PER Central at www.compadre.org/per.
Bergstrom, S. (2009, March). Learning and Teaching. Retrieved March 20, 2009, from
Brigham Young University-Idaho:
http://web.byui.edu/LearningAndTeaching/post/2009/03/The-Impact-of-LearningModel-Implementation-on-Student-Course-Evaluations.aspx
Blumenfeld, P., Krajcik, J.S., Marx, R.W., & Soloway, E. (1994). Lessons learned: How
collaboration helped middle grade science teachers learn project-based
instruction. The Elementary School Journal, 94, 539–551.
Bonwell, C.C. & Eison, J. (1991). Active Learning: Creating Excitement in the
Classroom. ED 340 212.
Borman, W.C., Hanson, M.A., Oppler, S.H., Pulakos, E.D., & White, L.A. (1993). Role
of Early Supervisory Experience in Supervisor Performance. Journal of Applied
Psychology, 78(3), 443-449.
100
Boyle, J.R. (2001). The Effects of Strategic Notetaking on the Recall and
Comprehension of Lecture Information for High School Students with Learning
Disabilities. Learning Disabilities Research & Practice, 16(3), 133-141.
Brockway, G., McNeely, S., & Neffennegger, P. (1992). A point-counterpoint classroom
exercise. Marketing Education Review, Summer, 34-36.
BYU-I. (2007, September). LM. Rexburg, Idaho, USA.
Chan, V., Pisegna, J.M., Rosian, R.R., & DiCarlo, S.E. (1991). Construction of a model
demonstrating neural pathways and reflex arcs. Advanced Physiology Education,
271, 14–42.
Clark, K. B. (2009). About BYU-Idaho. Retrieved April 22, 2009, from BYU-Idaho:
http://www.byui.edu/aboutbyuidaho/
Chickering, A. W., & Gamson, Z. F. (1987). Seven Principles for Good Practice. AAHE
Bulletin 39: 3-7. ED 282 491. 6 pp. MF-01; PC-01.
Cochran, Leslie H. 1989. Administrative Commitment to Teaching. Cape Girardeau,
Mo.: Step Up, Inc.
Cortright, R.N., Collins, H.L., & DiCarlo, S.E. (2005). Peer instruction enhanced
meaningful learning: ability to solve novel problems. Advanced Physiology
Education, 29, 107–111.
Cortright, R.N., Collins, H.L., Rodenbaugh, D.W., & DiCarlo, S.E. (2003). Student
retention of course content is improved by collaborative-group testing. Advanced
Physiology Educational, 27, 102–108.
Crouch, C.H. & Mazur, E. (2001). Peer Instruction: Ten Years of Experience and
Results. American Journal of Physics, 69 (9), 970-977.
101
Dev, C.S. (1990). Measuring the Value of Experiential Learning. The Cornell H.R.A.
Quarterly. August: 105-07.
Dewey, J. (1916). Democracy and Education. New York, NY: Macmillan.
Draper, S.W. & Brown, M.I. (2004). Increasing Interactivity in Lectures Using an
Electronic Voting System. Journal of Computer Assisted Learning, 20, 81-94.
Edelson, D.C., Gordin, D.N., & Pea, R.D. (1999). Addressing the challenges of inquirybased learning through technology and curriculum design. Journal of the Learning
Sciences, 8, 391–450.
Felder, R.M., & Brent, R. (1996). Navigating the bumpy road to student-centered
instruction. College Teaching, 44(2), 43–47.
General History of BYU-Idaho. (n.d.). Retrieved November 10, 2009, from
http://www.byui.edu/PR/General/BYUIHistory2.htm
Gilbert, C.G., Hunsaker, S., & Schmidt, B. (2007). Peer Instruction: Faculty as
Architects of Peer Learning Environment. Retrieved March 19, 2010, from
http://www.byui.edu/perspective/v7n2pdf/v7n2_gilbert_et_al.pdf
Goyak, A.. The effects of cooperative learning techniques on perceived classroom
environment and critical thinking skills of preservice teachers. Ed.D. dissertation,
Liberty University, United States -- Virginia. Retrieved July 9, 2009, from
Dissertations & Theses: Full Text.(Publication No. AAT 3352768).
Hake, R.R. (1998). Interactive Engagement Versus Traditional Methods: A SixThousand-Student Survey of Mechanics Test Data for Introductory Physics
Courses. American Journal of Physics, January, 64-74
102
Hartley, J., & Davies, I. (1978). Note Taking: A Critical Review. Programmed Learning
and Educational Technology, 15, 207-224.
Hunt, T., Lederman, L., Kreps, G., & Ruben, B. (1985). Instructional simulation helps
public relations students to understand roles in organizational communication.
ERIC, ED 259 365.
Hyman, Ronald T. (1980). Improving Discussion Leadership. New York: Columbia
Univ., Teachers College Press.
Johnson, D. W., & Johnson, R. T. (1999). Making cooperative learning work. Theory
Into Practice, 38(2), 67-73.
Johnson, D. W., Johnson, R. T., & Smith, K. (1998). Cooperative learning returns to
college. Change, 30(4), 26-35.
Kagan, D. M. (1992). Professional Growth Among Preservice and Beginning Teachers.
Review of Educational Research, 62,129-169.
Karabenick, S. & Collins-Eaglin, J. (1997). Relation of Perceived Structural Goals and
Incentives to College Students’ Use of Learning Strategies. Journal of
Experimental Education, 65, 331-341.
Kellogg, R.T. (2008). Training Writing Skills: A Cognitive Developmental Perspective.
Journal of Writing Research, 1(1), 1-26.
King, A., Staffieri, A., & Douglas, A. (1998). Mutual Peer Tutoring: Effects of
Structuring Tutorial Interaction to Scaffold Peer Learning. Journal of
Educational Psychology, 134-152.
Krueger, A.B. (1991). Experimental Estimates of Education Production Functions. The
Quarterly Journal of Economics, 114, (2), 497-532.
103
Kuh, G. D. (2001). Assessing What Really Matters to Student Learning: Inside the
National Survey of Student Engagement. Change, 33, 10-17.
Kuh, G.D. (2003). What We’re Learning About Student Engagement from NSSE:
Benchmarks for Effective Educational Practices. Change, 35, 24-32.
Kuipers, J.C. & Clemens, D.L. (1998). Do I dare? Using role-play as a teaching strategy.
Journal of Psychosocial Nursing Mental Health Services, 36, 12–17.
Laws, P., Sokoloff, D., & Thornton, R. (1999). Promoting Active Learning Using the
Results of Physics Education Research. UniServe Science News, 13.
LM. (2008, January). Retrieved April 20, 2009, from BYU-I:
http://www.byui.edu/learningmodel/src/default.htm
Lowman, Joseph. 1984. Mastering the Techniques of Teaching. San Francisco: JosseyBass.
Lujan, H. L. & DiCarlo, S. E. (2006). Too much teaching, not enough learning: what is
the solution? Advances in Physiology Education, 30, 17-22.
Lymna, F. (1981). The responsive classroom discussion. In: Mainstreaming Digest,
edited by Anderson AS. College Park, MD: Univ. of Maryland College of
Education.
Mallor, J.P., Near, J.P., & Sorcinelli, M.D. (1981). Increasing student involvement in
large lecture courses: two case studies. Journal of Experiential Learning and
Simulation, 3(4), 191-196.
Marx, R. W., Blumenfeld, P. C., Krajcik, J. S., Fishman, B., Soloway, E., Geier, R., &
Tal, R. T. (2004). Inquiry-Based Science in the Middle Grdes: Assessment of
104
Learning in Urban Systemic Reform. Journal of research in Science Teaching,
41 (10), 1063-1080.
Marx, R.W., Blumenfeld, P., Krajcik, J.S., & Soloway, E. (1997). Enacting project-based
science. Elementary School Journal, 97, 341–358.
Matlin, M.W. (2003). Cognition. Hoboken, New Jersey, (101), 176-178.
Mazur, E. (1997). Peer Instruction: A User’s Manual. Upper Saddle Valley, New Jersey:
Prentice Hall.
Mazur, E. (1997). Peer instruction: getting student to think in class. In: The Changing
Role of Physics Departments in Modern Universities, edited by Redish EF and
Rigden JS. Buenos Aires, Argentina: Proceedings of ICUPE 399, 981–988.
McDermott, L.C. (1993). How we teach and how students learn. Ann NY Acad Sci 701,
9–20.
McKeachie, W.J. (1994). Learning and Cognition in the College Classroom. Teaching
Tips: Strategies, Research and Theory for College and University Teachers.
Lexington, MA.
McKeachie, Wilbert J., Paul R. Pintrich, Yi-Guang Lin, and David A.F. Smith. 1986.
Teaching and Learning in the College Classroom: A Review of the Research
Literature. Ann Arbor: Regents of The Univ. of Michigan. ED 314 999. 124 pp.
MF-01; PC-05.
Michael, J.A. (1993). Teaching problem solving in small groups. Ann NY Acad Sci, 701,
37–48.
Mierson, S. (1999). Skits and games to enhance students' learning of physiology.
Advanced Physiology Education, 277, 283–284.
105
Moy, J.R., Rodenbaugh, D.W., Collins, H.L., & DiCarlo, S.E. (2000). Who wants to be a
physician? An educational tool for reviewing pulmonary physiology. Advanced
Physiology Education, 24, 30–37.
Odenweller, C.M., Hsu, C.T., & DiCarlo, S.E. (1998). Educational card games for
understanding gastrointestinal physiology. Advanced Physiology Education, 20,
78–84.
Penner, J.G. (1984). Why Many College Teachers Cannot Lecture. Springfield, Ill.:
Charles C. Thomas.
Piaget, J. (1928). The Development of Thought: Equilibration of Cognitive Structure.
New York, NY: Basic Books.
Prince, M.J. (2004). Does Active Learning Work? A Review of the Research. Journal of
Engineering Education, 93(3), 223-31.
Prince, M.J & Felder, R.M. (2006). Inductive Teaching and Learning Methods:
Definitions, Comparisons, and Research Bases. Journal of Engineering
Education, 95(2), 123-138.
Rao, S.P., Collins, H.L., & DiCarlo, S.E. (2002). Collaborative testing enhances student
learning. Advanced Physiology Education, 26, 37–41.
Rao, S.P. & DiCarlo, S.E. (2000). Peer instruction improves performance on quizzes.
Advanced Physiology Education, 24, 51–55.
Rao, S.P. & DiCarlo, S.E. (2001). Active learning of respiratory physiology improves
performance on respiratory physiology examinations. Advanced Physiology
Education, 25, 55–61.
106
Rodenbaugh, D.W., Collins, H.L., & DiCarlo, S.E. (1999). Construction of a model
demonstrating cardiovascular principles. Advanced Physiology Education, 277,
67–83.
Ruhl, Kathy L., Charles A. Hughes, and Patrick J. Schloss. Winter 1987. "Using the
Pause Procedure to Enhance Lecture Recall." Teacher Education and Special
Education 10: 14-18.
Rydalch, B.J. (2005). Inauguration set for BYU-Idaho President Kim B. Clark. BYUIdaho News Release, October 3. Retrieved from
http://www.byui.edu/News/NewsReleases2005/051003Innauguration.htm
Sapon-Shevin, M., & Schniedewind, N. (1992). If cooperative learning's the answer,
what are the questions? Journal of Education, 174(2), 11-37.
Scannapieco, F.A. (1997). Formal debate: an active learning strategy. Journal of Dental
Education, 61, 955–961.
Scott, R.G. (2005). To Understand and Live Truth. Address, CES Religious Educators at
the Jordan Institute of Religion, February 4.
Silberman, M. (1996). Active Learning: 101 Strategies to Teach Any Subject.
Philadelphia, PA: Allyn and Bacon.
Silverthorn, D.U. (1999). Using demonstrations to uncover student misconceptions: the
Law of LaPlace. Advanced Physiology Educational, 277, 281–282.
Slavin, R. E. (1995). Cooperative Learning. Boston: Allyn and Bacon.
Slavin, R.E. (1983). When does cooperative learning increase student achievement?
Psychology Bull, 94, 429–445
107
Smith, K.A., Sheppard, S.D., Johnson, D.W., & Johnson, R.T. (2005). Pedagogies of
Engagement: Classroom-Based Practices. Journal of Engineering Education,
January, 1-15.
Stronge, J. H. (1997). Improving schools through teacher evaluation. In J. H. Stronge
(Ed.), Evaluating teaching: A guide to current thinking and best practices (p. 123). Thousand Oaks, CA: Corwin Press, Inc.
Thurgood, L. & Gilbert, C. (2008). What is Unique about the BYU-Idaho Learning
Model? Learning and Teaching Website. Retrieved from
http://web.byui.edu/learningandteaching/post/2008/10/What-Is-Unique-about-the-BYU-Idaho-Learning-Model.aspx
Vygotsky, L. (1978). Mind in Society: The Development of Higher Psychological
Processes. Cambridge, Massachusetts: Harvard University Press.
Wankat, P. (2002). The Effective Efficient Professor: Teaching, Scholarship and Service.
Allyn and Bacon: Boston, MA.
Wiggins, G. (2004). Assessment as feedback. Retrieved May, 2009 from
http://www.newhorizons.org/strategies/assess/wiggins.htm.
Wolf, P.. Fostering critical reflection in higher learning: The self study of a practitioner
and coach. M.Ad.Ed. dissertation, St. Francis Xavier University (Canada),
Canada. Retrieved July 9, 2009, from Dissertations & Theses: Full
Text.(Publication No. AAT MR27250).
108
APPENDIX I - INITIAL RESEARCH FINDING BY SCOTT BERGSTROM
Using questions from the course evaluation instrument, I created five scales to
correspond to the five elements of the LM architecture – Outcomes, Architecture,
Prepare, Teach One Another, and Ponder/Prove. Space does not permit me to describe
each of these scales so I will use the Teach One Another scale as an example. Student
responses to seven questions from the instrument were used to create the Teach One
Another scale:
o I was
an active participant in online or face-to-face class discussions.
o I sought
o I feel
opportunities to share my learning with others outside of class.
that I made important contributions to the learning and growth of fellow
classmates.
o The course provided
o Group
opportunities to learn from and teach other students.
work, if assigned, was beneficial and meaningful.
o Students
were actively involved in this class through discussions, group work, and
teaching.
o The instructor
provided appropriate opportunities to be an active participant in
the class.
The scale score derived from these seven questions serves as an indicator of the extent to
which this element of the LM is present in a course.
Using the scores on the Teach One Another scale, I divided all courses that were
evaluated in Fall 2008 into four quartiles representing the degree to which the Teach One
Another element of the LM was present in the course. The first quartile represents
109
courses in the lowest 25% of the Teach One Another scores, the second represents the
next 25% of courses, and so on. I computed the average overall instructor rating1 and the
average level of perceived learning2 for each quartile group of courses. The results of
this analysis are in Figures 1 and 2.
Figure 1. The average overall instructor rating for each quartile of the Teach One Another
scale.
110
Figure 2. The average level of perceived learning for each quartile of the Teach One
Another scale.
As shown in Figure 1, the overall instructor rating rises steadily from the 1st to
the 4th quartile. The difference between the bottom and top quartiles is 1.34 points, a
rather large (and statistically significant) difference. As shown in Figure 2, the level of
perceived learning rises sharply from the 1st to the 2nd quartile and then steadily upward
thereafter. The top quartile average level of perceived learning is more than three times
larger than the bottom quartile average!
The pattern seems clear for both measures. The higher the presence of the Teach
One Another element in a course, the higher the overall instructor rating and the greater
the level of perceived learning. My analysis showed that this same pattern holds true for
all five elements of the LM: the greater the presence of a LM trait in a class, the
higher the overall instructor rating and the higher the level of perceived learning.
This is a very preliminary analysis and more work needs to be done before we can
draw firm conclusions. In the first place, this [is] a correlational analysis. We do not
know if there is a causal relationship at work. Do LM activities make effective teachers
or are effective teachers more likely to use LM activities? Second, there may be other
factors and traits that are more influential on the overall instructor rating or on the level
of perceived learning. Is there a halo effect in play here wherein teachers who are
popular and courses that are well-liked yield high scores on all questions no matter
what? We simply do not know from this preliminary analysis how influential LM traits
are on these selected measures. The presence of such a pronounced effect connected with
111
LM practices compels further examination. Third, the power of the indicators being used
remains to be determined. For example, do the seven items that make up the Teach One
Another scale really provide a good measure of the presence of that trait in a class? Are
there better indicators of the Teach One Another trait?
In the spirit of the scholarship of teaching and learning, I am hopeful that some
instructors will experiment with a well-defined LM practice and see if there are any
changes in the course evaluations or, more importantly, in overall student learning and
growth. For example, several members of the Physics faculty have conducted pre- and
post-test student assessments comparing the impact of different peer- and problem-based
teaching innovations. Such experiments may provide us with compelling evidence of
LM effects, beyond what we can derive from course evaluation data.
At the very least, it is exciting that this first empirical look at LM practices
implies a positive relationship with the overall instructor rating and the level of perceived
learning. It also suggests that, in general, teachers will not be penalized by students for
using elements of the LM in their classes. (Bergstrom, 2009)
112
APPENDIX II – OLD EVALUATION INSTRUMENT
BYU-IDAHO COURSE EVALUATION
Please evaluate the following instructor and course. If you feel an item is not applicable to your
course, then leave it blank. When you are finished, click on the SUBMIT button at the bottom of the
page. Your identity is completely anonymous. Please be as thorough and as accurate as possible. Your
feedback is highly valued. It is used by your instructor and the school's administration to improve
teaching.
Instructor:
CHECKETTS, MAX L
Course :
REL 233
CHURCH HISTORY
Section:
9
Overall Rating: CHECKETTS MAX L - REL 233
very poor
poor
fair
good
very good excellent
exceptional
What is your overall rating of this instructor.
What is your overall rating of this course .
Items about the Course: CHECKETTS MAX L - REL 233
Strongly
Disagree
1
Course objectives are clear.
2
Course is well-organized.
3
Student responsibilities are clearly defined.
4
Course content is relevant and useful.
5
Assigned workload is appropriate for credit hours.
6
Assigned homework is not just busywork.
7
T ext(s) and other materials have helped me
understand course content.
8
Exams concentrate on important points of the course.
9
Exams are clearly worded.
10
Exams are good measures of my knowledge,
understanding, or ability to perform.
11
Grading procedure is fair and impartial.
12
Assignments are appropriately distributed
throughout the course of the semester.
13
Course as a whole has produced new knowledge,
skills, and awareness in me.
113
Disagree
Somewhat Somewha
Disagree t Agree
Agree
Very
Strongly
Strongly
Agree
Agree
Items about the Instructor: CHECKETTS MAX L - REL 233
Strongly
Somewhat Somewha
Disagree
Disagree
Disagree t Agree
1
Has an excellent knowledge of the subject matter.
2
Is enthusiastic about the subject.
3
Is well prepared for each class.
4
Makes good use of class time.
5
Gives clear examples and explanations.
6
Makes helpful feedback of my work (e.g., papers,
exams).
7
Clearly explains difficult concepts, ideas, or theories..
8
Responds respectfully to student questions and
viewpoints.
9
Is genuinely interested in helping me understand
the subject matter.
10
Is available to students during regular and
reasonable office hours.
11
Motivates me by his/her example to want to learn
about the subject.
12
Has produced new knowledge, skills, and
awareness in me.
13
Starts/dismisses class at scheduled times.
14
Seldom misses class.
Agree
Very
Strongly
Strongly
Agree
Agree
Agree
Very
Strongly
Strongly
Agree
Agree
Items about Core Values:CHECKETTS MAX L - REL 233
Somewhat Somewha
Strongly
Disagree
Disagree t Agree
Disagree
1
Appropriately brings Gospel insights and values
into secular subjects.
2
Inspires students to develop good character.
3
Helps students prepare to live effectively in
society.
Is spiritually inspiring insofar as the subject
matter permits.
4
Other Information: CHECKETTS MAX L - REL 233
a gre at
de al le ss
a little
less
about
the
same
a little
more
a gre at
deal
more
5
6
7
8
9
Compared to other college courses you have taken, would you say that you
have learned . . .
T he approximate number of hours
per week that I have spent in
outside preparation for this class is
...
1
2
3
114
4
ge ne ral
educatio
n
e le ctive
other
major
minor
90% to
100%
75% to
90%
50% to
75%
le ss
than
50%
neve r
atte nde
d
B
C
D
F
O the r
T his course fills requirements for my . . .
My class attendance has been . . .
A
T he grade I expect from this course. . .
Is there anything about this course and/or instructor that was particularly good? If so, what?
What could be done to improve this course to help you learn more?
115
APPENDIX III – NEW EVALUATION INSTRUMENT
BYU-IDAHO COURSE EVALUATION
Please evaluate the following instructor and course. If you feel an item is not applicable to your
course, then leave it blank. When you are finished, click on the SUBMIT button at the bottom of the
page. Your identity is completely anonymous. Please be as thorough and as accurate as possible. Your
feedback is highly valued. It is used by your instructor and the school's administration to improve
teaching.
Instructor:
CHECKETTS, MAX L
Course :
REL 233
CHURCH HISTORY
Section:
9
Overall Rating: CHECKETTS MAX L - REL 233
very poor
poor
fair
good
very good excellent
exceptional
What is your overall rating of this instructor.
What is your overall rating of this course.
Items about Student Performance in this Course:
Strongly
Disagree
1
I was prepared for each class.
2
I arrived at class on time.
3
4
5
6
I was an active participant in online or face-toface class discussion
I sought opportunities to share my learning with
others outside of class.
I worked hard to meet the requirements of this
class.
I sought opportunities to reflect on what I had
learned in the class.
7
I feel that I made important contributions to the
learning and growth of fellow classmates.
8
T he course as a whole had produced new
knowledge, skills, and awareness in me.
116
Disagree
Somewhat Somewha
Disagree t Agree
Agree
Very
Strongly
Strongly
Agree
Agree
Items about the Course:
1
Course objectives are clear.
2
Course is well-organized.
3
Student responsibilities are clearly defined.
4
Instructional resources--textbook(s), course
guide(s), online material, etc--were useful and
helped me to achieve course objectives.
5
Assessment activities--exams, quizzes, papers,
hands-on demonstrations, presentations, etc.-accurately and fairly measured the knowledge and
abilities I acquired from the course.
6
Class assignments contributed to my learning and
growth.
7
T he course provided opportunities to learn from
and teach other students.
8
Group work, if assigned, was beneficial and
meaningful.
9
Students were actively involved in this class
through discussions, group work, and teaching.
Strongly
Disagree
Disagree
Somewhat Somewha
Disagree t Agree
Agree
Strongly
Disagree
Disagree
Somewhat Somewha
Disagree t Agree
Agree
Very
Strongly
Strongly
Agree
Agree
Items about the Instructor:
1
T he instructor effectively modeled problemsolving and application of subject matter.
2
3
T he instructor
made good
of class time.
When
given, examples
anduse
explanations
were
clear.
4
T he instructor gave helpful feedback of my work.
5
constructively to student questions and
viewpoints.
6
T he instructor was available to me when I
requested assistance, in class or outside of class.
7
8
9
10
11
T he instructor motivated me by his/her
enthusiasm to want to learn about the subject.
T he instructor starts/dismisses class at scheduled
times.
T he instructor held me accountable for coming to
each class prepared.
opportunities to be an active participant in the
class.
T he instructor provided opportunities to reflect
upon my learning and experiences in the class.
117
Very
Strongly
Agree Strongly
Agree
Items about Core Values:
Strongly
Disagree
1
Appropriately brings Gospel insights and values
into secular subjects.
2
Inspires students to develop good character.
3
Helps students prepare to live effectively in
society.
Is spiritually inspiring insofar as the subject
matter permits.
4
Disagree
Somewhat Somewha
Disagree t Agree
Agree
Very
Strongly
Agree Strongly
Agree
Other Information:
a great
de al less
a little
less
about
the
same
a little
more
a great
deal
more
a great
de al less
a little
less
about
the
same
a little
more
a great
deal
more
5
6
7
8
9
Compared to other college courses you have taken, would you say that you
have learned . . .
Compared to other college courses you have taken, would you say that your
satisfaction is . . .
1
2
3
4
per week that I have spent in
outside preparation for this class is
...
general
educatio
n
elective
major
minor
90% to
100%
75% to
90%
50% to
75%
le ss
than
50%
never
attende
d
B
C
D
F
O ther
other
T his course fills requirements for my . . .
My class attendance has been . . .
A
T he grade I expect from this course. . .
118
APPENDIX IV – CORRELATION MATRIX FOR COURSE RATINGS
Fall
2007
1
Winter
2008
Winter
2008
.87
1
Spring
2008
.60**
.76**
1
Fall
2008
.70**
.79**
.82**
1
Winter
2009
.79**
.84**
.78**
.82**
1
Spring
2009
.74**
.82**
.70**
.77**
.72**
1
Fall
2009
.65**
.75**
.71**
.78**
.80**
.76**
1
Winter
2010
.65**
.70**
.64**
.72**
.77**
.66**
.73**
Fall
2007
Spring
2008
**. Correlation is significant at the 0.01 level (2-tailed).
119
Fall
2008
Winter
2009
Spring
2009
Fall
2009
Winter
2010
1
APPENDIX V- TWO SAMPLE T-TEST MEASURING SIGNIFICANCE
Mean Differences in Course Ratings
Period
Semester
Mean
Std.
Std. Error
Deviation
Mean
N
Compared
Fall 2007
5.46
33
.67
.11
Fall 2008
5.68
33
.64
.11
Winter 2009
5.72
33
.67
.11
Winter 2010
5.75
33
.51
.08
Between T1
Fall 2008
5.68
33
.64
.11
& T2
Winter 2009
5.72
33
.67
.11
Semester 1
Fall 2007
5.46
33
.67
.11
&8
Winter 2010
5.75
33
.51
.08
T1
T2
120
APPENDIX VI – CORRELATION MATRIX FOR PERCEIVED LEARNING
Fall 2007
Fall Winter Spring Fall Winter Spring Fall Winter
2007 2008 2008 2008 2009 2009 2009 2010
1
Winter 2008
.87
1
Spring 2008
.70
.67
1
Fall 2008
.79
.83
.73
1
Winter 2009
.73
.81
.58
.84
1
Spring 2009
.64
.66
.59
.83
.73
1
Fall 2009
.55
.68
.51
.78
.83
.72
1
Winter 2010
.67
.68
.51
.71
.71
.68
.73
**. Correlation is significant at the 0.01 level (2-tailed).
121
1
APPENDIX VII – PARAMETER ESTIMATES COMPARING PERCEIVED
LEARNING SCORES WITH EMPLOYMENT STATUS
Parameter Estimates
Dependent Variable: Perceived Learning Score
Parameter
Intercept
Std.
B
Error
-2.13
.05
t
-39.71
Sig.
.00
95% Confidence
Interval
Lower
Upper Partial Eta
Bound
Bound
Squared
-2.23
-2.02
.21
3.65
.06
56.89
.00
3.52
3.77
.36
Employment
Status=1YR
.00
.02
.37
.70
-.03
.05
.00
Employment
Status=ADJ
-.04
.01
-3.08
.00
-.07
-.01
.00
Employment
Status=CFS
.10
.01
7.36
.00
.07
.12
.00
Employment
Status=VET
0a
.
.
.
.
.
.
LM Scores
a. This parameter is set to zero because it is redundant.
122
APPENDIX VIII – MEAN LM SCORES BY SEMESTER BASED ON CLASS
SIZE
Mean LM Scores by Semesters Based on Class Size
Semester
Class Size
Mean
F09
0-10
.82
11-20
.84
21-30
.84
Over 31
.83
Total
.83
S09
0-10
.86
11-20
.85
21-30
.83
Over 31
.82
Total
.83
0-10
.84
11-20
.84
W09
21-30
.84
Over 31
.82
Total
.83
0-10
.84
11-20
.84
W10
21-30
.84
Over 31
.83
Total
.84
0-10
.84
11-20
.84
Total
21-30
.84
Over 31
.82
123
N
102
263
417
703
1485
126
238
308
532
1204
240
332
449
597
1618
97
268
435
589
1389
565
1101
1609
2421
Std. Deviation
.18
.11
.06
.06
.08
.13
.06
.09
.09
.09
.17
.07
.07
.06
.09
.14
.10
.06
.06
.08
.16
.09
.07
.07
APPENDIX IX – LM SCORES COMPARED TO PERCEIVED LEARNING
SCORES BY CLASS SIZE
Correlation Matrix of LM Scores and Perceived Learning Scores by Class Size
Class Size
LM Scores
Small
LM Scores
Perceived Learning
Score
Medium
LM Scores
Perceived Learning
Score
Large
LM Scores
Perceived Learning
Score
Very Large LM Scores
Perceived Learning
Score
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
Pearson Correlation
Sig. (2-tailed)
N
124
1
565
.50
.00
565
1
1101
.56
.00
1101
1
1609
.67
.00
1609
1
2421
.71
.00
2421
Perceived
Learning
Score
.50
.00
565
1
565
.56
.00
1101
1
1101
.67
.00
1609
1
1609
.71
.00
2421
1
2421