Realizeit at the
University of Central Florida
Results from initial trials of Realizeit at the
University of Central Florida, Fall 2014
1
Based on the research of:
Dr. Charles D. Dziuban, Director
charles.dziuban@ucf.edu
Dr. Patsy D. Moskal, Associate Director
patsy.moskal@ucf.edu
Research Initiative for Teaching Effectiveness
University of Central Florida
Written by:
Dr. Colm Howlin, Principal Researcher
colm.howlin@realizeitlearning.com
Realizeit Learning
Disclaimer: This document presents Realizeit’s interpretation of the results presented by the researchers.
The University of Central Florida does not endorse or recommend any commercial products, processes,
or services.
Table of Contents
02
Introduction
03
Effect on Student Outcomes
09
Student Perceptions
15
Conclusions
2
Introduction
This document summarizes the results of a research study
carried out by Dr. Charles D. Dziuban (Director) and Dr. Patsy
D. Moskal (Associate Director) of the Research Initiative for
Teaching Effectiveness at the University of Central Florida
(UCF). The study was conducted during an initial trial of the
Realizeit system in two Fall 2014 courses and was designed to
provide an initial gauge of the effectiveness and impact of
Realizeit on student outcomes. In addition to effectiveness,
the UCF research team also measured student’s perceptions
of instruction and Realizeit through the use of surveys.
gauge the
effectiveness
and impact of
Realizeit on
student
learning
The study followed two courses: one in the field of Psychology
(General Psychology), and one in the field of Nursing
(Physiospathology). To allow the effectiveness of Realizeit to
be measured and compared, the courses were run utilizing
three different instructional models. For each course, one
group of students used Realizeit, one group used the current
UCF online platform, while the remaining group engaged in
traditional face-to-face or non-adaptive online learning. The
Psychology and Nursing comparison courses had several
sections, each with a different instructor.
The Psychology course consisted of 8 objectives, covered 213
concepts and had 125 students enrolled. The Nursing course
consisted of 3 objectives, covered 42 concepts and had 34
students enrolled. These students covered a wide range of
demographics.
The remainder of this document is divided into two sections.
The first examines research into the effect of Realizeit on
student outcomes. This is achieved by comparing outcomes on
both internal and external exams. Following this, the UCF team
built two predictive models which use key learning metrics
from Realizeit to predict outcomes on an external exam.
The second section examines student’s perceptions and
satisfaction with their course experience with Realizeit and the
course format. In addition to the survey, the researchers ran
a principal component analysis, to determine if there were any
underlying themes present in the response data.
3
Effect on Student Outcomes
One of the core questions the UCF research team wished to
address was the effect Realizeit had on student outcomes.
Faculty from Psychology and Nursing had agreed to redesign
their online courses to incorporate Realizeit. The instructors of
courses delivered through Realizeit had access to the
instructor dashboard of Realizeit, which offered them a
detailed real-time view of the progress and achievement of
their students. The research team compared these courses
with corresponding Nursing and Psychology sections which
were taught face-to-face or fully online with different
instructors.
Realizeit...
outperformed
both face-toface and the
online
supported
groups
The research team used the grades achieved by the students
in their courses as a measure of learning. For the Psychology
course they also had available the results obtained by the
students on an external General Education Program (GEP)
exam. One interesting component to this research was the
construction of a predictive model that utilized learning
metrics gathered by Realizeit to predict the student outcomes
on the external GEP exam. Each of these components is
discussed in more detail in the following subsections.
Grade Distributions
The first comparison carried out by the researchers was to
examine the student success rates on each of the courses
across the three different instructional models. In this case, a
successful student was defined to be a student that achieved
a grade of A, B, or C.
The results are summarized in Figure 1. During this initial trial,
students who were supported by Realizeit throughout their
course outperformed both face-to-face and the online
supported groups in both courses. While the differences were
not found to be statistically significant due to the small sample
sizes, the differences are of interest, particularly the 7 point
difference in Psychology.
As mentioned earlier, within the face-to-face Psychology group
and the online Nursing group, there were several classes; each
4
with a different instructor. Each instructor was responsible for
grading their class according to their own grading scheme.
This led to a wide variability in the distribution of results
obtained across these classes. Using an adaptive system such
as Realizeit can provide students with a common basis for
assessment. This allows this variability in learning and grading
to be minimized, all while ensuring that each student receives
instruction that is both personalized and adapted to his or her
unique needs.
Realizeit is a system that improves with time and data. These
initial findings suggests that Realizeit not only performed well,
but allowed the students it supported to moderately
outperform those in the other groups, some of whom were
supported by highly experienced instructors. This is certainly
not typical as most new learning environments would yield an
initial drop in outcomes due to the learning curve associated
with the rollout.
Additionally, as the system improves with data and course
content iterations we should expect Realizeit to improve and
surpass what has already been achieved. Due to the granular
nature of data collected by Realizeit, visibility into how these
results were achieved by the students and instructors is
available to the institution, providing a basis for improvement
with each iteration of the course.
Figure 1: Student
success rates (A, B, C
grades) with Realizeit
compared to other
course formats
5
Psychology GEP exam
As a part of their final assessment, students on the
Psychology course were required to complete an external
General Education Program (GEP) exam. This provided an
opportunity to use an objective measure of achievement
across all groups within the course regardless of the
instructional model they used.
Realizeit
produced a far
more
homogenous
set of results,
without the
extreme lows
experienced
in the other
groups
The researchers found that there was no statistical or practical
difference between the average performance of students from
each group, with each group achieving an average of
approximately 85% (depicted via the circles in Figure 2).
However there is a large difference in the range of results
obtained. For each group, the range is depicted using the
horizontal bars while the average of the low outliers are
depicted using the diamonds in Figure 2.
The researchers found that the group of students who used
Realizeit produced a far more homogenous set of results,
without the extreme lows experienced in the other groups.
This suggests that there may be a level of consistency in the
outcomes and learning experienced by students using Realizeit
that is not found across those in the other groups.
As mentioned by the UCF researchers, one possible
explanation for the large range of results in both the face-toface and online groups is the huge variability in the experience
and standard of instructors and their grading schemes. This
variability can be smoothed out by a system such as Realizeit,
which can account for individual needs.
Figure 2: The outcomes
on the GEP exam. The
horizontal bar
represents the range of
results for each section,
the square is the
average result, and the
circle represents the
average of the low
outliers.
6
It is also worth noting that the GEP exam accounted for a
different percentage of a student’s final course grade
depending on which group they were in. This could bias the
results somewhat due to it being more important for some
groups than others.
Model to Predict Outcome on
Psychology GEP Exam
Given the results on the external Psychology GEP exam, the
researchers were interested in determining if there is a
relationship between these results and the metrics reported
by Realizeit. Building on this, they wished to use complex
relationships to construct a predictive model. They ultimately
built two models. The first looked at predicting the GEP exam
results from Realizeit module scores. The second used
Realizeit-specific learning metrics to achieve the same goal.
The Psychology course was broken into 8 objectives or
modules, each of which contained several nodes or concepts.
Realizeit provides detailed granular metrics on each of these
concepts and allows them to be aggregated to the module or
course level.
Realizeit Module Scores
The first model used the learner’s score, as generated by the
course specific grading formula within Realizeit, aggregated at
the module level to predict the GEP exam scores. Figure 3
displays the correlation of each of the module scores with the
GEP exam score. All modules correlated positively with the
exam score with most having a strong to very strong linear
relationship. This means that the higher the score the student
achieved on these modules, the higher their exam score. The
main outlier module is “History and Research.” This could be
explained by the fact this is the first module in the course and
mainly serves as an introduction.
These strong relationships suggest that a predictive model is,
indeed, possible. Using a stepwise linear regression procedure,
the research team at UCF were able to construct a model using
scores on just three of the eight modules that produced an
adjusted 𝑅2 of 0.64.
7
Sensation, Perception & Learning
Emotions and Health & Personality} 𝑅 2 (𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑) = 0.64
Human Diversity & Development
This means that the model explained 64% of the variability
seen in the GEP exam scores and also suggests that this model
could potentially act as a method to predict student readiness
for this external exam. Further work would need to be
completed to trial this model on unseen cases to determine its
accuracy in a real world setting.
Realizeit Learning Metrics
This model examined the relationship between the Realizeit
learning metrics aggregated to the course level and GEP exam
score. Figure 4 displays the correlation between each metric
and the exam score. There is more variation observed here
than with the previous model: there are 2 metrics with strong
positive correlations, 2 that are very weak and negative, and
the remainder cover the full range in between.
Again, the UCF researcher team used a stepwise linear
regression procedure to construct the model and found that
just three of the metrics could produce a model with an
adjusted 𝑅2 of 0.67.
Figure 3: The correlation
of individual Psychology
module scores with the
GEP Exam score
8
The metrics that were selected are Num Records (which can
be seen as a measure of course engagement), Calculated
grade (average score as calculated by the grading formula
across all modules) and Knowledge Covered Growth (how
much knowledge the student has attained since they started
using Realizeit).
Num Records
Calculated grade
} 𝑅 2 (𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑) = 0.67
Knowledge covered growth
Both of these models show that there is a strong relationship
between the Realizeit measures of performance and learning
and the external independent measure. By developing these
models, the UCF research team has demonstrated the
potential for these metrics when building predictive models for
other external exams. This would allow institutions to make
accurate predictions for individual learners and build an exam
readiness tool that could function as an early warning indicator
of at-risk students.
Figure 4: The correlation
of individual Realizeit
metrics with the GEP
Exam score
9
Student Perceptions
83.7% would
use Realizeit
again
In addition to measuring the effect of Realizeit on student
learning outcomes, Dr. Dziuban and Dr. Moskal were also
interested in tracking students’ perceptions of their learning.
This involved the students completing two surveys. The first
was designed specifically for students who used Realizeit in
order to measure their perceptions of the system. The second
was the Student Perception of Instruction (SPI) survey, which
is administered to all UCF students at the end of their courses.
In the following subsections we will outline the results from
both surveys. This will include a principal component analysis
conducted by the UCF researchers of the student responses to
the Realizeit survey. This will determine if there are any
underlying factors that may exist behind their response
patterns.
Student Perceptions of Realizeit
82.8% felt
they learned
the course
material
better as a
result of using
Realizeit
The responses of the students to the survey questions are
summarized in Figure 5. Some of these questions address
Realizeit directly; others address components that are under
the control of the course designers and administrators, such
as difficulty and clarity of content. It is worth noting that some
of these metrics will improve with further iterations of the
course as the curriculum and content improve.
In general, the student responses were positive. If given the
choice, 83.7% would use Realizeit again and 82.8% felt they
learned the course material better as a result of using
Realizeit. The remaining questions in the survey help shed
light on how these high percentages were achieved.
Students responded (89.4%) that Realizeit was easy to use
and provided useful and clear feedback and guidance; 91.2%
found the instructions clear, and 86.9% found the feedback
provided by the system helped them stay on track.
For a learning system to be effective the students must trust
it. From the questions on accuracy, the UCF researchers found
that the students trusted that Realizeit accurately captured
10
their learning progress. Further, 77.7% found that the
assessments were effective and 80.9% reported that the
ability levels as measured by Realizeit were accurate.
The key to any adaptive learning system is that it becomes
personalized to each learner over time, and this then drives
Figure 5:
The student
responses
to survey
questions
11
engagement. The students in this study found this to be the
case with 73.5% reporting that the system became
personalized to them and 80.9% indicated it increased their
engagement.
In addition, a learning system must select and deliver
concepts and learning content to each student at the right
time, pace, and difficulty. The learning must be difficult
enough to be challenging but easy enough to be achievable.
Most students felt the learning fell into this category with 55%
reporting that the material was neither too easy nor too hard.
73.5% felt
that the
system
became
personalized
to them and
80.9% felt it
increased
their
engagement
The trade-off for students was time, with 53.9% feeling they
spent more time in the Realizeit-supported class than
compared to a traditional class. However, as we can see from
the previous responses, this additional time was associated
with higher engagement, it resulted in them learning the
course material better, and overall most would like to use
Realizeit again.
The results from the questions on interactions with other
students were quite interesting. The students felt that they
interacted less with others as a result of using Realizeit
(70.1%). However, most students prefer little or no interaction
(57.7%). This insight demonstrates that there is potential for
further study in this area in order to create an environment
that fosters useful, effective and engaging interactions
between learners in an online setting.
From the comments section of the survey, the students noted
the Realizeit features they liked most. The comments broadly
fell under the headings of:








Ease of use (51 comments)
Personalization (31 comments)
Organization (31 comments)
Improved outcomes (12 comments)
Timing (12 comments)
Self-paced (8 comments)
Interactive (6 comments)
Guidance (6 comments)
12
Principal Component Analysis of
Student Perception of Realizeit
Principal component analysis determines if there are any
underlying hidden traits or components that exist behind a set
of data. The research team at UCF used this technique to
analyze the student responses. The team found two traits. The
relationship between each of the survey questions and the
traits can be visualized in Figure 6. The researchers labeled
the traits as follows:


Did Realizeit create an “Effective Learning Climate?”
What was the level of “Engagement Effort” required?
These new traits allowed the researchers to summarize the
students’ answers to the survey questions using these two
components.
Following on from this analysis, the research team then
compared the students’ scores on these components to their
answers to the survey item “Realizeit helped me learn.” They
found that the more positive the student’s response to
“Realizeit helped me learn” the higher the “Effective Learning
Climate” and the higher the “Engagement Effort” components.
Interestingly, this suggests that although it required more
effort, the willingness to engage was high because students
believed it created an effective learning climate. This supports
the conclusions from the previous analysis and implies that if
Figure 6: The reduction
of the space from 17
questions to 2
underlying principal
components. The
shading represents the
size of the coefficient
that relates a question
to the components.
13
you can create the right learning climate, then students will
engage and expend the effort.
The key ingredient to creating an effective learning climate is
to create a system that the learner trusts and believes will help
them learn. The best instructors know how to create this
environment in their classrooms.
Student Perception of Instruction
the use of
Realizeit lead
to much
improved
ratings of
excellence in
instruction
Figure 7: SPI – The
ability to communicate
ideas effectively –
percentage who gave a
rating of excellent
The student perception of instruction survey is given to all
students at UCF at the end of each course. The research team
found that students who used Realizeit were more (and in
some cases much more) satisfied with the instruction they
received than those in other groups. The figures below show
the percentage of students who provided a rating of excellent
for three different areas split by course.
A higher percentage of students who used Realizeit provided
a rating of excellent on the courses’ ability to communicate
ideas effectively (Figure 7), the creation of an effective
learning environment (Figure 8) and on overall satisfaction
(Figure 9). This holds true for both the nursing and psychology
courses. This demonstrates that the personalized learning
14
environment led to very positive ratings of excellence in
instruction.
Figure 8: SPI – The
ability of the course to
create an effective
learning environment –
percentage who gave a
rating of excellent
Figure 9: SPI – Overall –
percentage who gave a
rating of excellent
15
Conclusion
The main conclusions drawn from the initial trials of Realizeit
at UCF are as follows:









Students who used Realizeit on average moderately
outperformed those students who used traditional and
online instructional models.
Realizeit students’ results on the external Psychology
exam were far more homogeneous, without the low
outlier values found in the other groups.
Realizeit metrics correlated well with an external
measure of achievement and could be used to generate
a predictive model.
Overall the students who used Realizeit where satisfied
with the system. 83.7% would use Realizeit again and
82.8% felt they learned the course material better as a
result of using Realizeit. Students who used Realizeit
were more satisfied with the instruction they received
than those in other groups.
Realizeit was easy to use and provided useful and clear
feedback and guidance. 89.4% agreed that Realizeit
was easy to use while 91.2% found the instructions
clear.
Students trusted Realizeit: 77.7% found that the
assessments were effective and 80.9% reported that
the ability levels as measured by Realizeit were
accurate.
73.5% found that Realizeit became personalized to
them over time
80.9% felt Realizeit increased their engagement.
The principal component analysis suggests that
although it required more effort, the willingness to
engage in learning in Realizeit was high because
students believed it created an effective learning
climate.
Dr. Dziuban and Dr. Moskal are continuing their research on
Realizeit. In the Spring 2015 semester they extended this
study to include 3 full courses delivered with the help of
Realizeit. This includes:
16



One group of 172 students from the General Psychology
course,
Three groups (one fully online and two blended) totaling
94 students from the Nursing (Physiopathology) course,
Two groups totaling 59 students from a College Algebra
course.
As the work presented here is published by the researchers,
appropriate references will be added to this document.
17
Contact Us
Sales & Marketing
www.realizeitlearning.com
Chicago
220 N Smith St, Suite 228,
Palatine, IL 60067
Telephone (Toll Free): (888) 959 2814
Telephone (Local): (224) 836 5316
Email: info@realizeitlearning.com
Research & Development
Dublin
Hibernian Estate, Greenhills Road,
Tallaght, Dublin 24
Telephone: +353 (1) 4683 800
Email: research@realizeitlearning.com
www.realizeitlearning.com