IN
PRACTICE
Purdue Signals
Mining Real-Time Academic Data
to Enhance Student Success
How Purdue University is changing the academic behavior of struggling students.
By Matthew D. Pistilli and Kimberly E. Arnold
T
hose of us who work with college students outside
of the classroom often spend a great deal of time
working to ensure that our students are excelling
in their coursework and integrating themselves into the
social culture of the campus. Our experience, however, has indicated most undergraduates, particularly
first-year students at large research-intensive institutions, seldom if ever have a good grasp on how they
are doing in their courses. A conversation we had with
a student around the eighth week of courses went
something like this.
Us: How are your classes going?
Student: Um . . . OK, I guess.
Us:You’re not sure? Are you going to class? Have you
been turning in assignments?
S tudent : Yeah, I go to class. I usually do the
homework. I’ve only gotten a couple of things back, and
I did OK on those.
Us: Do you know what you’re getting in the class?
Student: Not really. I don’t even know how to figure
out what I need to do to figure out my grades . . .
And you get the picture from here. We’re willing
to bet that at some point in your career you’ve had
one, if not several, conversations like this one.
The fact of the matter is that students often do not
know how they’re doing in a class for any number of
reasons. It could be that they only have a few grades in
a class—a midterm, a term paper, and a final—and so
it’s difficult to tell how they are doing until final grades
are released. In other classes, there may be a multitude
of assignments, and students just do not understand
Published online in Wiley InterScience (www.interscience.wiley.com).
© 2010 by American College Personnel Association and Wiley Periodicals, Inc.
DOI: 10.1002/abc.20025
22
ABOUT CAMPUS / july–august 2010
how to calculate how they are doing in the class or
course management system, with students’ use of
determine how well they need to perform on exams
academic resources and help labs. Through combinto achieve a passing grade. Furthermore, by the time
ing this information in a way that provides feedback
they do have an understanding of how they are doing
to students, we can help them understand both their
in their classes, it’s usually past the point where they
current grades in their classes and what they can do to
can withdraw from the class, which sometimes results
earn a higher grade while there is still time to act.
in students simply failing the course because they’ve
Most early warning systems (EWSs) rely on midgone beyond the point of being able to salvage a passterm grades reported by faculty members. The problem
ing grade.
with this model is that by the middle of a semester,
At Purdue, we’ve
it is generally too late for
developed Signals as a
students to rectify the issue
means of helping stu with their courses, thereby
Through combining this
dents better understand
rendering them with at
where they stand gradebest a W (withdrawal), or,
information in a way that
wise early enough so that
more often, a D or an F in
they can seek help and
the
course. Other systems
provides feedback
raise their grade or drop
use high-touch models,
to students, we can help
the course without the
wherein students bring
penalty of a failing grade.
weekly or biweekly grade
them understand both their
We knew that many of
reports to academic adviour large first-year classes
sors or success coaches
current grades in their
had significant numbers of
who work with students to
classes and what they can do
students who earned Ds or
improve their grades. This
Fs and that many of these
requires a huge amount of
to earn a higher grade while
same classes had instrucstaffing and time, sometors who utilize a centhing that most researchthere is still time to act.
tral course management
intensive institutions simply
system (CMS). We also
don’t have. Some EWSs
knew, through some data
utilize attendance behavior
mining, that many of those instructors kept students’
as a predictor of success. This, however, requires attengrades in the CMS gradebook—meaning that a studance to be taken, and that doesn’t always happen in
dent could look up his or her grades on assignments
250–400-student lecture classes. By tapping into existat any time. What became clear to us was that these
ing CMS data, we were able to create an EWS that
three things could operate in a coordinated effort that
allowed us to overcome these obstacles and let students
could result in higher levels of student success. Thus
know how they are doing.
was born Signals, a system that allows for the integraWhile EWSs have been around for quite some
tion of assignment grades and attendance behavior,
time, Signals goes a step beyond the traditional modboth of which can be recorded in the University’s
els. First, Signals was built from the ground up using
empirical data at every stage to ensure the most predictive student-success algorithm. This data-driven menMatthew D. Pistilli holds his doctorate in higher education
tality has allowed us to show its usefulness to faculty
administration and is a senior assistant director in student
access, transition, and success programs. In that role, he
and staff at Purdue—so much so that over 7,000 stuconducts evaluation on student-success programs, including
dents were enrolled in a course utilizing Signals during
orientation, learning communities, and low-income student
the fall 2009 semester.
scholarship/support programming.
Second, Signals is a behaviorally based model. This
Kimberly E. Arnold is an educational assessment specialist
means that while some consideration is given to data
for information technology at Purdue University. She is the
such as past academic performance, far more weight is
project manager for Purdue’s Signals program and is curgiven to student effort in terms of interaction with the
rently completing her doctorate in educational technology
CMS (e.g., taking online quizzes, reading articles, and
at Purdue.
exploring resources) and help-seeking behavior, so that
students start on an even playing field at the beginning
We love feedback. Send letters to executive editor Jean M.
of the course. After all, a student of average intelligence
Henscheid (aboutcampus@uidaho.edu), and please copy her
on notes to authors.
that works hard is just as likely to get a good grade as
23
ABOUT CAMPUS / july–august 2010
a student that has above-average intelligence but does
sages from their instructor) also sought help earlier in
not exert any effort; hence, the inclusion of effort in
the semester as well as more frequently than those that
the model. Help-seeking behavior is important because
did not. We also know that over 65 percent of students
it is an indication of the effort a student exerts outin danger of earning a D or an F increased help-seekside of class—going to see a
ing behavior, interaction
professor, consulting with a
with the CMS, and course
teaching assistant, visiting a
attendance, which resulted
By comparing a student’s
resource room, or attendin higher grades at the end
ing a review session. By
of the intervention period.
performance in a course
comparing a student’s perBeyond the perforformance in a course with
mance
and behavior data,
with other students in
other students in the same
surveys, focus groups, and
the same course, the true
course, the true innovation
interviews with students
of Signals is realized. The
produced a resoundingly
innovation of Signals is
comparison gives students
favorable perception of
a true sense of what they
Signals and their instrucrealized.
need to do to continue
tors’ effort. Students got
excelling in the course or
a true impression of direct
improve their performance
interaction with their
by letting them know (for students not doing well)
instructor—they no longer felt like “just a number in
what their peers are doing (e.g., turning in homework,
a huge lecture class,” and they shared surprise with the
attending help sessions).
fact that their instructor “really cared about how they
A final advantage of Signals over traditional
did.” Further, 89 percent of students that experienced
EWSs is that the identification of an at-risk student
Signals had a positive experience and 73 percent said they
directly triggers interventions set up by instructors.
would like Signals in every course.
For example, an instructor could opt to have a stuSo, let’s revisit our previous conversation with a
dent currently earning a C receive an automatically
student who was enrolled in a course utilizing Signals.
generated e-mail telling him to attend a help sesUs: How are your classes going?
sion that weekend, whereas a student earning an F
would get an e-mail telling her to make use of help
Student: Well, it was rocky at first, but it’s going
resources but would also get a phone call from her
much better now.
advisor. When grade or attendance information in the
Us:That’s great to hear.What changed?
CMS is updated, the student dashboard is automatiStudent:Well, I got e-mails from my instructor telling
cally “refreshed,” allowing the student to track his or
me things I could do to do better in the course. And I
her progress as the semester proceeds. In short, Signals
got an update every week so I knew if my grade was
is an EWS that provides feedback and directs students
improving.
to resources that can result in increased levels of stuUs: So what kinds of things are you doing now to keep
dent success—and decreased numbers of Ds and Fs in
your grades up?
first-year courses.
To assess the effect of Signals, we looked at two
Student:Well, I have been going to help sessions, and
outcomes: performance, as measured by final grade outI found my instructor’s office and I go see her during
come, and behavior, as indicated by the students’ interoffice hours sometimes, and I study with other students
action with the CMS and their help-seeking behavior.
in the class . . .
We compared the grades of control and experimental
Clearly this student benefited from the real-time
groups of students in the same courses over two semesfeedback provided by Signals and has gained skills that
ters in order to eliminate as many confounding variwill benefit him as he persists through college. Here,
ables as possible. The results of two semesters of data
in our minds, is what it comes down to. Our job is to
showed us that there is a consistently higher level of Bs
help students be successful in their coursework, and
and Cs and lower levels of Ds and Fs from students that
Signals gives us one more tool to increase their acahad a course that utilized Signals. Students that received
demic prowess.
interventions (explicit and detailed e-mails or text mes-
•
24
ABOUT CAMPUS / july–august 2010