Florida Association of Institutional Research
2010 Annual Conference
Transiting to an Online Course Evaluation
Model: The Online Student Survey of
Instruction
February 2010
Student Survey of Instruction
Transiting to an Online Course Evaluation Model
February 2010
Presenters
 Dr. James Coraggio, Director, Academic
Effectiveness and Assessment
 Magaly Tymms, Assessment Coordinator,
Academic Effectiveness and Assessment
With contributions from…
Debbie Montalvo, SSI Coordinator
Richard Jakubowski, SSI DB Administrator
February 25, 2010
Florida Association of Institutional Research
2
SPC Background
Transiting to an Online Course Evaluation Model
February 2010
 SPC, established in 1927, is the oldest
2-year college in Florida
 First Community College in Florida to offer
4 year degrees (2002)
 Nine Campuses throughout Pinellas
County
 2008-09 FTE: 16,967 (LD), 1,677 (UD)
 Opening Fall 2009 credit enrollment:
29,560
 Annual 2008-09 headcount (credit and
non-credit): 61,414
February 25, 2010
Florida Association of Institutional Research
3
Purpose
Transiting to an Online Course Evaluation Model
February 2010
 This presentation describes the
development and implementation of a
college-wide online process to administer
course evaluations, the Student Survey of
Instruction (SSI), which takes advantage
of existing technology to efficiently collect
student responses, disseminate the results
in a more timely manner, and save the
institution time and money.
February 25, 2010
Florida Association of Institutional Research
4
Agenda
Transiting to an Online Course Evaluation Model






February 2010
Issues with the prior process
Intended outcomes
New online process
Dissemination of results and reports
Response rates
Next steps
February 25, 2010
Florida Association of Institutional Research
5
Issues with Prior Process
Transiting to an Online Course Evaluation Model
February 2010
 Expenses related to Scantron forms, and
numerous hours of sorting, organizing, and
transporting forms
 Time consuming process of transcribing openended items
 Not all Scantrons were scanned and
processed (scanning issues, data integrity
issues, etc.)
 Scantron results and online results captured
separately and reported separately
 Process conducted by multiple departments
with no central ownership
February 25, 2010
Florida Association of Institutional Research
6
Original SSI Items
Transiting to an Online Course Evaluation Model
February 2010
 Original SSI forms were developed by a
college committee a number of years ago
 Items were reviewed for content and a lecture
form was aggregated into four categories
(e.g., faculty student interaction, organization,
presentation, and evaluation)
 There were four SSI forms (Scantron):
Lecture, Non-lecture, Clinical A*, and Clinical
B*
 Results were scanned by a third party (PCSB)
and entered into PeopleSoft
February 25, 2010
Florida Association of Institutional Research
7
eCampus SSI Items
Transiting to an Online Course Evaluation Model
February 2010
 Online SSI form was developed later by
eCampus to assess online courses
 This form was similar to the Lecture
SSI form with some slightly modified
items and the same four categories
 Original online survey tool created to
capture summative information in
database
February 25, 2010
Florida Association of Institutional Research
8
Performance Improvement
Transiting to an Online Course Evaluation Model
February 2010
 From Compliance to Performance Improvement
Community
College Leadership
Program Directors
Faculty
Students
Assessment
Data
Reports
SPC
Data
Information
Knowledge
February 25, 2010
Improved Outcomes
Florida Association of Institutional Research
Improvement
9
Intended Outcomes
Transiting to an Online Course Evaluation Model
February 2010
 New SSI online process was designed to:
 Provide stakeholders (Faculty, Program
Administrators, etc.) information to assist in
making decisions that will result in
performance improvement
 Provide more reliable and valid course
evaluation information
 Create centralized coordination of the process
 Save the college time and money (est.
$45,000 per year)
 Eliminate the need for in-class administration
(more course time for instruction)
February 25, 2010
Florida Association of Institutional Research
10
SSI Online Process Timeline
Transiting to an Online Course Evaluation Model
February 2010
 Fall 2007
 Established an SSI Team
 Factor analysis (exploratory and confirmatory cross
validation) determined that lecture and online SSI forms
had three factors (e.g., faculty engagement, preparation
and organization, and course instruction)
 Spring 2008
 Pilot of 100 FTF course sections conducted
 Fall 2008
 First collegewide administration (All course ANGEL shells
enabled)
 Created an online reporting tool (program
administrators only)
 Revised older survey administration tool
February 25, 2010
Florida Association of Institutional Research
11
SSI Online Process Timeline
Transiting to an Online Course Evaluation Model
February 2010
 Spring 2009
 Created customized usage reporting tools
 Created new survey interface with an opt-out
capability
 Added Faculty Representative to SSI Team
 Created online reporting tool access for Faculty
(No more paper)
February 25, 2010
Florida Association of Institutional Research
12
SSI Online Process Timeline
Transiting to an Online Course Evaluation Model
February 2010
 Summer 2009
 First collegewide summer administration
 Modified faculty email notices to include link to
real-time response data
 Began working with HEC on Clinical Solutions
 Evaluated current SSI items and forms (SSI
Revision Committee)
 Fall 2009
 Implement self-service module so faculty can
revise their own administration dates
February 25, 2010
Florida Association of Institutional Research
13
SSI Online Process Timeline
Transiting to an Online Course Evaluation Model
February 2010
 Spring 2010
 College-wide Pilot of new SSI items set
 Pilot Clinical SSI in the areas of MLT, EMS and
COE
 Add standard deviation information to reports
 Identify best practices from faculty members
and programs with the highest response rates
February 25, 2010
Florida Association of Institutional Research
14
SSI Link in ANGEL
Transiting to an Online Course Evaluation Model
February 2010
 Course instructors are required at the start of
each term to:
 enable their ANGEL course shells,
 upload their course syllabus, and
 attached their SSI link into their course.
February 25, 2010
Florida Association of Institutional Research
15
Faculty Notification Emails
Transiting to an Online Course Evaluation Model
February 2010
 Faculty are provided four informational
emails during their SSI administration.
 Five days before the beginning
 At the 50% mark
 At the 75% mark
 Two days after the end
 Emails contain course, form, date, and
response information.
February 25, 2010
Florida Association of Institutional Research
16
Faculty Notification Emails
Transiting to an Online Course Evaluation Model
February 25, 2010
February 2010
Florida Association of Institutional Research
17
SSI Link in ANGEL
Transiting to an Online Course Evaluation Model
February 25, 2010
February 2010
Florida Association of Institutional Research
18
SSI Link in ANGEL
Transiting to an Online Course Evaluation Model
February 25, 2010
February 2010
Florida Association of Institutional Research
19
SSI Link in ANGEL
Transiting to an Online Course Evaluation Model
February 25, 2010
February 2010
Florida Association of Institutional Research
20
Accessing Information
Transiting to an Online Course Evaluation Model
February 2010
 A site was developed to provide faculty
and program administrators information
about the SSI online process:
http://www.spcollege.edu/central/ssi/
February 25, 2010
Florida Association of Institutional Research
21
Information about the Process
Transiting to an Online Course Evaluation Model
February 25, 2010
February 2010
Florida Association of Institutional Research
22
Information about the Process
Transiting to an Online Course Evaluation Model
February 25, 2010
February 2010
Florida Association of Institutional Research
23
Accessing Results
Transiting to an Online Course Evaluation Model
February 2010
 All SSI survey reports are available on the
online SSI Administration site:
http://it.spcollege.edu/ssi/
 Faculty and Program Administrators can
login by entering their SPC network
username and password
 Access is limited to only those courses
under their area of responsibility.
February 25, 2010
Florida Association of Institutional Research
24
Accessing Results
Transiting to an Online Course Evaluation Model
February 25, 2010
February 2010
Florida Association of Institutional Research
25
Accessing Results
Transiting to an Online Course Evaluation Model

February 2010
Instructors and Program Administrators can select ‘Term’ from the
drop-down menu (i.e. 0405) and select how they would like to
view the results
 Each individual section with its own results OR Combined
sections by instructor
February 25, 2010
Florida Association of Institutional Research
26
Accessing Results
Transiting to an Online Course Evaluation Model


February 2010
After selecting a term and indicating ‘Each Individual Section with
its own results’, users can view each individual section with its
own results, as selected on the previous screen for Term: 0405.
By selecting any of the courses, users can view the SSI report for
that course.
February 25, 2010
Florida Association of Institutional Research
27
Accessing Results
Transiting to an Online Course Evaluation Model
February 25, 2010
February 2010
Florida Association of Institutional Research
28
Accessing Results
Transiting to an Online Course Evaluation Model



February 2010
For the ‘Combined sections by Instructor’ options, users can view
combined sections with results, as selected on the term selection
screen (Term: 0405 in this example).
By selecting any of the instructor names, users can view a
combined SSI report for that instructor.
Note: multiple reports for a single instructor may be listed if the
instructor taught courses with either different SSI survey forms,
different academic orgs, or on different campuses.
February 25, 2010
Florida Association of Institutional Research
29
Accessing Results
Transiting to an Online Course Evaluation Model
February 25, 2010
February 2010
Florida Association of Institutional Research
30
Accessing Results
Transiting to an Online Course Evaluation Model
February 2010
 For an electronic version of the report (pdf), users can
select ‘View in Adobe Reader’
 ‘Each Individual Section with its own results’ report
produces individual SSI results, three comparative
measures, graph, and student comments.
 ‘Combined sections by Instructor’ report produces
an instructor’s combined sections SSI results with
three comparative measures, graph, and student
comments by course. This report is somewhat
similar to the SSI reports generated by AIS in the
past.
February 25, 2010
Florida Association of Institutional Research
31
Administration Dates
Transiting to an Online Course Evaluation Model
February 2010
 SSI is intended to be an evaluation of the instruction
provided during the course.
 It is administered after the students have participated in
the majority of the course (no sooner than the 60% mark
of the course) in order to provide the student an
opportunity to experience the course instruction.
 Length of the administration period is approximately the
last 20%-25% of the course.
 Faculty are provided default SSI administration start and
end dates, and are permitted to make minor adjustments
to these dates within a predetermined timeframe. This
ensures a more standardized and consistent data collection
process.
February 25, 2010
Florida Association of Institutional Research
32
Changing Administration Dates
Transiting to an Online Course Evaluation Model
February 2010
Users can locate the course that they would like to
change and select the Calendar icon
February 25, 2010
Florida Association of Institutional Research
33
Changing Administration Dates
Transiting to an Online Course Evaluation Model
February 2010
 They then click on “Next Month” on the upper right
hand side of the calendar to select the next month. The
sample calendar above shows the course ends on
12/18 and the SSI closes on 12/11.
February 25, 2010
Florida Association of Institutional Research
34
Changing Administration Dates
Transiting to an Online Course Evaluation Model




February 2010
To extend the SSI to remain open until 12/18…
Select 12/18. It will turn blue through 12/18. This changes in real-time.
Should you make a mistake, simply click back on the original survey end
date of 12/11 or select another.
The latest you can extend the survey to is 12/24.
February 25, 2010
Florida Association of Institutional Research
35
Response Rates
Transiting to an Online Course Evaluation Model
February 2010
 Student response rates are calculated and
compared each term by program
(academic org), campus, form type, and
overall college.
February 25, 2010
Florida Association of Institutional Research
36
Response Rates
Transiting to an Online Course Evaluation Model
Campus
Allstate Center
Submission Entries by Campus for Submission Entries by Campus for
0400 Term
0405 Term
Roster Entries Opt-outs* Average Roster Entries Opt-outs* Average
February 2010
SSI Submissions Split by Campus
0410 Term
Roster Entries Opt-outs Average
1811
1388
0 76.6%
2540
1537
23 60.5%
1226
624
25
50.9%
Clearwater
Campus
16001
7194
0 45.0%
15292
5469
48 35.8%
6527
2209
35
33.8%
Electronic
Campus
EpiCenter
9563
908
5937
451
0 62.1%
0 49.7%
10658
1022
4785
295
100 44.9%
1 28.9%
5679
271
2674
113
57
2
47.1%
41.7%
Health
Education
Center
Open Campus
8484
654
5121
151
0 60.4%
0 23.1%
9271
556
4438
132
74 47.9%
3 23.7%
4242
52
2091
24
97
0
49.3%
46.2%
Seminole
Campus
8461
4911
0 58.0%
8101
3610
58 44.6%
4226
1942
41
46.0%
20991
9332
0 44.5%
20706
7786
71 37.6%
8336
2739
23
32.9%
15736
7034
0 44.7% 12634
6006
51 47.5%
82609 41519
0 50.3% 80780 34058
429 42.2%
*Opt-outs are included in the number of Entries and were not available during Fall Term
4691
35250
2310
14726
48
328
49.2%
41.8%
St Pete/Gibbs
Campus
Tarpon
Springs
Campus
Totals
February 25, 2010
Florida Association of Institutional Research
37
Response Rate Comparison
Transiting to an Online Course Evaluation Model
February 2010
 Comparison with another Florida public
community college
 Another community college debuted its online course
evaluation system in Fall 2007. Until the Spring 2007 term,
this community college relied on a paper-based course
evaluation system for its traditional classroom courses.
 Response rates:
Pilot test
Spring 2007: 30% classroom, 26% distance learning
Collegewide
Fall 2007: 20% classroom, 26% distance learning
Spring 2008: 24% classroom, 29% distance learning
February 25, 2010
Florida Association of Institutional Research
38
SSI Revision Committee
Transiting to an Online Course Evaluation Model
February 2010
 An SSI Revision Committee was established
in the Summer of 2009 to review the current
SSI forms and make suggestions for revisions
to the current items sets.
 The new modality-neutral form was designed
to address faculty and student engagement in
the learning process and is being
administered this term as a college-wide
pilot.
February 25, 2010
Florida Association of Institutional Research
39
Increasing Response Rates
Transiting to an Online Course Evaluation Model
February 2010
 While not the primary focus of the SSI
Revision Committee, there was
considerable interest in identifying and
implementing ideas and processes to
increase the response rate of the SSI.
 Through creative brainstorming at the
meetings and online, a list of “best
practices” was developed.
February 25, 2010
Florida Association of Institutional Research
40
Increasing Response Rates
Transiting to an Online Course Evaluation Model
February 2010
SSI Revision Committee Recommendations
1.
2.
3.
4.
5.
6.
7.
8.
9.
Send Constant Reminders (‘inescapable messaging’)
Stress Confidentiality
Imbed link in announcement page -- “inescapable messaging”
Move every week to top of unit
Link to “content” – graded or non-graded
Bring students to lab /use of lab
Give extra credit
Genuinely convey the message that the information will be used
to make this class better
Hold activities on campus, e.g., a table highlighting importance
for accreditation
February 25, 2010
Florida Association of Institutional Research
41
Increasing Response Rates
Transiting to an Online Course Evaluation Model
February 2010
 There was some concern that item #5 (linking the SSI
to accessing exams or other course materials) was
not acceptable from a legal standpoint.
 The question was posed to the Office of General
Counsel, and, in brief, the answer was ‘yes’, the
faculty could block access to course materials
assuming that the content ‘blocked’ is just the next
module or assignment the student would see, not
specifically an exam.
 The Office of General Counsel also advised against
giving extra credit for taking the SSI.
February 25, 2010
Florida Association of Institutional Research
42
Next Steps
Transiting to an Online Course Evaluation Model
February 2010
 Work with programs to improve
response rates
 Finish creating forms for programs
using clinical items
 Develop reports which provide trend
information across terms and years
 Finalize new items
February 25, 2010
Florida Association of Institutional Research
43
Questions
Transiting to an Online Course Evaluation Model
February 25, 2010
February 2010
Florida Association of Institutional Research
44
Florida Association of Institutional Research
2010 Annual Conference
Transiting to an Online Course Evaluation
Model: The Online Student Survey of
Instruction
February 2010