2009 RP Group Awards
Excellence in Research—College/District Project
Title of Entry
Author(s)
Institution
Address
City/state/zip
Phone
Email
Project Summary
Analyzing Data with Appropriate Methods for a Class Schedule
Mode Survey
Xi Zhang, Campus Based Researcher and Jessica Luedtke,
District Research and Planning Analyst
San Diego Community College District
3375 Camino del Rio So.
San Diego, CA 92108
619-388-6941
xzhang@sdccd.edu
jluedtke@sdccd.edu
The study used a survey to explore student satisfaction and
preferences regarding three types of class schedules intended to
assist students in registration decisions and behavior. A 22question online survey was developed, and 2,165 students
responded to it. ANOVA, repeated measures ANOVA, and
item response theory were used in the analysis. Results
revealed no obvious differences in student satisfaction with the
schedules. Nevertheless, there is a slight difference in students’
preferences for one type of class schedule.
Analyzing Data with Appropriate Methods
for a Class Schedule Mode Survey
Xi Zhang, Campus Based Researcher, City College
Jessica Luedtke, District Research and Planning Analyst
San Diego Community College District
Presented at the:
California Association for Institutional Research
2008 Annual Conference
Pasadena, CA: November 14, 2008
Purpose of the Presentation
The primary purpose of this presentation is to share
the research methodology and innovative
techniques for instrument refinement and data
analysis rather than to disseminate results of the
study.
A second purpose is to support the idea of data
driven decision making in community college
districtwide research and planning.
Introduction of the District
San Diego Community College District
z 2nd largest district in the state
z Three 2-year colleges and eleven continuing ed
campuses
z Serves approx. 100,000 students each semester
Introduction of the Study
z
Three class schedules offered at the district
1)
2)
3)
z
z
New trend in student enrollment behavior
demonstrates more multi-campus registration
Two proposals were made in Fall 2007
1)
2)
z
Individual paper class schedule for each of the 3 colleges
Paper class schedule combining all 3 colleges
Online class schedule combining all 3 colleges
Broader distribution of combined college class schedule
Elimination of individual college class schedule
Two surveys were developed and implemented
1)
2)
Spring 2008 Baseline Survey
Fall 2008 Follow up Survey
The Spring 2008 Baseline Survey
Framework
z
Purpose of the Survey
z
Population and Sample
z
Survey Instrument
z
Sampling Procedure
and Data Collection
z
Data Analysis
z
Results
Purpose of the Study
z Two
proposals require research to provide
data-driven decision-making
z Baseline
data were collected to examine
the use of and preference for schedule
modality and student satisfaction with
schedule modality
Population and Sample
z Population
– Spring 08 credit students
(N=45,856)
z Random Sample (n=8,200)
z Sampling Procedure – student identified
email addresses
Baseline Survey Instrument
z 22
survey questions
z Rating scale questions with 11-point scale
z Multiple choice
z Open-ended questions
Sampling and Data
Collection Procedures
z Online
Survey
z Snap survey software
z Invitation with survey embedded URL
z Reminder Emails
z 26% response rate
z Export Data to SPSS
Data Analysis
z
1.
2.
3.
4.
5.
Student Satisfaction with schedule features (dependent
variable) including:
Speed of Finding Information
Ease of Use
Clarity of Use
How Informative
Overall Rating
Measures
z Independent
Samples
-Students who used only one schedule to make registration
decisions
z Paired
Samples
-Students who used more than one schedule to make
registration decisions
An example of statistical tests
performed and results
Table 6a. City/ECC:
Satisfaction with Online, Individual, and Combined Paper Schedule:
(One Schedule Used to Register)
Online
Schedule
N
Mean
Speed of Finding Information
Ease of Use
Clarity of Use
How Informative
Overall Rating
321
321
321
321
321
8.32
8.49
8.48
8.17
8.45
Individual Paper
Schedule
N
Mean
65
65
65
65
65
8.12
8.68
8.65
8.14
8.68
Combined Paper
Schedule
N
Mean
27
27
27
27
27
a. Combined schedule mean is significantly greater than the individual and online schedule at the .05 level.
Table 6b. City/ECC:
Students that Used The Online and Combined Schedule
(More Than One Schedule Used to Register)
Speed of Finding Information
Ease of Use
Clarity of Use
How Informative
Overall Rating
a. Significance at .05 level. a
9.15
8.56
8.85
8.67
8.89
N
Online
Schedule
Mean
Combined Paper
Schedule
Mean
65
66
66
65
65
8.11
8.27
8.32
7.89
8.23
8.45
8.29
8.50
8.32a
8.38
ANOVA test to detect mean
differences between
independent samples
(students who used only
one type of class
schedule)
Paired sample t-test to detect
mean differences between
dependent samples
(students who used more
than one class schedule)
Findings
z Results
revealed no obvious differences in
student satisfaction across schedules.
z Nevertheless,
there was a slight difference
in student preference for one type of class
schedule with the online schedule being the
most popular schedule modality.
Fall 2008 FollowFollow-Up Survey
Framework
z Purpose
of the Survey
z Population and Sample
–
–
Population – Fall 08 credit students at census (N=50,679)
Sample (n=9,384)
z Sampling
Procedure - Quota Sampling
Instrument Design
z Refinement
of the baseline survey
instrument
-Rasch analysis performed in WinSteps
software to verify validity and reliability of
responses
-Survey item revision
Findings
z Online
schedule preferred over combined
paper schedule in terms of use.
z No
significant difference in satisfaction
with functional features of the differing
schedules.
z Low
probability of future individual college
paper schedule use.
Discussion
z Analysis
included recognition of types of
samples (Independent vs. Repeated
Samples) and applied appropriate analytical
methods.
Discussion
* Instrument Refinement using Rasch modeling technique showed that
the 55-point scale in the followfollow-up survey performed better than the 1111-point
scale in the baseline survey.
* The 55-point scale demonstrated better student interpretation of the
meaning of the response categories with a more distinct distribution
distribution as well
as a higher endorsement probability for each response category.
Example of rating scale improvement from baseline survey to
follow-up survey
Spring 2008 Baseline Survey:
CATEGORY PROBABILITIES: MODES - Structure measures at intersections
P
-+---------+---------+---------+---------+---------+---------+R 1.0 +
+
O
|
|
B
|
0|
A
|0
00 |
B .8 + 00
00 +
I
| 0
0
|
L
|
0
0
|
I
|
0
0
|
T .6 +
0
0
+
Y
|
0
8888
0
|
.5 +
0
55
8
88 999 0
+
O
|
0
55 5
8
899 9*
|
F .4 +
011
5
5
777*
98
099
+
|
1110 11
5
5 77 8 7
9 8 0 9
|
R
|
1
0 2*2 5
6* 8 7
9
80
9
|
E
| 11
* **3*444 667 *68
77 9
*
99
|
S .2 + 1
22 031 * 33 446 7 5866
9*
08
9
+
P
|11
2 * *52 36647 85 6 9 7
0 8
99 |
O
|
22 33 **5112263 744 8 55 6*9
7700
88
99|
N
| 222 33 4455006*6***338*44 9** 666 000777
8888
|
S .0 +*************************************************************+
E
-+---------+---------+---------+---------+---------+---------+-5
-3
-1
1
3
5
7
PERSON [MINUS] ITEM MEASURE
Fall 2008 Follow-Up Survey (Pilot):
CATEGORY PROBABILITIES: MODES - Structure measures at intersections
P
-+-----+-----+-----+-----+-----+-----+-----+-----+-----+R 1.0 +
+
O
|
|
B
|
|
A
|
5|
B .8 +1
55 +
I
| 11
55 |
L
| 1
444
5
|
I
|
1
444 444
5
|
T .6 +
1
4
44
55
+
Y
|
1
4
4 5
|
.5 +
1 222222
3
44
4*
+
O
|
*2
22 333 333*
54
|
F .4 +
22 1
*
43
5 4
+
|
2
1
33 2
4 33
55
44
|
R
| 22
11 3
22 4
3
5
4
|
E
| 2
13
*
33
5
44 |
S .2 +22
331
42
33 55
44 +
P
|
33 11 44 22
5*3
4|
O
|
33
**
22
55 33
|
N
|
3333
4444 111
5****
33333
|
S .0 +*************555555555*****11111***********************+
E
-+-----+-----+-----+-----+-----+-----+-----+-----+-----+-4
-3
-2
-1
0
1
2
3
4
5
PERSON [MINUS] ITEM MEASURE
Limitations of the Research
z The
representativeness of the sample is
restricted to those who were able to respond
to the online survey. Therefore, the
generalization of the results is limited to
that particular group.
Questions?