The Relationship Between CCSSE and
Student Institutional Achievement
Contents
Executive Summary
The Relationship Between CCSSE and Student Institutional Achievement...................................... 1
Introduction ................................................................................................................................................... 4
Methods ........................................................................................................................................................ 4
Participants................................................................................................................................................ 4
Outcome Measures.................................................................................................................................... 4
Table 1. Descriptive Statistics of the Institutional Achievement Outcome Variables .............................. 5
Results ........................................................................................................................................................... 5
Validation of CCSSE Outcome Variables (Research Question 1)............................................................ 5
Table 2. Spearman-Brown Correlation Coefficients Between CCSSE and Institutional Outcome
variables .................................................................................................................................................... 6
Relationship Between Institutional Outcomes and CCSSE Benchmarks (Research Question 2) ............ 7
Table 3. Spearman-Brown Correlation Coefficients between Institutional Outcome Measures and
CCSSE Benchmark Raw Ratings. ............................................................................................................ 7
Relationship Between CCSSE Items and Institutional Outcome Measures (Research Question 3) ......... 8
Table 4. Spearman-Brown Correlation Coefficients Between CCSSE Items and GPA/Course Success
Rate That Are Higher Than 0.15............................................................................................................... 9
Contact Information: ................................................................................................................................... 11
1
Executive Summary
Description: The Community College Survey of Student Engagement (CCSSE) has been a source for
providing data and analysis about student engagement at the national level for community college
students. Kapi‘olani Community college has participated in the survey since 2002 but only recently was
granted permission to track the identities of survey respondents for the Spring 2012 survey in order to
correlate their CCSSE responses with their actual student achievement. The purpose of this was to
understand how the engagement benchmark measures of the CCSSE relate to institutional student success
measures, and to utilize those findings to create or modify interventions to improve future student success.
Research Questions: The research questions investigated in this study are:
1. How outcomes measures on CCSSE are correlated with institutional outcome measures?
2. Are there significant relationships between CCSSE benchmarks and the institutional outcome
measures?
3. Are there significant relationships between individual CCSSE items with the institutional
outcome measures?
Institutional Outcomes Measures: Three individual level institutional outcome measures were used in
this study: (1) cumulative GPA by the end of Spring 2012 semester (GPA); (2) the proportion of the
successfully completed courses with grades of C or better, CR or CR+ (Course Success Rate) between
summer 2011 to Spring 2012; and (3) whether a student re-enrollment in the next semester (Fall 2012),
being certificated or graduated in Spring 2012, or transferred in Fall 2012 (Persistence).
Major Findings: Based on 599 out of 978 useful respondents’ records, using Spearman-Brown
correlation method, the major findings are:
1. Students’ self-reported GPA (Item 21) was moderately highly related to institutional GPA (r
= .75). This variable can serve as an outcome variable when a researcher wants to include all the
respondents in the analysis or when students’ IDs are not available.
2. CCSSE benchmarks: Active and Collaborative Learning, Student Effort, and Academic
Challenge were found to be significantly correlated with GPAs and course success rates. The
more a student is engaged in active and collaborating learning, the more effort they put in, and the
more they feel academically challenged, the higher their GPAs and their course success rates are.
A multiple regression analysis shows that if a student’s Active and Collaborative Learning
benchmark raw score increase by 0.10 in a range between 0 and 1, their GPA would increase by
0.05. In addition if a student’s Student Effort benchmark raw score increases by .10, their GPA
would increase by 0.08.
2
3. The students who have higher GPAs and/or course success rates are those who have the following
demographic characteristics:

earned higher degrees or attended a 4-year college or university

are older and non-traditional age students

are females

rely on grants and scholarships as a source to pay tuition
4. The students who have higher GPAs and/or course success rates are those who have the following
engagement characteristics:

spending more hours preparing for class

having higher evaluation of their education experience at the college

more frequently asking questions in class or contributing to class discussions

planning or having done internship, field experience, co-op experience or clinical
assignment

feeling that their experience at this college contributed to their skills in learning
effectively on their own or acquiring job or work-related knowledge and skills to a higher
degree
These correlational results warrant further investigations in evaluating the causal effects of the
engagement variables on student success.
3
Introduction
Student engagement has been regarded as an important facet in student success at Kapi‘olani
Community College. The college has been actively participating in the Community College Survey of
Student Engagement since 2002 in an effort to understand how our students engaged in their education in
comparison with national benchmarks and how that engagement changed across years. However, it
remains unknown how the engagement benchmark measures on CCSSE are related to the institutional
student success measures at the college. Such investigation was not possible using CCSSE results prior to
2012 due to student responses being anonymous. In order to understand how a student's actual
institutional achievement is related to their student engagement and to develop or upscale interventions
that improve engagement areas that are closely related to achievement, the UH Community College
(UHCC) System petitioned to include student IDs (UH email usernames) in the spring 2012 CCSSE
administration to the UH Human Subject Research Review Committee and received approval. This study
is a preliminary investigation of the relationship between institutional success measures and the five
CCSSE benchmark areas, as well as all the items on CCSSE. In addition, the students’ actual institutional
performance can be used as a criterion to validate the potential outcome measures on CCSSE (student
self-reported GPA or intend to re-enroll). Thus this study has three research questions:
1. How outcomes measures on CCSSE are correlated with institutional outcome measures?
2. Are there significant relationships between CCSSE benchmarks and the institutional outcome
measures?
3. Are there significant relationships between individual CCSSE items with the institutional
outcome measures?
Methods
Participants
In the Spring 2012 term, 1024 CCSSE surveys were returned between Feb 27 to Apr 27. There
were 978 first-time survey respondents. Among them, 599 students provided valid IDs and were actually
enrolled at the college in Spring 2012. These 599 students were used in the data analysis.
Outcome Measures
The outcome measures used in this study were: cumulative GPAs by the end of the Spring 2012
semester, successful course completion rates, and persistence rates. Successful course completion rate is
defined as the proportion of successfully completed course in Summer 2011, Fall 2011, and Spring 2011
per respondent. Courses with grades of A, B, C, CR, or CR+ were considered successful. Courses with
grades of CE, NCE, L, RD were not included in the analysis. A student is considered persistent if they reenrolled in Fall 2012, received a certificate, graduated, or transferred either to a two year or four year
institution. Table 1 illustrates the characteristics of the outcome measures.
4
Table 1. Descriptive Statistics of the Institutional Achievement Outcome Variables
Variable
N
Mean
Standard
Median Minimum Maximum
Deviation (SD)
549
2.91
0.83
3
0
4
Course Success Rate 599
0.82
0.25
1
0
1
599
0.85
0.36
1
0
1
GPA
Persistence Rate
As one can observe, the average GPA for the respondents with valid IDs is 2.91 with at least 50%
of them having a GPA of 3 (Median = 3). Respondent GPA varied greatly from each other with the
average difference as 0.83 away from the mean (SD = 0.83).
In terms of successful course completion rates, on average, the respondents in the analysis were
able to successfully complete about 4 of 5 enrolled courses, with an average success rate of 82%. At least
50% of the students were able to complete all the enrolled courses successfully (median = 1). Again,
there is great variation among students. The average difference in course success rate is 25% away from
the mean (SD = 0.25).
Regarding persistence rate, 85% of the students re-enrolled, earned certificates, graduated, or
transferred.
Overall, the respondents showed a high degree institutional achievement. From a statistical
analysis point of view, the data did not represent normal distribution well and tended to be negatively
skewed, meaning there were more students with higher level achievement. These characteristics would
tend to suppress significance results. Therefore, non-parametric analysis methods or methods that are
robust against mild violation of normal distribution are more appropriate to use. Two methods were
selected for conducting inferential statistics: Spearman-Brown Product Moment correlation and simple
multiple regression.
Results
Validation of CCSSE Outcome Variables (Research Question 1)
Although using institutional outcome variables is more accurate, the process of matching student
institutional records and their survey responses led to an exclusion of 379 (= 978-599), or 38.8% of the
respondents from the analysis. To increase the power of the analysis in the future, the researcher also
identified the potential outcome measures in the CCSSE itself. There are two variables on CCSSE that
can potentially serve as outcome variables: Item 20: When do you plan to take classes at this college
again (Re-enrollment) and Item 21: At this college, in what range is your overall college grade average
(GPA). On the CCSSE Re-enrollment variable, the values “I will accomplish my goal(s) during this term
5
and will not be returning” and “Within the next 12 months” are coded as 1, and the values “I have no
current plan to return” and “Uncertain” were coded as 0. The coding of CCSSE GPA variable is as
follows:

Pass/fail classes only 
missing

Do not have a GPA at this school 
missing

C- or lower 
1

C
2

B- to C+ 
3

B
4

To B+ 
5

A
6
To investigate criterion-related validity using the actual institutional GPA, course success rates, and
persistence as the criteria, Spearman-Brown correlation was conducted among the CCSSE outcome
measures and institutional outcome measures. The correlation coefficients are listed in Table 2.
Correlation coefficients theoretically range from -1 to 1. The higher the absolute value of the coefficient,
the strong a relationship it indicates. Positive coefficients indicate positive relationship – when one
variable’s value goes up, the other variable’s value goes up as well. Negative coefficients indicate the
opposite – when one variable’s value goes up, the other variable’s value goes down.
Table 2. Spearman-Brown Correlation Coefficients Between CCSSE and Institutional
Outcome variables
CCSSE
Institutional
Course
Institutional
Re-
GPA
Success
Persistence
Rate
Rate
enrollment
CCSSE GPA
.09
.75
.53
.06
.14
.07
.11
.67
.14
Code
CCSSE Reenrollment
Institutional
GPA
Course
.14
Success Rate
6
Table 2 shows that all correlation coefficients are statistically significant at alpha = .05 level,
except between CCSSE re-enrollment and course success rates and between CCSSE re-enrollment and
persistence. It turns out that CCSSE GPAs has high correlation with the institutional GPAs at r = .75,
meaning that a student's self-reported GPA reflected their actual GPAs to a great degree. CCSSE GPAs
also moderately correlated with institutional course success rates, r = .53, meaning the higher a student
reported their GPAs on CCSSE, the higher their course success rates was. This is likely because that the
two institutional variables: GPAs and course success rates are moderately highly correlated already, r
= .67. Interestingly, institutional persistence measure is only weakly related to institutional GPAs and
course success rates, r = .14, and the CCSSE re-enrollment measure is significantly but weakly related to
institutional GPAs, r = .14, and with institutional persistence rate, r = .11.
These results show that the CCSSE GPA is a good proxy to measure student institutional success,
whereas the CCSSE persistence measure is yet to be proved to be a useful outcome measure.
Relationship Between Institutional Outcomes and CCSSE Benchmarks (Research Question 2)
Spearman-Brown Product Moment correlations were calculated between the Institutional
outcome variables, namely GPAs, course success rates, persistence, and the raw CCSSE benchmark
ratings, which range between 0 and 1. Using CCSSE benchmark raw scores will allow more respondents
to be included in the analysis than using standardized scores (with a mean of 50 and standard deviation of
25). Table 3 shows the results.
Table 3. Spearman-Brown Correlation Coefficients between Institutional Outcome Measures
and CCSSE Benchmark Raw Ratings.
GPA
Success Rates
Persistence
Active and Collaborative Learning
.10 *
.09 *
-.01
Student Effort
.18 *
.13 *
.04
Academic Challenge
.15 *
.10 *
-.03
Student-Faculty Interaction
.07
.05
-.02
-.05
-.04
.08
Student Support
Table 3 shows that the persistence measure has no significant relationship with any of the CCSSE
benchmarks. This is probably because of the lack of variation in this variable – 85% of the respondents
persisted. Persistence turned out to be a not very useful institutional outcome measure. GPAs and course
success rates, on the other hand, are significantly correlated with three out of five benchmarks: Active and
Collaborative Learning, Student Effort, and Academic Challenge. The more a student is engaged in
active and collaborating learning, the more effort they put in, and the more they feel academically
challenged, the higher their GPAs and their course success rates are. However, the relationship is rather
7
weak. The strongest relationship observed is between Student Effort and GPAs, r = .18, meaning if the
Student Effort raw score increases by one standard deviation unit, a students’ GPAs are expected to
increase by 0.18 standard deviation units.
To make the results easier to interpret, a multiple regression was conducted, with GPAs as the
outcome variable and the three benchmarks that had significant correlation with GPAs as predictor
variables. Stepwise selection method was used. The model only explained 4.26% of the variance with
only two variables being significant predicators – Student Effort (explains 3.45% of the variance) and
Active and Collaborative Learning (which explains an additional 0.81% of the variance). The resulting
regression equation is as follows:
GPA = 2.34 + 0.49 x Active and Collaborative Learning + 0.79 x Student Effort
[Formula 1]
The result shows if a student’s Active and Collaborative Learning benchmark raw score increase
by 0.10, their GPA would increase by 0.049 (= 0.49 * 0.10), ranging between 0.004 to 0.09 based on the
95% confidence interval, holding student effort constant. In addition if a student’s Student Effort
benchmark raw score increases by .10, their GPA would increase by 0.080, ranging between 0.032 and
0.127 based on the 95% confidence interval.
Since the correlation between GPAs and course success rates is moderately high, a separate
multiple regression would be redundant and was not conducted with courses success rates as the outcome
variable.
Relationship Between CCSSE Items and Institutional Outcome Measures (Research Question
3)
Because of the exploratory nature of this investigation, Spearman-Brown Product Moment
correlations were also calculated between the three institutional outcome measures and every CCSSE
item. The results where correlation coefficients were higher than .15 between CCSSE variables and GPAs
and Course Success Rate were presented in Table 4. All the correlation coefficients in Table 4 are
statistically significant.
8
Table 4. Spearman-Brown Correlation Coefficients Between CCSSE Items and GPA/Course
Success Rate That Are Higher Than 0.15.
CCSSE items that have meaningful correlations
Interpretation of
with GPA/Course Success Rate
Higher Values on
Success
CCSSE Variables
Rate
35. What is the highest academic credential you
GPA
Course
Higher degree
0.29
0.18
More hours
0.29
0.25
1 = Traditional Age
0.22
have earned?
10a. Preparing for class (studying, reading,
writing, rehearsing, doing homework, or other
activities related to your program)
Traditional/Nontraditional-Age students
2 = Non-Traditional Age
29. Age group
Older age
0.20
27. How would you evaluate your entire
Higher evaluation
0.20
More frequent
0.18
1= Male
0.17
0.18
educational experience at this college?
4a. Asked questions in class or contributed to class
discussions
30. Your sex
2 = Female
8a. Internship, field experience, co-op experience,
1 = I have not done nor
or clinical assignment
plan to do
0.16
2 = I plan to do
3 = I have done
18d. Indicate which of the following are sources
1 = Not a source
you use to pay your tuition at this college –
2 = Minor source
Grants & scholarships.
3 = Major source
19. Since high school, which of the following types
0 = no response
of schools have you attended other than the one
1 = response
0.16
0.15
you are now attending?
- 4-year college or university
12i. How much has your experience at this college
Higher degree
0.16
contributed to your knowledge, skills, and personal
development in the following areas?
9
- Learning effectively on your own
12b. How much has your experience at this college
Higher degree
0.15
contributed to your knowledge, skills, and personal
development in the following areas?
- Acquiring job or work-related knowledge and
skills
There are some demographic variables that are significantly and meaningfully correlated with
GPAs and/or course success rates. For example, students who earned a higher level degree (Item 35) and
attended a 4-year college or university (item 19) had better GPAs, r = .29 and .15, respectively. Older
students (Item 29) or non-traditional age students and female students tended to have higher GPAs, r
= .20, .22, and .17, respectively. In addition, students relying more on grants and scholarships as their
source to pay tuition had higher GPAs. It is possible that students who have better academic performance
receive more grant and scholarship aid. Therefore, it is no surprise that this variable is significantly
correlated with GPAs.
There are five non-demographic variables that are significantly correlated with GPAs and/or
course success rates. Although causal relationship cannot be established through correlational analysis,
confirmatory investigations of the effect of these five variables will lead to implication for interventions
that can improve student success. The interpretations of the relationship of these five variables and
GPAs/course success rates are as follows:
-
Students who spent more hours preparing for class (Item 10a.) had higher GPAs, r = .29, and
course success rates, r = .25.
-
Students who had higher evaluation of their educational experience at the college (Item 27) had
higher GPAs, r = .20, and course success rates, r = .18.
-
Students who asked questions in class or contribute to class discussions more often (Item 4a.) had
better GPAs, r = .18.
-
Students who planned to or had done internship, field experience, co-op experience or clinical
assignment (Item 8a.) had better GPAs, r = .16.
-
The higher degree that students felt that their experience at this college contributed to their skills
in learning effectively on their own (Item 12i) or acquiring job or work-related knowledge and
skills (Item 12b.), the higher their course success rates were, r = .16 and .15, respectively.
The interpretation can go the other way as well. For example, students with higher GPAs spent more
hours preparing for class (Item 10a.), or students with higher GPAs landed themselves more opportunities
for doing internships, field experience, co-op experience or clinical assignment (Item 8a.).
10
However, it would be conceivable that if there is an intervention that promotes more time to be
spent on class preparation or allows for more contribution to class discussion, a student will be more
engaged in their education, and hence more successful in attaining higher GPAs and successfully
completing courses. Further evaluation studies can be designed to evaluate the causal effects of the
variables that are significantly correlated with GPAs.
Contact Information:
Yao Hill, Ph.D.
Institutional Researcher
Office for Institutional Effectiveness
Kapi'olani Community College
Phone: (808) 734-9763
ofie@hawaii.edu
Jeffrey Yamashiro
Institutional Analyst
Office for Institutional Effectiveness
Kapi‘olani Community College
Phone: (808) 734-9778
Fax: (808) 734-9443
jeffreyy@hawaii.edu
11