Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
Collegiate Assessment of Academic Proficiency:
Writing Skills Test
Spring 2012 Administration
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 1 of 11
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
Table of Contents
Executive Summary....................................................................................................................................... 3
Introduction .................................................................................................................................................. 4
Sample....................................................................................................................................................... 5
Results ........................................................................................................................................................... 7
CAAP Scores Overall .................................................................................................................................. 7
CAAP Scores by Student Classification, Sex, Effort, and GPA ................................................................... 8
Conclusion ..................................................................................................................................................... 9
Appendix ..................................................................................................................................................... 10
Appendix A: Information about the CAAP Writing Skills Test................................................................. 10
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 2 of 11
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
Executive Summary
The Collegiate Assessment of Academic Proficiency (CAAP) Writing Skills Test is designed to
measure skills in punctuation, grammar, sentence structure, strategy, organization, and style.
In coordination with the various Texas Tech University (TTU) colleges, the Office of Planning and
Assessment administered the Writing Skills Test in select courses across campus at the
beginning of the spring 2012 semester. Courses were chosen based on their enrollment of
juniors and seniors to achieve a representative number from each College.
A total of 516 Texas Tech University (TTU) students participated in the spring 2012
administration of the CAAP Writing Skills Test. The answers of 400 participants were sent to
ACT for scoring. A final sample of 400 participants (136 juniors and 264 seniors) is included in
this analysis. The final sample seems to be a fairly good representation of the population of all
TTU juniors and seniors in terms of sex, and ethnicity. While most colleges are represented
somewhat proportionate to actual junior and senior enrollment, there are significantly more
students from the College of Agricultural Sciences and Natural Resources and less from the
Rawls College of Business Administration.
From the 2012 administration of the CAAP Writing Skills Test it appears that TTU juniors (mean
= 63.5, SD = 4.8) score similarly on average to the national sample of juniors (mean = 63.1, SD =
4.7) and that the TTU seniors (mean = 62.7, SD = 5.2) score similarly on average to the national
sample of seniors (mean = 63.0, SD = 5.1).
For the TTU sample, a regression model was used to find if the variables student classification,
sex, GPA, and self-reported effort are significant predictors of a participant’s CAAP Writing Skills
Test score. The results suggest that sex (t = 3.13, p = 0.0019), GPA (t = 4.73, p < .0001), and selfreported effort (t = 4.28, p < .0001) are significant predictors and that student classification is
not a significant predictor (t = -1.33, p = 0.1827). The results suggest that on average female
students, students with higher GPA’s, and students reporting that they gave more effort score
higher on the CAAP Writing Skills Test – if all the other variables are held equal.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 3 of 11
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
Introduction
The Collegiate Assessment of Academic Proficiency (CAAP) “is the standardized, nationally
normed assessment program from ACT that enables postsecondary institutions to assess,
evaluate, and enhance student learning outcomes and general education program outcomes.” 1
The CAAP offers six different modules: Reading, Writing Skills, Writing Essay, Mathematics,
Science, and Critical Thinking. 2 As per decision of the University Assessment Committee in fall
2010, the Office of Planning and Assessment administered the Writing Skills Test of the CAAP
during the spring 2012 semester. The Writing Skills Test “is a 72-item, 40-minute test
measuring students’ understanding of the conventions of standard written English in
punctuation, grammar, sentence structure, strategy, organization, and style. Spelling,
vocabulary, and rote rules of grammar are not tested.” 3 Please find more information on how
the test is constructed and scored in Appendix A.
In coordination with the various colleges, the Office of Planning and Assessment administered
the Writing Skills Test in select courses across campus. Courses were chosen based on their
enrollment by student classification and size. Juniors and seniors were the target group, so
courses with high junior and senior enrollment were selected to participate. Larger courses
were chosen from colleges with higher junior and senior enrollment and smaller courses were
chosen from colleges with lower junior and senior enrollment. Included in the sample were
courses from the following colleges: College of Agricultural Sciences and Natural Resources
(AG), College of Architecture (AR), College of Arts and Sciences (AS), College of Education (ED),
College of Human Sciences (HS), College of Mass Communications (MC), College of Visual and
Performing Arts (VP), Edward E. Whitacre Jr. College of Engineering (EN), Honors College (HR),
and Jerry S. Rawls College of Business Administration (BA). The CAAP was administered during
the regularly scheduled class time. This allowed most students from each class to participate
and helped reduce the bias that may have been created if students had been asked to
volunteer to take the survey on their own time.
A total of 516 Texas Tech University (TTU) students participated in the spring 2012
administration of the Writing Skills Test. The Office of Planning and Assessment identified one
irregularity (e.g., pattern scoring) that was exempt from scoring. As juniors and seniors were
the target group, 100 students from other student classifications were excluded. Of the
remaining 415 participants, 15 students were randomly selected from the colleges that were
overrepresented in the sample to be excluded. The remaining 400 answer sheets were sent to
ACT to be scored. The final sample includes 136 juniors and 264 seniors.
1
http://www.act.org/caap/, accessed 3/21/2012
http://www.act.org/caap/about/modules.html, accessed 3/21/2012
3
http://www.act.org/caap/test/writing.html, accessed 3/21/2012
2
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 4 of 11
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
Sample
ACT provided scores for the 400 answer sheets of students who participated in the spring 2012
administration of the CAAP. Since the students’ R numbers were used as the student ID on the
Writing Skills Test, the Office of Planning and Assessment was able to obtain demographic
information for the participants from Institutional Research and Information Management
(IRIM). The demographic information obtained includes sex, ethnicity, student classification,
and college. The following graphs compare the final samples of juniors (n = 136) and seniors (n
= 264) to the population of all TTU juniors and seniors by sex, ethnicity, and college.
Sample and Population by Sex:
Juniors
Sample and Population by Sex:
Seniors
80%
56%
54%
52%
50%
48%
46%
44%
42%
60%
Sample
Sample
40%
Population
Population
20%
0%
Female
Female
Male
Male
The samples of juniors and seniors both appear to be good representations of their respective
populations in terms of sex.
Sample and Population by Ethnicity: Juniors
80%
60%
Sample
40%
Population
20%
0%
AAM
AI
AS
B
HI
M
NH/PI
NR
U
WH
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 5 of 11
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
Sample and Population by Ethnicity: Seniors
80%
60%
Sample
40%
Population
20%
0%
AAM
AI
AS
B
HI
M
NH/PI
NR
U
WH
The samples of juniors and seniors both appear to be good representations of their respective
populations in terms of ethnicity.
Sample and Population by College: Juniors
35%
30%
25%
20%
Sample
15%
Population
10%
5%
0%
AG
AR
AS
BA
ED
EN
HR
HS
MC
UC
UN
VP
Sample and Population by College: Seniors
35%
30%
25%
20%
Sample
15%
Population
10%
5%
0%
AG
AR
AS
BA
ED
EN
HR
HS
MC
UC
UN
VP
It appears that for both the juniors and the seniors the College of Agricultural Sciences is
overrepresented and that the Rawls College of Business Administration is underrepresented in
the sample. Overall it appears that both samples are fairly representative of their respective
populations in terms of college.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 6 of 11
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
Results
CAAP Scores Overall
The following table and graph give a summary of the juniors’ and seniors’ CAAP scores.
Summary of CAAP Scores by Student Classification
n
136
264
Junior
Senior
Mean
63.5
62.7
SD
4.8
5.2
Min
50
51
Median
65
63
Max
72
72
Histogram of CAAP Scores by Student Classification
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
Juniors
Seniors
45
50
55
60
65
70
75
80
The TTU juniors (mean = 63.5, SD = 4.8) and seniors (mean = 62.7, SD = 5.2) performed similarly
on the CAAP. The ACT website has a summary of the scores from 19,792 juniors from 124
institutions and 15,040 seniors from 132 institutions that participated in the Writing Skills Test
in fall 2011.4 On average, the TTU juniors scored similarly to the national sample of juniors
(mean = 63.1, SD = 4.7). 5 The difference for the juniors is not statistically significant (t = 0.99, p
= 0.3227). On average, the TTU seniors also scored similarly to the national sample of seniors
(mean = 63.0, SD = 5.1). 6 The difference for the seniors is not statistically significant (t = -0.95, p
= 0.3436).
4
http://act.org/caap/norms/, accessed 3/21/2012
http://act.org/caap/norms/pdf/11Table12.pdf, accessed 3/21/2012
6
http://act.org/caap/norms/pdf/11Table13.pdf, accessed 3/21/2012
5
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 7 of 11
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
CAAP Scores by Student Classification, Sex, Effort, and GPA
The following table gives a summary of the overall sample by student classification, sex,
participants’ self-report of effort, and GPA. These variables will be included in a regression
model for predicting CAAP Writing Skills Test score.
Summary of Predictor Variables for CAAP Scores
Student Classification
n
Junior
136
Senior
264
Sex
n
Female
165
Male
235
Self-Reported Effort
n
Tried my best
240
Gave moderate effort
122
Gave little effort
21
Gave no effort
1
n
Mean
GPA
397
3.02
%
34.0
66.0
%
41.3
58.8
%
62.5
31.8
5.5
0.3
SD
0.53
The following table gives a summary of a regression model for predicting a participant’s CAAP
Writing Skills Test score. For the variable “Self-Reported Effort”, the “Gave little effort” group
and the “Gave no effort” group were combined in the model because of the small number of
participants in the “Gave no effort” group. For this variable, the “Gave moderate effort” group
was used as the reference group for the “Tried my best” group and the “Gave little or no effort
group”.
Linear Regression Model for Predicting CAAP Writing Skills Test Scores
Variable
Intercept
Student Classification
Sex
GPA
Gave little or no effort
Tried my best
df
1
1
1
1
1
1
Parameter Standard
Estimate
Error
55.16
1.43
-0.66
0.50
1.55
0.50
2.18
0.46
-0.15
1.06
2.19
0.51
t value
38.45
-1.33
3.13
4.73
-0.14
4.28
p value
< .0001
0.1827
0.0019
< .0001
0.8883
< .0001
95% Confidence Interval
Lower Limit Upper Limit
52.34
57.98
-1.64
0.31
0.57
2.52
1.27
3.08
-2.24
1.94
1.19
3.20
Overall, the model was found to be statistically significant (F = 14.56, p < .0001, R2 = 0.16).
Within the model, the significant predictors of CAAP score were sex (t = 3.13, p < .0001), GPA (t
= 4.73, p < .0001), and tried my best (t = 4.28, p < .0001). Specifically, this model suggests that
on average female students score higher than male students, students with higher GPA’s score
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 8 of 11
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
higher, and students reporting that they tried their best score higher than students reporting
that they gave moderate effort. Student classification was not a significant predictor of CAAP
scores (t = -1.33, p = 0.1827). The following graphs show the average CAAP scores by the
significant predictor variables from the model: sex, GPA, and self-reported effort.
Average CAAP Scores by Sex
75
70
65
CAAP Scores by GPA
75
70
64.5
62.0
65
60
60
55
55
50
50
45
45
Female
Male
1.50
2.00
2.50
3.00
3.50
4.00
Average CAAP Scores by Self-Reported Effort
75
70
65
64.3
61.8
61.1
Gave moderate
effort
Gave little or no
effort
60
55
50
45
Tried my best
Conclusion
Overall it appears that TTU juniors and seniors score similarly on average to other juniors and
seniors nationally on the CAAP Writing Skills Test. For the TTU sample it appears that on
average female students, students with higher GPA’s, and students reporting that they gave
more effort score higher on the CAAP Writing Skills Test – if all other variables are held equal.
Student classification was not a significant predictor of CAAP Writing Skills Test score.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 9 of 11
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
Appendix
Appendix A: Information about the CAAP Writing Skills Test 7
7
http://www.act.org/caap/test/writing.html, accessed 3/21/2012
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 10 of 11
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2012
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, March 2012
Page 11 of 11