Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Collegiate Assessment of Academic Proficiency:
Writing Skills Test
Spring 2011 Administration
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 1 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Table of Contents
Executive Summary....................................................................................................................................... 5
Introduction .................................................................................................................................................. 7
Sample....................................................................................................................................................... 7
Results ......................................................................................................................................................... 10
CAAP Scores Overall ................................................................................................................................ 10
CAAP Scores by Sex, GPA, and Effort ...................................................................................................... 14
Additional Questions............................................................................................................................... 16
Conclusion ................................................................................................................................................... 21
Appendix ..................................................................................................................................................... 22
Appendix A: Information about the CAAP Writing Skills Test ................................................................ 22
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 2 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
................................................................................................................................................................ 22
Appendix B: Copy of the Additional Questions....................................................................................... 24
Appendix C: Analysis of CAAP Scores ...................................................................................................... 27
Appendix D: Additional Questions Analysis ............................................................................................ 30
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 3 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 4 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Executive Summary
The Collegiate Assessment of Academic Proficiency (CAAP) Writing Skills Test is designed to
measure skills in punctuation, grammar, sentence structure, strategy, organization, and style.
In coordination with the various Texas Tech University (TTU) colleges, the Office of Planning and
Assessment administered the Writing Skills Test in select courses across campus during the
spring 2011 semester. Courses were chosen to achieve a representative number from each
college based on enrollment of juniors and seniors. ACT allows the administering institution to
add up to nine additional questions to the chosen CAAP module. TTU chose to add nine
additional questions regarding participants’ college writing experience.
A total of 435 TTU students participated in the spring 2011 administration of the CAAP Writing
Skills Test. The goal was to score tests from 200 juniors and 200 seniors. 43 answer sheets
were excluded due to participants’ answer sheet irregularities or student classification. A final
sample of 129 juniors (including sophomores with 50 or more credit hours) and 263 seniors is
included in this analysis. The final sample seems to be a fairly good representation of the
population of all TTU juniors and seniors in terms of sex, ethnicity, and college.
In comparing scores from the CAAP Writing Skills Test, it appears that TTU juniors (mean = 62.9)
and seniors (mean = 62.7) score slightly lower than the national samples of juniors (mean =
63.1) and seniors (mean = 63.2). However, these differences between the TTU samples and the
national samples are not statistically significant at the 0.05 level. This suggests that TTU juniors
and seniors score similarly to other juniors and seniors nationally on the CAAP Writing Test.
Considering the TTU CAAP scores by sex, GPA, and self-report of effort shows that female
participants, participants with higher GPA’s, and participants reporting more effort score higher
on average than other participants on the CAAP Writing Test.
Responses to the additional questions suggest that most TTU students complete their first-year
composition credit at TTU or through transfer credit from another university. Where
participants completed their first-year composition credit does not have a statistically
significant impact on CAAP scores. A majority of the juniors and seniors report that they have
completed their required six hours of writing intensive major courses. Participants’ completion
of the six hours of writing intensive major courses does not have a statistically significant
impact on CAAP scores. On average, participants report that they are required to write 0-10
pages in most of their classes. Participants’ average number of pages of required writing in
classes does not have a statistically significant impact on CAAP scores. A large majority of the
participants report that they have never used the University Writing Center in the past two
semesters. Participants’ use of the University Writing Center does not have a statistically
significant impact on CAAP scores. The most commonly valued aspect of writing instruction
reported by participants is the opportunity to express ideas and the least commonly valued
aspect of writing instruction reported by participants is the opportunity to collaborate with
peers. A large majority of participants report that they are most confident when writing in
English. For the juniors, participants reporting a language other than English scored
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 5 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
significantly lower on average. All of the results related to the additional questions need to be
interpreted with caution due to the small sample sizes involved for some of the question
responses.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 6 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Introduction
The Collegiate Assessment of Academic Proficiency (CAAP) “is the standardized, nationally
normed assessment program from ACT that enables postsecondary institutions to assess,
evaluate, and enhance student learning outcomes and general education program outcomes.” 1
The CAAP offers six different modules: Reading, Writing Skills, Writing Essay, Mathematics,
Science, and Critical Thinking. 2 As per decision of the University Assessment Committee in fall
2010, the Office of Planning and Assessment administered the Writing Skills Test during spring
2011. The Writing Skills Test “is a 72-item, 40-minute test measuring students’ understanding
of the conventions of standard written English in punctuation, grammar, sentence structure,
strategy, organization, and style. Spelling, vocabulary, and rote rules of grammar are not
tested.” 3 See Appendix A for more information on how the test is constructed and scored.
ACT allows the administering institution to add up to nine additional questions to the chosen
module. Nine questions were added as additional questions to ask participants about their
writing experiences in college (see Appendix B for a copy of the additional questions).
In coordination with the various colleges, the Office of Planning and Assessment administered
the Writing Skills Test in select courses across campus. Courses were chosen to achieve a
representative number from each college based on enrollment of juniors and seniors. Juniors
and seniors were the target group, so courses with high junior and senior enrollment were
selected to participate. Larger courses were chosen from colleges with higher junior and senior
enrollment and smaller courses were chosen from colleges with lower junior and senior
enrollment. Included in the sample were courses from the following colleges: College of
Agricultural Sciences and Natural Resources (AG), College of Architecture (AR), College of Arts
and Sciences (AS), College of Education (ED), College of Human Sciences (HS), College of Mass
Communications (MC), College of Visual and Performing Arts (VP), Edward E. Whitacre Jr.
College of Engineering (EN), and Jerry S. Rawls College of Business Administration (BA). The
CAAP Writing Skills Test and the additional questions were administered during the regularly
scheduled class time for the participating classes. This allowed most students from each class
to participate and helped reduce the bias that may have been created if students had been
asked to volunteer to take the survey on their own time instead.
Sample
A total of 435 Texas Tech University (TTU) students participated in the spring 2011
administration of the CAAP Writing Skills Test. The goal was to score tests from 200 juniors and
200 seniors. The Office of Planning and Assessment identified three irregularities (e.g., pattern
scoring) that were exempt from scoring. Also exempt from scoring were the 39 participants
with less than 50 credit hours (the participants with 50 - 60 credit hours were selected to be
included with the sample of juniors). The remaining 393 answer sheets were sent to ACT to be
1
http://www.act.org/caap/, accessed 6/2/2011
http://www.act.org/caap/test_modules.html, accessed 6/2/2011
3
http://www.act.org/caap/test_writing.html, accessed 6/2/2011
2
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 7 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
scored. One of the returned answer sheets did not match with any participants from the TTU
sample, leaving a final sample size of 392 participants (129 juniors and 263 seniors).
Since the students’ R numbers were used as the student ID on the Writing Skills Test, the Office
of Planning and Assessment was able to obtain demographic information for the participants
from Institutional Research (IR). The demographic information obtained includes sex, ethnicity,
student classification, and college. While participants were asked similar questions about their
background on the CAAP Writing Skills Test, it was decided to use official registration data
provided by IR instead. Using the IR data facilitates comparison to the population numbers and
has a greater chance of being accurate.
The following graphs compare the final junior and senior samples to the population of all TTU
juniors and seniors, respectively, by sex, ethnicity, and college.
Junior Sample and Population by Sex
Senior Sample and Population by Sex
60%
70%
50%
60%
50%
40%
40%
30%
Sample
20%
Population
10%
Sample
30%
Population
20%
10%
0%
0%
Female
Male
Female
Male
The junior and senior samples both appear to be fairly good representations of their respective
populations in terms of sex.
Junior Sample and Population by Ethnicity
80.0%
60.0%
Sample
40.0%
Population
20.0%
0.0%
AAM
AI
AS
B
HI
M
NR
U
WH
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 8 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Senior Sample and Population by Ethnicity
80.0%
60.0%
Sample
40.0%
Population
20.0%
0.0%
AAM
AI
AS
B
HI
M
NR
U
WH
The junior and senior samples both appear to be fairly good representations of their respective
populations in terms of ethnicity.
Junior Sample and Population by College
50%
40%
30%
Sample
20%
Population
10%
0%
AG
AR
AS
BA
ED
EN
HS
MC
UC
VP
Senior Sample and Population by College
50%
40%
30%
Sample
20%
Population
10%
0%
AG
AR
AS
BA
ED
EN
HS
MC
UC
VP
The junior sample appears to include more participants from the College of Arts and Sciences
and fewer participants from the College of Business Administration and the College of
Engineering than would be expected from the population. The senior sample appears to
include more participants from the College of Engineering and the College of Human Sciences
and fewer participants from the College of Arts and Sciences than would be expected from the
population.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 9 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Results
CAAP Scores Overall
Three scores are reported for the CAAP Writing Skills Test: a total test score (72 items), a
usage/mechanics subscore (32 punctuation, grammar, and sentence structure items), and a
rhetorical skills subscore (40 organization, strategy, and style items). 4 Potential total test scores
range from 40 (low) to 80 (high) and potential subscores range from 5 (low) to 25 (high). The
following table gives a summary of these three scores for the TTU junior and senior samples.
Summary of
CAAP Scores
n
Mean
Standard Deviation
Minimum
Median
Maximum
Total Test
Juniors
Seniors
129
263
62.9
62.7
4.9
4.6
53
50
63
63
73
72
Usage/Mechanics
Juniors
Seniors
129
263
16.7
16.4
2.4
2.2
11
11
17
17
21
21
Rhetorical Skills
Juniors
Seniors
129
263
16.3
16.3
2.4
2.4
11
10
16
16
21
21
It appears that on average the TTU juniors scored slightly better than the TTU seniors in total
test scores (62.9 vs. 62.7) and usage/mechanics scores (16.7 vs. 16.4) and that the TTU juniors
scored similarly to the TTU seniors in rhetorical skills scores (16.3 vs. 16.3). The differences in
total test scores, usage/mechanics scores, and rhetorical skills scores by student classification
are not statistically significant at the 0.05 level (see Appendix C for details). This suggests that
on average TTU juniors and seniors score similarly on the Writing Skills Test.
4
http://act.org/caap/test_writing.html, accessed 6/2/2011
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 10 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
The following graphs show the distributions of total test scores, usage/mechanics scores, and
rhetorical skills scores for the TTU junior and senior samples.
Writing Skills Test: Total Test Scores
40%
33%
Percent of Sample
35%
35%
28% 28%
30%
27% 27%
25%
20%
Juniors
15%
10%
Seniors
7% 6%
5%
5%
0.4%
0%
45
50
55
60
65
4%
70
75
80
Total Test Score
The total test score distributions are similar for the TTU juniors and seniors.
Writing Skills Test: Usage/Mechanics Score
45%
Percent of Sample
40%
36%
35%
38%
40%
35%
30%
25%
18%
20%
15%
12% 11%
Juniors
11%
Seniors
10%
5%
0%
7
10
13
16
19
22
25
Usage/Mechanics Score
The usage/mechanics score distributions are similar for the TTU juniors and seniors with a few
more seniors scoring between 16 and 19 and a few more juniors scoring between 19 and 22.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 11 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Percent of Sample
Writing Skills Test: Rhetorical Skills Score
50%
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
40%
14%
43%
36%
38%
Juniors
11%
10%
Seniors
8%
1%
7
10
13
16
19
22
25
Rhetorical Skills Score
The rhetorical skills score distributions are similar for the TTU juniors and seniors.
The ACT website has a summary for the scores of 18,902 juniors from 130 institutions and
14,405 seniors from 127 institutions who participated in the Writing Skills Test in the fall of
2010.5 On average the TTU juniors scored similarly to the national sample of juniors in total
test scores (62.9 vs. 63.1), usage/mechanics scores (16.7 vs. 16.6), and rhetorical skills scores
(16.3 vs. 16.6). 6 On average the TTU seniors scored similarly to the national sample of seniors
in total test scores (62.7 vs. 63.2), usage/mechanics scores (16.4 vs. 16.7), and rhetorical skills
scores (16.3 vs. 16.6). 7 The comparison of seniors’ usage/mechanics scores is statistically
significant at the 0.05 level (see Appendix C for details). This suggests that on average TTU
seniors score lower than other seniors nationally on the usage/mechanics questions of the
CAAP Writing Skills Test. The other comparisons are not statistically significant at the 0.05 level.
This suggests that on average TTU juniors and seniors score similarly to other juniors and
seniors nationally for all the other scores from the CAAP Writing Skills Test.
5
http://act.org/caap/norms/, accessed 6/2/2011
http://act.org/caap/norms/pdf/10Table12.pdf, accessed 6/2/2011
7
http://act.org/caap/norms/pdf/10Table13.pdf, accessed 6/2/2011
6
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 12 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
The following table shows the number of TTU juniors and seniors who received each total test
score and the national percentile for each of these scores. 8 For each score, the national
percentile shows the percentage of junior or senior participants, respectively, from the national
sample that scored at or below that score. The highlighted cells show those TTU participants
who scored higher than the 50th national percentile. Just over half of the TTU juniors (66
participants, 51.1%) and just under half of the TTU seniors (117 participants, 44.5%) scored
above the 50th percentile nationally.
National Percentiles for CAAP Scores
Score TTU Juniors Natl. % TTU Seniors Natl. %
73
72
1
2
99
99
0
4
99
99
71
70
69
68
67
66
65
64
63
62
61
60
59
58
57
56
55
54
53
52
51
50
4
8
5
6
6
10
5
11
8
14
4
9
12
8
5
2
2
2
5
0
0
0
99
96
91
86
79
73
66
58
52
43
37
30
24
19
14
10
7
4
3
1
1
0
6
9
12
6
26
18
19
17
17
18
21
18
23
12
12
9
9
2
2
1
1
1
98
94
89
83
76
70
63
56
49
42
37
30
24
20
15
11
8
5
3
2
1
0
8
http://act.org/caap/norms/pdf/10Table12.pdf, http://act.org/caap/norms/pdf/10Table13.pdf, accessed
6/2/2011
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 13 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
CAAP Scores by Sex, GPA, and Effort
The following graphs show the average CAAP total test scores by sex, GPA, and participants’
self-report of effort (See Appendix C for sample sizes).
Average CAAP Scores by Sex
66.0
65.0
65.0
63.7
64.0
63.0
Juniors
62.0
62.0
61.2
Seniors
61.0
60.0
59.0
Female
Male
For both the juniors and the seniors the female participants scored higher on average than the
male participants. These differences by sex are statistically significant at the 0.05 level (see
Appendix C for details). This suggests that on average female TTU juniors and seniors score
higher than male TTU juniors and seniors on the CAAP Writing Skills Test.
Average CAAP Scores by GPA
65.3 65.0
66
64
63.1
63.3
62.8
61.7
62
61.7
62.2
Juniors
59.4
60
Seniors
58
56
below 2.00
2.01 - 2.50
2.51 - 3.00
3.01 - 3.50
3.51 and above
Note that there is no average score for the seniors in the “below 2.00” group because there
were no senior participants with a GPA lower than 2.00. It appears that on average the juniors
and seniors with higher GPA’s scored higher than the juniors and seniors with lower GPA’s.
These differences in scores by GPA are statistically significant at the 0.05 level (see Appendix C
for details). The results suggest that on average TTU juniors and seniors with higher GPA’s
score higher on the CAAP Writing Skills Test.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 14 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Average CAAP Scores by Effort
65
64
63
62
61
60
59
58
57
56
55
63.7 63.5
62.7 62.7
60.8
58.3
tried my best
gave moderate effort
Juniors
Seniors
gave little effort
Note that the participants that selected “gave no effort” are excluded from this comparison
because there was only one junior and two seniors in this group. For both the juniors and
seniors it appears that on average participants who reported more effort on the assessment
scored higher. These differences between the groups are statistically significant at the 0.05
level for the seniors, but not for the juniors (see Appendix C for details). The lack of significance
for the juniors may be due to the small sample size of juniors selecting “gave little effort”. The
results suggest that on average TTU seniors who put forth more effort on the CAAP Writing
Skills Test score higher.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 15 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Additional Questions
The following graphs give a summary of the junior and senior responses to the nine additional
questions that were administered with the CAAP Writing Test (see Appendix B for a copy of the
additional questions).
I received credit for ENGL 1301 (the first semester of first-year composition) by:
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
42% 41%
32%
26%
18% 17%
Juniors
11%
2% 2%
TTU
transfer
credit
dual credit SAT or ACT
Seniors
5%
0% 0%
AP
IB
0%
2%
CLEP
I received credit for ENGL 1302 (the second semester of first-year composition)
by:
50%
40%
34%
38%
42% 41%
30%
Juniors
16% 16%
20%
10%
2% 1%
0%
TTU
transfer
credit
dual credit SAT or ACT
6%
3%
AP
Seniors
0% 0%
1% 1%
IB
CLEP
It appears that most of the participants completed their first-year composition courses through
a course at TTU or through transfer credit from another institution. The responses to these
questions were analyzed to assess if where participants completed their first-year composition
credit has an impact on CAAP scores. The participants that completed at least one of their firstyear composition semesters at TTU were compared to the participants that completed both of
their first-year composition semesters through any of the other options. Testing at the 0.05
level, the analysis found that completing first-year composition credit at TTU as opposed to any
of the other options did not have a statistically significant impact on CAAP scores for the juniors
or the seniors (see Appendix D for details). This suggests that on average TTU juniors and
seniors that complete their first-year composition credit at TTU do not score differently on the
CAAP Writing Test than other TTU juniors and seniors.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 16 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
I have completed the required six credit hours of writing
intensive courses in my major:
100%
80%
89%
69%
60%
Juniors
40%
Seniors
20%
20%
11%
8%
4%
0%
Yes
No
I don't know
Most of the junior and senior participants report that they have completed their six credit
hours of writing intensive major courses. The responses to this question were analyzed to
assess if completion of writing intensive courses has an impact on CAAP scores. The
participants reporting that they do not know if they have completed the requirement were
excluded from this comparison. Testing at the 0.05 level, the analysis found that completing
the required six credit hours of writing intensive courses does not have a statistically significant
impact on CAAP scores for the juniors or the seniors (see Appendix D for details). This suggests
that on average TTU juniors and seniors that have completed their writing intensive major
courses do not score differently on the CAAP Writing Test than other TTU juniors and seniors.
Given that the sample includes juniors and seniors, it seems likely that many of the participants
in the not completed group have completed at least part of the six hour requirement and do
have some writing intensive course experience. This may help explain the lack of significant
findings.
In the typical course in my major, I am required to write the following number of
pages:
60%
50%
55%
47%
40%
31% 28%
30%
Juniors
20%
7%
10%
Seniors
10%
2%
7%
3%
4%
2%
5%
0%
0-5
6-10
11-15
16-20
21-25
26 or more
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 17 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
In the writing intensive courses in my major, I am required to write the
following number of pages:
40%
35%
30%
25%
20%
15%
10%
5%
0%
35%
28%
26%
21%
20%
18% 16%
Juniors
11%
4%
0-5
6-10
11-15
16-20
7%
21-25
6%
8%
Seniors
26 or more
In most courses outside my major, I am required to write the following number
of pages:
80%
70%
60%
50%
40%
30%
20%
10%
0%
68%
58%
21%
Juniors
27%
Seniors
7%
0-5
6-10
9%
11-15
4%
4%
16-20
0%
1%
21-25
0%
1%
26 or more
It appears that a majority of the juniors and seniors report that they have been required to
write 0-10 pages in their typical major courses and in most of their courses outside their major.
A larger percentage of the juniors and seniors reports that they been required to write more
than 10 pages in their writing intensive major courses, although a large percentage of the
juniors and seniors report writing 0-10 pages in these courses as well. The responses to these
three questions were averaged for each participant to find if a participant’s average number of
pages written in courses has an impact on their CAAP score. Testing at the 0.05 level, the
analysis found that participants’ average number of pages written in courses does not have a
statistically significant impact on CAAP scores for the juniors or the seniors (see Appendix D for
details). This suggests that on average TTU juniors and seniors that write more in their courses
do not score differently on the CAAP Writing Test than other TTU juniors and seniors. The lack
of significant difference might be partially explained by the small proportion of participants
reporting more than 10 pages of writing in their average course.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 18 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
In the past two semesters, I have utilized the services of the University Writing
Center (onsite or online) for my course assignments:
100%
80%
80%
76%
60%
Juniors
40%
18%
20%
Seniors
20%
2%
0%
never
1-3 times
1%
4-6 times
0%
1%
7-9 times
0%
1%
10 or more times
It appears that a large majority of the participants have never used the services of the
University Writing Center in the past two semesters. Those participants reporting any use of
the University Writing Center in the past two semesters were compared to those participants
reporting no use of the University Writing to assess for an impact on CAAP scores. Testing at
the 0.05 level, the analysis found that use of the University Writing Center in the past two
semesters does not have a statistically significant impact on CAAP scores for the juniors or the
seniors (see Appendix D for details). This suggests that on average TTU juniors and seniors that
use the University Writing Center do not score differently on the CAAP Writing Test than other
TTU juniors and seniors.
What I value most about writing instruction is the opportunity to:
35%
30%
25%
20%
15%
10%
5%
0%
30%
29% 31%
26%
18% 16%
10% 11%
5%
engage
subject
matter
express my
ideas
improve
critical
thinking
Juniors
6% 7%
7%
Seniors
2% 2%
collaborate
with peers
instructor
feedback
all of the
above
none of the
above
It appears that the most common response for the junior and senior participants is that they
value writing instruction because it is an opportunity to express ideas or because of all of the
listed reasons. The opportunity to collaborate with peers is the least common response for
what the juniors and senior participants value about writing instruction.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 19 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
I am most confident when I am writing in the following language:
95% 93%
100%
80%
60%
Juniors
40%
Seniors
20%
0%
1% 0%
0% 1%
Arabic
Chinese
English
1% 2%
0% 1%
0% 1%
0% 0.4%
1% 0.4%
French
German
Hindi
Russian Spanish
2% 1%
Other
A large majority of the junior and senior participants report that they are most confident when
writing in English. This group was compared to the participants reporting any other language as
their most confident writing language to assess for an impact on CAAP scores. Testing at the
0.05 level, the analysis found that having a most confident writing language other than English
does have a statistically significant impact on CAAP scores for the juniors, but not for the
seniors (see Appendix D for details). This suggests that on average TTU juniors that are most
confident writing in English score higher on the CAAP Writing Test than TTU juniors that are
most confident writing in another language. This result, though, needs to be taken with caution
due to the small sample size of juniors reporting a language other than English.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 20 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Conclusion
Overall it appears that on average TTU juniors and seniors score similarly to other juniors and
seniors nationally on the Writing Skills Test from the CAAP. At TTU, it appears that on average
the juniors score similarly to the seniors. This pattern is consistent with the national samples of
juniors and seniors. The results also suggest that at TTU female juniors and seniors score higher
on average than male juniors and seniors. Juniors and seniors with higher GPA’s score higher
on average than TTU juniors and seniors with lower GPA’s. TTU seniors who report that they
put forth more effort on the assessment also score higher on average.
Responses to the additional questions suggest that most TTU students complete their first-year
composition credit at TTU or through transfer credit from another university. Where
participants completed their first-year composition credit does not have a statistically
significant impact on CAAP scores. A majority of the juniors and seniors report that they have
completed their required six hours of writing intensive major courses. Participants’ completion
of the six hours of writing intensive major courses does not have a statistically significant
impact on CAAP scores. On average, participants report that they are required to write 0-10
pages in most of their classes. Participants’ average number of pages of required writing in
classes does not have a statistically significant impact on CAAP scores. A large majority of the
participants report that they have never used the University Writing Center in the past two
semesters. Participants’ use of the University Writing Center does not have a statistically
significant impact on CAAP scores. The most commonly valued aspect of writing instruction
reported by participants is the opportunity to express ideas and the least commonly valued
aspect of writing instruction reported by participants is the opportunity to collaborate with
peers. A large majority of participants report that they are most confident when writing in
English. For the juniors, participants reporting a language other than English scored
significantly lower on average. All of the results related to the additional questions need to be
interpreted with caution due to the small sample sizes involved for some of the question
responses.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 21 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Appendix
Appendix A: Information about the CAAP Writing Skills Test 9
9
Screenshots from http://www.act.org/caap/test_writing.html (accessed on 6/2/2011)
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 22 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 23 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Appendix B: Copy of the Additional Questions
Student ID: R______________________________
INSTRUCTIONS: Please turn to page 3 in your answer document and find block P, Additional
Questions. Each question below corresponds to a column in block P. Read the question and
then mark only one circle in each column that best describes your response. For example, if for
question A you think answer option 7 is correct, you would bubble in the 7 in column A, etc.
QUESTIONS
Column A
I received credit for ENGL 1301 (the first semester of first-year composition) by
0. taking the course at Texas Tech University
1. taking the course at another institution and transferring it in
2. dual credit through my high school and a neighboring college
3. high scores on the SAT or ACT
4. AP (Advanced Placement) examination
5. IB (International Baccalaureate)
6. CLEP (College Level Examination Program)
Column B
I received credit for ENGL 1302 (the second semester of first-year composition) by
0. taking the course at Texas Tech University
1. taking the course at another institution and transferring it in
2. dual credit through my high school and a neighboring college
3. high scores on the SAT or ACT
4. AP (Advanced Placement) examination
5. IB (International Baccalaureate)
6. CLEP (College Level Examination Program)
Column C
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 24 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
I have completed the required six credit hours of writing Intensive courses in my major:
0. Yes
1. No
2. I don’t know
Column D
In the typical course in my major, I am required to write the following number of pages:
0. 0-5
1. 6-10
2. 11-15
3. 16-20
4. 21-25
5. 26 or more
Column E
In the Writing Intensive courses in my major, I am required to write the following number of
pages:
0. 0-5
1. 6-10
2. 11-15
3. 16-20
4. 21-25
5. 26 or more
Column F
In most courses outside my major, I am required to write the following number of pages:
0. 0-5
1. 6-10
2. 11-15
3. 16-20
4. 21-25
5. 26 or more
Column G
In the past two semesters, I have utilized the services of the University Writing Center (onsite
or online) for my course assignments
0. never
1. 1-3 times
2. 4-6 times
3. 7-9 times
4. 10 or more times
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 25 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Column H
What I value most about writing instruction is the opportunity to
0. engage fully with the subject matter I am writing about
1. express my ideas
2. improve my critical thinking skills
3. collaborate with my peers
4. receive detailed feedback from my instructor
5. all of the above
6. none of the above
Column I
I am most confident when I am writing in the following language:
0. Arabic
1. Chinese
2. English
3. French
4. German
5. Hindi
6. Russian
7. Spanish
8. Other
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 26 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Appendix C: Analysis of CAAP Scores
CAAP Scores by Student
Classification
Juniors
Seniors
n
129
263
Mean
62.9
62.7
St. Dev.
4.86
4.6
t value
0.43
p value
0.6705
Usage/Mechanics Scores
by Student Classification
Juniors
Seniors
n
129
263
Mean
16.7
16.4
St. Dev.
2.44
2.24
t value
0.91
p value
0.3640
Rhetorical Skills Scores
by Student Classification
Juniors
Seniors
n
129
263
Mean
16.3
16.3
St. Dev.
2.40
2.38
t value
-0.05
p value
0.9592
CAAP Scores: TTU vs.
National Sample (Juniors)
TTU
National Sample
n
129
18,902
Mean
62.9
63.1
St. Dev.
4.9
4.8
t value
-0.4716
p value
0.6372
CAAP Scores: TTU vs.
National Sample (Seniors)
TTU
National Sample
n
263
14,405
Mean
62.7
63.2
St. Dev.
4.6
5.1
t value
-1.5782
p value
0.1145
Usage/Mechanics Scores: TTU
vs. National Sample (Juniors)
TTU
National Sample
n
129
18,902
Mean
16.7
16.6
St. Dev.
2.4
2.3
t value
0.4920
p value
0.6227
Usage/Mechanics Scores: TTU
vs. National Sample (Seniors)
TTU
National Sample
n
263
14,405
Mean
16.4
16.7
St. Dev.
2.2
2.4
t value
-2.0118
p value
0.0443
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 27 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Rhetorical Skills Scores: TTU
vs. National Sample (Juniors)
TTU
National Sample
n
129
18,902
Mean
16.3
16.6
St. Dev.
2.4
2.5
t value
-1.3587
p value
0.1743
Rhetorical Skills Scores: TTU
vs. National Sample (Seniors)
TTU
National Sample
n
263
14,405
Mean
16.3
16.6
St. Dev.
2.4
2.6
t value
-1.8568
p value
0.0634
CAAP Scores by Sex
(Juniors)
Female
Male
n
57
71
Mean
65.0
61.2
St. Dev.
4.16
4.75
t value
4.75
p value
< 0.0001
CAAP Scores by Sex
(Seniors)
Female
Male
n
107
156
Mean
63.7
62.0
St. Dev.
4.48
4.57
t value
2.95
p value
0.0035
CAAP Scores by GPA
(Juniors)
A) below 2.00
B) 2.01 - 2.50
C) 2.51 - 3.00
D) 3.01 - 3.50
E) 3.51 and above
CAAP Scores by GPA
(Seniors)
A) below 2.00
B) 2.01 - 2.50
C) 2.51 - 3.00
D) 3.01 - 3.50
E) 3.51 and above
n
9
20
33
34
30
n
0
19
86
94
64
Mean
63.1
59.4
62.8
63.3
65.3
Mean
.
61.7
61.7
62.2
65.0
St. Dev.
5.09
3.98
4.59
4.27
4.98
St. Dev.
.
5.15
4.28
4.62
4.13
F value
5.14
F value
7.72
p value
0.0007
p value
< 0.0001
significant
comparisons
E>B
D>B
significant
comparisons
E>D
E>C
E>B
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 28 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
CAAP Scores by Effort
(Juniors)
A) tried my best
B) gave moderate effort
C) gave little effort
D) gave no effort*
n
65
50
10
1
Mean
63.7
62.7
60.8
59
St. Dev.
5.27
4.05
4.80
.
F value
1.90
p value
0.1543
CAAP Scores by Effort
(Seniors)
A) tried my best
B) gave moderate effort
C) gave little effort
D) gave no effort*
n
119
117
16
2
Mean
63.5
62.7
58.3
55
St. Dev.
4.53
4.21
4.76
1.41
F value
9.85
p value
< 0.0001
significant
comparisons
significant
comparisons
A>C
B>C
*excluded from analysis because of small sample size
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 29 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
Appendix D: Additional Questions Analysis
CAAP Scores by completion of first
year composition credit (Juniors)
TTU
Other
n
51
74
Mean
62.4
63.3
St. Dev.
4.24
5.33
t value
-0.97
p value
0.3320
CAAP Scores by completion of first
year composition credit (Seniors)
TTU
Other
n
103
143
Mean
62.9
62.9
St. Dev.
4.46
4.58
t value
0.05
p value
0.9587
CAAP Scores by completion of
writing intensive courses (Juniors)
Yes
No
n
86
25
Mean
63.1
62.2
St. Dev.
4.89
5.33
t value
0.76
p value
0.4476
CAAP Scores by completion of
writing intensive courses (Seniors)
Yes
No
n
218
19
Mean
62.9
61.3
St. Dev.
4.42
4.56
t value
1.48
p value
0.1409
CAAP Scores by average pages of
writing in writing classes (Juniors)
0–5
6 – 10
11 – 15
16 or more
n
72
37
10
6
Mean
62.8
62.7
63.5
65.0
St. Dev.
4.63
5.66
4.67
4.34
t value
0.44
p value
0.7274
CAAP Scores by average pages of
writing in writing classes (Seniors)
0–5
6 – 10
11 – 15
16 or more
n
110
75
37
24
Mean
62.8
63.5
62.9
61.0
St. Dev.
4.36
4.27
4.83
5.44
t value
1.8
p value
0.1487
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 30 of 31
Collegiate Assessment of Academic Proficiency: Writing Skills Test
Spring 2011
CAAP Scores by use of the
University Writing Center (Juniors)
Yes
No
n
25
100
Mean
61.8
63.2
St. Dev.
4.86
4.92
t value
1.29
p value
0.2011
CAAP Scores by use of the
University Writing Center (Seniors)
Yes
No
n
58
186
Mean
62.0
63.2
St. Dev.
4.50
4.87
t value
1.71
p value
0.0888
CAAP Scores by most confident
writing language (Juniors)
English
Other
n
118
6
Mean
63.2
57.8
St. Dev.
4.92
1.17
t value
8.12
p value
< 0.0001
CAAP Scores by most confident
writing language (Seniors)
English
Other
n
212
16
Mean
63.3
61.1
St. Dev.
4.35
6.33
t value
1.37
p value
0.1886
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2011
Page 31 of 31