Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
Collegiate Assessment of Academic Proficiency:
Science Test
Fall 2011 Administration
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 1 of 11
Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
Table of Contents
Executive Summary....................................................................................................................................... 3
Introduction .................................................................................................................................................. 4
Sample....................................................................................................................................................... 5
Results ........................................................................................................................................................... 7
CAAP Scores Overall .................................................................................................................................. 7
CAAP Scores by Student Classification, Sex, Effort, and GPA ................................................................... 8
Conclusion ..................................................................................................................................................... 9
Appendix ..................................................................................................................................................... 10
Appendix A: Information about the CAAP Science Test ......................................................................... 10
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 2 of 11
Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
Executive Summary
The Collegiate Assessment of Academic Proficiency (CAAP) Science Test is designed to measure
skills in scientific reasoning. In coordination with the various Texas Tech University (TTU)
Colleges, the Office of Planning and Assessment administered the Science Test in select courses
across campus at the beginning of the fall 2011 semester. Courses were chosen based on their
enrollment of juniors and seniors to achieve a representative number from each college.
A total of 422 Texas Tech University (TTU) students participated in the fall 2011 administration
of the CAAP Science Test. The answers of 365 participants were sent to ACT for scoring. A final
sample of 365 participants (169 juniors and 196 seniors) is included in this analysis. The final
sample seems to be a fairly good representation of the population of all TTU juniors and seniors
in terms of sex, ethnicity, and college.
From the 2011 administration of the CAAP Science Test it appears that TTU juniors (mean =
61.2, SD = 4.2) score similarly on average to the national sample of juniors (mean = 61.3, SD =
4.7) and that the TTU seniors (mean = 61.6, SD = 4.3) score slightly higher on average than the
national sample of seniors (mean = 60.4, SD = 4.8). This difference in scores from the TTU
seniors and the national sample of seniors is statistically significant (t = 3.46, p < 0.001).
For the TTU sample, a regression model was used to find if the variables student classification,
sex, GPA, and self-reported effort are significant predictors for a participant’s CAAP Science Test
score. The results suggest that sex (t = 2.06, p < 0.0001), GPA (t = 1.28, p = 0.0034), and selfreported effort (t = -2.77, p < 0.0001; t = 1.33, p = 0.0049) are significant predictors, and that
student classification is not a significant predictor (t = 0.35, p = 0.4133). The results suggest
that on average male students, students with higher GPA’s, and students reporting that they
gave more effort score higher on the CAAP Science Test – all else being equal.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 3 of 11
Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
Introduction
The Collegiate Assessment of Academic Proficiency (CAAP) “is the standardized, nationally
normed assessment program from ACT that enables postsecondary institutions to assess,
evaluate, and enhance student learning outcomes and general education program outcomes.” 1
The CAAP offers six different modules: Reading, Writing Skills, Writing Essay, Mathematics,
Science, and Critical Thinking. 2 As per decision of the University Assessment Committee in
April 2010, the Office of Planning and Assessment administered the Science Test of the CAAP
during fall 2011. The Science Test “is a 45-item, 40-minute test designed to measure students'
skills in scientific reasoning. The contents of the Science Test are drawn from biological sciences
(e.g., biology, botany, and zoology), chemistry, physics, and the physical sciences (e.g., geology,
astronomy, and meteorology). The test emphasizes scientific reasoning skills rather than recall
of scientific content or a high level of skill in mathematics or reading.” 3 Please find more
information on how the test is constructed and scored in Appendix A.
In coordination with the various colleges, the Office of Planning and Assessment administered
the Science Test in select courses across campus. Courses were chosen based on their
enrollment by student classification and size. As juniors and seniors were the target group,
courses with high junior and senior enrollment were selected to participate. In order to
represent the colleges equally larger courses were chosen from larger colleges and smaller
courses were chosen from smaller colleges. Included in the sample were courses from the
following colleges: College of Agricultural Sciences and Natural Resources, College of
Architecture, College of Arts and Sciences, College of Education, College of Human Sciences,
College of Mass Communications, College of Visual and Performing Arts, Edward E. Whitacre Jr.
College of Engineering, Honors College, and Jerry S. Rawls College of Business Administration.
For those courses that were selected to participate, the CAAP was administered during the
regularly scheduled class time. This allowed most students from each class to participate and
helped reduce the bias that may have been created if students had volunteered to take the
survey on their own time.
A total of 422 Texas Tech University (TTU) students participated in the fall 2011 administration
of the Science Test. The Office of Planning and Assessment identified 6 irregularities (e.g.,
pattern scoring) that were exempt from scoring. As juniors and seniors were the target group,
another 49 participants were excluded because they had less than 50 earned credit hours (it
was decided to include sophomores with more than 50 earned credit hours as part of the group
of juniors in the analysis). The remaining 365 answer sheets were scored by ACT.
1
http://www.act.org/caap/, accessed 12/2/11
http://www.act.org/caap/about/modules.html, accessed 12/2/11
3
http://www.act.org/caap/test/science.html, accessed 12/2/11
2
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 4 of 11
Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
Sample
ACT provided scores for the 365 answer sheets of students who participated in the fall 2011
administration of the CAAP. Since the students’ R numbers were used as the student ID on the
Science Test, the Office of Planning and Assessment was able to obtain demographic
information for the participants from Institutional Research (IR). The demographic information
obtained includes sex, ethnicity, student classification, and college. The following graphs
compare the final samples of juniors (n = 169) and seniors (n = 195) to the population of all TTU
juniors and seniors by sex, ethnicity, and college. Note that the group of juniors for this analysis
includes 31 sophomores with more than 50 earned credit hours.
Sample and Population by Sex:
Juniors
Sample and Population by Sex:
Seniors
80%
60%
50%
60%
40%
40%
20%
Sample
30%
Sample
Population
20%
Population
10%
0%
0%
Female
Male
Female
Male
The samples of juniors and seniors both appear to be good representations of their respective
populations in terms of sex.
Sample and Population by Ethnicity: Juniors
80%
60%
Sample
40%
Population
20%
0%
AAM
AI
AS
B
HI
M
PI
NR
U
WH
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 5 of 11
Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
Sample and Population by Ethnicity: Seniors
80%
60%
Sample
40%
Population
20%
0%
AAM
AI
AS
B
HI
M
PI
NR
U
WH
The samples of juniors and seniors both appear to be good representations of their respective
populations in terms of ethnicity.
Sample and Population by College: Juniors
40%
30%
Sample
20%
Population
10%
0%
AG
AR
AS
BA
ED
EN
HR
HS
MC
UC
UN
VP
Sample and Population by College: Seniors
35%
30%
25%
20%
Sample
15%
Population
10%
5%
0%
AG
AR
AS
BA
ED
EN
HR
HS
MC
UC
UN
VP
It appears that for the juniors the Whitacre College of Engineering is overrepresented, and that
the College of Arts and Sciences and the Rawls College of Business Administration are
underrepresented. It appears that for the seniors the Whitacre College of Engineering is
overrepresented, and that the College of Arts and Sciences is underrepresented.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 6 of 11
Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
Results
CAAP Scores Overall
The following table and graph give a summary of the juniors’ and seniors’ CAAP scores.
Summary of CAAP Scores by Student Classification
n
169
196
Junior
Senior
Mean
61.2
61.6
SD
4.2
4.3
Min
52
52
Median
61
61.5
Max
72
71
Histogram of CAAP Scores by Student Classification
90
80
70
60
50
40
30
20
10
0
67
80
75
50
Juniors
34
18
45
50
13
55
Seniors
22
4
60
65
70
2
75
80
Note that the TTU juniors (mean = 61.2, SD = 4.2) and seniors (mean = 61.6, SD = 4.3)
performed similarly on the CAAP. The ACT website has a summary of the scores from 10,477
juniors from 88 institutions and 8,116 seniors from 98 institutions that participated in the
Science Test in fall 2011. 4 On average the TTU juniors scored similarly to the national sample of
juniors (mean = 61.3, SD = 4.7). 5 On average the TTU seniors scored just above the national
sample of seniors (mean = 60.4, SD = 4.8). 6 This difference is statistically significant (t = 3.46, p
< 0.001).
4
http://act.org/caap/norms/, accessed 12/16/11
http://act.org/caap/norms/pdf/11Table12.pdf, accessed 12/16/11
6
http://act.org/caap/norms/pdf/11Table13.pdf, accessed 12/16/11
5
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 7 of 11
Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
CAAP Scores by Student Classification, Sex, Effort, and GPA
The following table gives a summary of the overall sample by student classification, sex,
participants’ self-report of effort, and GPA. These variables will be included in a regression
model for predicting CAAP Science Test score.
Summary of Predictor Variables for CAAP Scores
Student Classification
n
Junior
169
Senior
196
Sex
n
Female
162
Male
203
Self-Reported Effort
n
Tried my best
129
Gave moderate effort
173
Gave little effort
48
Gave no effort
9
n
Mean
GPA
346
2.99
%
46.3
53.7
%
44.4
55.6
%
35.9
48.2
13.4
2.5
SD
0.58
The following table gives a summary of a regression model for predicting a participant’s CAAP
Science Test score. For the variable “Self-Reported Effort”, the “Gave little effort” group and
the “Gave no effort” group were combined in the model because of the small number of
participants in the “Gave no effort” group. For this variable, the “Gave moderate effort” group
was used as the reference group for the “Tried my best” group and the “Gave little or no effort
group”.
Linear Regression Model for Predicting CAAP Science Test Scores
Variable
Intercept
Student Classification
Sex
GPA
Gave little or no effort
Tried my best
df
1
1
1
1
1
1
Parameter Standard
Estimate
Error
56.26
1.43
0.35
0.42
2.06
0.44
1.28
0.43
-2.77
0.61
1.33
0.47
t value
39.32
0.82
4.62
2.95
-4.56
2.83
p value
39.32
0.82
4.62
2.95
-4.56
2.83
95% Confidence Interval
Lower Limit Upper Limit
53.44
59.07
-0.49
1.18
1.18
2.93
0.42
2.13
-3.97
-1.58
0.40
2.25
Overall, the model was found to be statistically significant (F = 15.78, p < 0.0001, R2 = 0.19).
Within the model, the significant predictors of CAAP score were sex (t = 2.06, p < 0.0001), GPA
(t = 1.28, p = 0.0034), gave little or no effort (t = -2.77, p < 0.0001), and tried my best (t = 1.33, p
= 0.0049). Specifically, this model suggest that on average male students score higher than
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 8 of 11
Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
female students, students with higher GPA’s score higher, students reporting that they gave
little or no effort score lower than students reporting that they gave moderate effort, and
students reporting that they tried their best score higher than students reporting that they gave
moderate effort. Student classification was not a significant predictor of CAAP scores (t = 0.35,
p = 0.4133). The following graphs show the average CAAP scores by the significant predictor
variables from the model: sex, GPA, and self-reported effort.
Average CAAP Scores by Sex
62.5
62.2
62
75
70
61.5
65
61
60.5
CAAP Scores by GPA
60
60.4
55
60
59.5
50
Female
Male
1
1.5
2
2.5
3
3.5
4
Average CAAP Scores by Self-Reported Effort
64
62.9
62
61.4
58.9
60
58
55.2
56
54
52
50
Tried my best
Gave moderate Gave little effort Gave no effort
effort
Conclusion
Overall it appears that TTU juniors score similarly on average to other juniors nationally and
that TTU seniors score slightly higher on average than other seniors nationally on the CAAP
Science Test. For the TTU sample it appears that on average male students, students with
higher GPA’s, and students reporting that they gave more effort score higher on the CAAP
Science Test. Student classification was not a significant predictor of CAAP Science Test score.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 9 of 11
Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
Appendix
Appendix A: Information about the CAAP Science Test 7
7
Screenshots from http://www.act.org/caap/test/science.html (accessed on 12/14/11)
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 10 of 11
Collegiate Assessment of Academic Proficiency: Science Test
Fall 2011
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, December 2011
Page 11 of 11