Online Senior Assessment 2010: Mathematics
Introduction
The third component of the OSA contains questions for the Mathematics core. The first
questions ask the participants about how they fulfilled their Mathematics core
requirement.
The following table shows the number and percentage of participants who selected
each response to the first question regarding where students took their core curriculum
course. The number of participants selecting each response adds up to more than the
755 total participants because those who did not select “I took all my core curriculum
classes in mathematics or logic at Tech” could select more than one of the other
responses.
How did you complete your core curriculum requirement in Mathematics?
% of all
% of all
Response
N
Responses Participants
I took at least one mathematics or logic core
85
10.4%
11.3%
curriculum class through dual credit in high school.
I took at least one advanced placement
78
9.5%
10.3%
mathematics core curriculum class in high school.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 1 of 19
Online Senior Assessment 2010: Mathematics
How did you complete your core curriculum requirement in Mathematics? (Cont.)
I took at least one CLEP exam for mathematics
21
2.6%
2.8%
core curriculum credit.
I received transfer core curriculum mathematics
credit for at least one class that I took at another
251
30.6%
33.2%
institution.
I took all my core curriculum classes in
386
47.0%
51.1%
mathematics or logic at Tech.
821
100.0%
Total Responses
For the analysis in this report the 755 participants are divided into the “TTU” group and
the “ELSE” group. The TTU group represents the 386 participants (i.e., 51.1%) who
selected “I took all my core curriculum classes in mathematics or logic at Tech” and the
ELSE group represents the 369 participants (i.e., 48.9%) who selected one or more of
the other responses indicating that they took their core curriculum class in Mathematics
elsewhere. The following pie chart shows this division of the sample.
Mathematics
369 (48.9%)
TTU
386 (51.1%)
ELSE
The 369 participants in the ELSE group were also asked if the class they took outside of
Tech counted for their core curriculum credit. Of the 369 participants who reported
taking a Mathematics course elsewhere, 21 (i.e., 5.7%) reported that they did not know
if the course counted for their Mathematics core curriculum credit and 348 (i.e., 94.3%)
reported that the course did count for their Mathematics core curriculum credit. The 348
participants who reported that the course taken outside of Tech did count for their
Mathematics core curriculum credit were also asked which one counted. The following
table shows the number and percentage of the 348 participants who selected each
response.
Which one?
Response
A dual credit class.
An advanced placement class.
A CLEP exam.
A class I took at another institution.
I don't know.
Total
N
%
68
19.5%
43
12.4%
9
2.6%
225
64.7%
3
0.9%
348 100.0%
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 2 of 19
Online Senior Assessment 2010: Mathematics
The system stores some data for each of the participants and so it was possible to
identify the respondents who major in Mathematics. Only 9 participants reported that
they are mathematics majors, so our group of so-called experts will include all of the
participants with majors that require any mathematics beyond Calculus II. The following
majors were thus identified and classified as Mathematics majors: Chemical
Engineering, Civil Engineering, Computer Engineering, Computer Science, Electrical
Engineering, Engineering Technology, Industrial Engineering, Mathematics, Mechanical
Engineering, Petroleum Engineering, and Physics. The following table shows that there
were a total of 127 Mathematics majors in the OSA sample. It also displays how many
participants were in each of the Mathematics majors.
Mathematics Majors
Major
Frequency Percentage
Chemical Engineering
11
8.7%
Civil Engineering
13
10.2%
Computer Engineering
3
2.4%
Computer Science
15
11.8%
Electrical Engineering
13
10.2%
Engineering Technology
10
7.9%
Industrial Engineering
6
4.7%
Mathematics
9
7.1%
Mechanical Engineering
25
19.7%
Petroleum Engineering
20
15.7%
Physics
2
1.6%
Total
127
100.0%
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 3 of 19
Online Senior Assessment 2010: Mathematics
Results
The student learning outcomes for Mathematics are:
Apply arithmetic, algebra, geometry and statistics to solve problems.
Represent and evaluate basic mathematical information numerically,
graphically, and symbolically.
Use mathematical and logical reasoning to evaluate the validity of an
argument.
Interpret mathematical models such as formulas, graphs, tables and
schematics, and draw inference from them.
The first learning outcome seems to align well with the third and fourth question. The
second learning outcome seems to align well with the first and second question. The
third and fourth learning outcomes seem to align with the fifth question.
The Mathematics section of the OSA contains five knowledge questions. These are
shown below as a screenshot from the actual instrument. For analysis purposes, the
answers were coded from 1 to 4 in the order they appear on the actual instrument.
Attachment D shows how many times each answer choice was selected by the different
participants for all of the Mathematics questions.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 4 of 19
Online Senior Assessment 2010: Mathematics
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 5 of 19
Online Senior Assessment 2010: Mathematics
Mathematics 1:
The chart below shows the distributions of answers for the first question for participants
who took their class for the Mathematics core requirement at TTU (blue) and
participants who took their class for the Mathematics core requirement elsewhere (red).
Answer 3 is the correct choice. It can be seen that just over half of both groups chose
the correct answer. Overall, a few more people in the ELSE group chose the correct
answer as compared to the TTU group (54.2% vs. 53.4%). This difference is not
statistically significant at the 0.05 level (see attachment A for details). This means that
on average students who take their class for the Mathematics core requirement
elsewhere do not do better with this question than the students who take their
Mathematics class at TTU. Since the first question aligns with the second learning
outcome, this suggests that on average students who take their Mathematics course
elsewhere meet this learning outcome similar to students who take their course at TTU.
Mathematics 1
60.0%
53.4% 54.2%
50.0%
40.0%
35.2%
32.8%
30.0%
TTU
ELSE
20.0%
9.6%
10.0%
11.4%
1.8% 1.6%
0.0%
1
2
3
4
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 6 of 19
Online Senior Assessment 2010: Mathematics
Mathematics 2:
The chart below shows the distributions of answers for the second question for
participants who took their class for the Mathematics core requirement at TTU (blue)
and participants who took their class for the Mathematics core requirement elsewhere
(red). Answer 1 is the correct choice. It can be seen that a majority of participants in
both groups chose the correct answer, with a few in each group that chose other
answers. Overall, a few more people in the TTU group chose the correct answer as
compared to the ELSE group (73.8% vs. 71.3%). This difference is not statistically
significant at the 0.05 level (see attachment A for details). This means that on average
students who take their class for the Mathematics core requirement at TTU do not do
better with this question than the students who take their Mathematics class elsewhere.
Since the second question aligns with the second learning outcome, this suggests that
on average students who take their Mathematics course at TTU meet this learning
outcome similar to students who take their course elsewhere.
Mathematics 2
80.0%
73.8%
71.3%
70.0%
60.0%
50.0%
40.0%
TTU
30.0%
ELSE
20.0%
13.5% 15.4%
10.0%
11.1% 11.9%
1.6% 1.4%
0.0%
1
2
3
4
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 7 of 19
Online Senior Assessment 2010: Mathematics
Mathematics 3:
The chart below shows the distributions of answers for the third question for participants
who took their class for the Mathematics core requirement at TTU (blue) and
participants who took their class for the Mathematics core requirement elsewhere (red).
Answer 1 is the correct choice. It can be seen that approximately half of both groups
chose the correct answer. The fact that almost 40% in both groups chose the second
answer option might indicate that this question is a little bit too hard or that the answer
choices are too similar. Overall, a few more people in the TTU group chose the correct
answer as compared to the ELSE group (52.3% vs. 50.1%). This difference is not
statistically significant at the 0.05 level (see attachment A for details). This means that
on average students who take their class for the Mathematics core requirement at TTU
do not do better with this question than the students who take their Mathematics class
elsewhere. Since the third question aligns with the first learning outcome, this suggests
that on average students who take their Mathematics course at TTU meet this learning
outcome similar to students who take their course elsewhere.
Mathematics 3
60.0%
52.3%
50.1%
50.0%
37.8%
40.0%
39.8%
30.0%
TTU
ELSE
20.0%
8.3% 7.6%
10.0%
1.6% 2.4%
0.0%
1
2
3
4
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 8 of 19
Online Senior Assessment 2010: Mathematics
Mathematics 4:
The chart below shows the distributions of answers for the fourth question for
participants who took their class for the Mathematics core requirement at TTU (blue)
and participants who took their class for the Mathematics core requirement elsewhere
(red). Answer 1 is the correct choice. It can be seen that a large majority of both groups
chose the correct answer. Overall, just slightly more in the ELSE group chose the
correct answer as compared to the TTU group (85.1% vs. 85.0%). This difference is not
statistically significant at the 0.05 level (see attachment A for details). This means that
on average students who take their class for the Mathematics core requirement
elsewhere do not do better with this question than the students who take their
Mathematics class at TTU. Since the fourth question aligns with the first learning
outcome, this suggests that on average students who take their Mathematics course
elsewhere meet this learning outcome similar to students who take their course at TTU.
Mathematics 4
90.0%
85.0% 85.1%
80.0%
70.0%
60.0%
50.0%
TTU
40.0%
ELSE
30.0%
20.0%
10.9% 10.8%
10.0%
3.9% 4.1%
0.3% 0.0%
0.0%
1
2
3
4
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 9 of 19
Online Senior Assessment 2010: Mathematics
Mathematics 5:
The chart below shows the distributions of answers for the fifth question for participants
who took their class for the Mathematics core requirement at TTU (blue) and
participants who took their class for the Mathematics core requirement elsewhere (red).
Answer 4 is the correct choice. It can be seen that approximately half of both groups
chose the correct answer. The fact that only about 50% of the participants in both
groups selected the correct answer might indicate that this question is too difficult.
Overall, more people in the TTU group chose the correct answer as compared to the
Else group (50.5% vs. 47.2%). This difference is not statistically significant at the 0.05
level (see attachment A for details). This means that on average students who take their
class for the Mathematics core requirement at TTU do not do better with this question
than the students who take their Mathematics class elsewhere. Since the fifth question
aligns with the third and fourth learning outcomes, this suggests that on average
students who take their Mathematics course at TTU meet these learning outcomes
similar to students who take their course elsewhere.
Mathematics 5
60.0%
50.5%
47.2%
50.0%
40.0%
32.5%
28.0%
30.0%
TTU
ELSE
20.0%
15.8%
13.6%
5.7% 6.8%
10.0%
0.0%
1
2
3
4
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 10 of 19
Online Senior Assessment 2010: Mathematics
Mathematics Average:
The table below compares the differences between TTU and ELSE when the results for
all the questions are averaged (e.g., if a student got 4 out of the 5 questions correct, his
score will be 4 / 5 = .80). The mean is slightly higher for students who took their core
requirement for Mathematics at TTU. However, this difference is not statistically
significant at the 0.05 level. This means that on average students who take their
Mathematics course at TTU do not do better with the Mathematics section of the OSA
than students who take their Mathematics course elsewhere.
N
Mathematics
Overall
386
Core at TTU
Mean
SD
63.0%
Core Elsewhere
Mean
SD
N
27.5%
369
61.6%
T-stat
28.7%
0.701
P-value
0.483
The chart below shows the distributions of the average scores for participants who took
their class for the Mathematics core requirement at TTU (blue) and participants who
took their class for the Mathematics core requirement elsewhere (red). The distributions
are similar, but it looks like a few more participants in the TTU group answered 4 out of
5 correct and that a few more in the ELSE group answered 2 out of 5 correct.
Mathematics Overall: TTU vs. ELSE
30.0%
24.9%
24.4%
25.0%
21.2%
20.0%
21.0%
21.1%
21.7%
19.9%
18.2%
15.0%
TTU
11.7%
10.4%
ELSE
10.0%
5.0%
2.6%3.0%
0.0%
0%
20%
40%
60%
80%
100%
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 11 of 19
Online Senior Assessment 2010: Mathematics
The table below shows a comparison of the average scores for the participants
selecting each course option within the ELSE group (those selecting more than one
course option were excluded from the analysis). The table includes the F value and P
value for an analysis of variance comparing the means. Although the table shows
information for all four ELSE options, the group with less than 15 participants (i.e.,
CLEP Exam) was excluded from the analysis of variance.
Elsewhere
Dual Credit
Advanced Placement
CLEP Exam
Another Institution
Total
N
47
45
11
215
318
Mean
65.5%
74.2%
61.8%
56.5%
60.5%
St. Dev.
28.2%
30.0%
28.9%
28.1%
29.0%
F-value P-value
8.174 < 0.001
Based on the mean, students who took their Mathematics course through advanced
placement are the highest-performing group and students who took their course at
another institution are the lowest-performing group. The means are significantly
different at the 0.05 level. This suggests that on average for the students who take their
Mathematics course elsewhere, which course option they use to take their Mathematics
course makes a difference in how they perform on the Mathematics section of the OSA.
Tukey’s method for multiple comparisons was used to find which course option means
are significantly different. The following table shows the significant differences at the
0.05 level.
Comparison
Advanced Placement vs. Another Institution
P-value
< 0.001
The table shows the one difference that was significant at the 0.05 level. This
difference suggest that on average students who take their Mathematics course through
advanced placement do better on the Mathematics section of the OSA than students
who take their Mathematics course at another institution. This outcome makes sense
when considering that students who take an AP course need to pass an exam to
receive credit, which suggests that they performed well in their AP class.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 12 of 19
Online Senior Assessment 2010: Mathematics
The chart below shows the distributions of the average scores for those participants in
the Mathematics majors group (blue) and those participants who are not in the
Mathematics majors group (red). Almost half of the Mathematics majors answered all of
the questions correctly, while the non-majors seem to center more around two or three
correct answers. Overall, the majors have a higher average than the non-majors
(78.1% vs. 59.1%). This difference is statistically significant at the .05 level (see
attachment B for details). This means that on average Mathematics majors perform
better than non-majors on the Mathematics section of the OSA.
Mathematics Overall: Majors vs. Non-majors
45.0%
42.5%
40.0%
35.0%
30.0%
25.0%
20.2%
20.0%
22.3%
Majors
16.5%
15.0%
16.4%
Non-Majors
12.4%
8.7%
10.0%
5.0%
26.8%
25.6%
3.0% 3.9%
1.6%
0.0%
0%
20%
40%
60%
80%
100%
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 13 of 19
Online Senior Assessment 2010: Mathematics
The chart below shows the distribution of the average scores for the female participants
(blue) and the male participants (red). The male participants had higher overall average
scores than the female participants (mean of 72.7% vs. 55.0%). This is significant at the
0.05 level (see attachment C). This suggests that on average male students do better
than female students with the Mathematics section of the OSA. It appears in the chart
below that the significant difference is due to more male participants answering 4 or 5
out of 5 questions correct and more female participants answering 0, 1, or 2 out of 5
questions correct.
Mathematics Overall by Sex
35.0%
29.8%
29.5%
30.0%
31.4%
25.0%
20.6%
18.3% 18.4%
20.0%
14.7%
15.0%
Female
13.5%
13.2%
Male
10.0%
5.0%
3.6%
1.6%
5.4%
0.0%
0%
20%
40%
60%
80%
100%
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 14 of 19
Online Senior Assessment 2010: Mathematics
The following table shows the correlations between the overall average for the
Mathematics questions and time to complete OSA, GPA, SAT score, ACT score,
transfer credits, total credits earned, and age (p-values for the correlations are in
parenthesis). The correlations with GPA, SAT score, ACT score, transfer credits, and
total credits earned are significant at the 0.05 level. These correlations suggest that on
average students with higher GPA’s, higher SAT scores, higher ACT scores, less
transfer credit, and more total credits do better on the Mathematics section of the OSA.
Some of these correlations are small and are more likely to be found statistically
significant because of the large sample size.
Correlation
Mathematics
P-value
Overall
N
Time
-0.069
(0.058)
755
GPA
0.138
(<0.001)
755
SAT
0.444
(<0.001)
511
ACT
0.462
(<0.001)
388
Transfer
Credits
-0.103
(0.005)
755
Total
Credits
0.162
(<0.001)
755
Age
0.021
(0.562)
755
The following tables show the results of regression models for the overall average for
the Mathematics questions including all of the variables that have been explored in this
analysis. There are three separate regression models because not all of the
participants have SAT or ACT scores. Since few students have both a SAT score and
an ACT score, there would be too many missing values if both scores were included in
the same regression model. The first model excludes both in order to include most
respondents in the analysis.
Mathematics Model 1
N
F
P-value
753
20.68
< 0.001
Variable
Coefficient P-value
Intercept
0.2415
0.007
Time
-0.000035
0.039
Sex
-0.150 < 0.001
GPA
0.097 < 0.001
Transfer Credits
-0.00029
0.606
Total Credits
0.0010
0.020
Age
0.0028
0.126
Mathematics Major
0.123 < 0.001
Mathematics class taken at TTU
-0.0055
0.797
This first model excludes SAT and ACT score to include 753 of the 755 participants.
The model overall is significant at the 0.05 level (R2 = 0.1819). For this model time to
complete OSA, sex, GPA, total credits earned, and Mathematics major are the
significant predictors at the 0.05 level for the overall average for the Mathematics
questions. These predictors suggest that on average students who took less time to
complete the OSA, male students, students with higher GPA’s, students with more total
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 15 of 19
Online Senior Assessment 2010: Mathematics
credits earned, and Mathematics majors do better on the Mathematics section of the
OSA when the other variables in the model are held constant.
Mathematics Model 2
N
F
P-value
511
24.01
< 0.001
Variable
Coefficient P-value
Intercept
-0.419
0.003
Time
-0.000050
0.028
Sex
-0.137 < 0.001
GPA
0.036
0.085
Transfer Credits
0.00051
0.485
Total Credits
0.00062
0.258
Age
0.0073
0.026
Mathematics Major
0.092
0.004
Mathematics class taken at TTU
0.035
0.162
SAT
0.00068 < 0.001
This second model includes SAT score and excludes ACT score to include 511 of the
755 participants. The model overall is significant at the 0.05 level (R2 = 0.3014). For
this model time to complete OSA, sex, age, Mathematics major, and SAT score are the
significant predictors at the 0.05 level for the overall average for the Mathematics
questions. These predictors suggest that on average students who took less time to
complete the OSA, male students, older students, Mathematics majors, and students
with higher SAT scores do better on the Mathematics section of the OSA when the
other variables in the model are held constant.
Mathematics Model 3
N
F
P-value
387
20.07
< 0.001
Variable
Coefficient P-value
Intercept
-0.246
0.086
Time
-0.000053
0.379
Sex
-0.131 < 0.001
GPA
0.027
0.275
Transfer Credits
-0.000160
0.826
Total Credits
0.00036
0.543
Age
0.0050
0.186
Mathematics Major
0.104
0.003
Mathematics class taken at TTU
0.037
0.166
ACT
0.029 < 0.001
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 16 of 19
Online Senior Assessment 2010: Mathematics
This third model includes ACT score and excludes SAT score to include 387 of the 755
participants. The model overall is significant at the 0.05 level (R2 = 0.3239). For this
model sex, Mathematics major, and ACT score are the significant predictors at the 0.05
level for the overall average for the Mathematics questions. These predictors suggest
that on average male students, Mathematics majors, and students with higher ACT
scores do better on the Mathematics section of the OSA when the other variables in the
model are held constant.
Limitations
It is difficult to measure Mathematics knowledge with only five questions. The
questions, though, do seem to challenge participants and seem to be good
discriminators of Mathematical understanding. However, two questions might be a little
bit too hard since only about 50% selected the correct answers.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 17 of 19
Online Senior Assessment 2010: Mathematics
Attachments
Attachment A: Summary of Chi-Square Tests for Questions 1 - 5
TTU
(N=386)
correct incorrect
206
180
285
101
202
184
328
58
195
191
Mathematics 1
Mathematics 2
Mathematics 3
Mathematics 4
Mathematics 5
Else
(N=348)
correct incorrect
200
169
263
106
185
184
314
55
174
195
Chi Statistic
0.05
0.62
0.36
0.00
0.85
Chi Probability
0.8186
0.4305
0.5462
0.9629
0.3554
Attachment B: 2-Sample T-Test for Average Scores of Majors and Non-Majors
Majors
Mean
SD
N
Mathematics
Overall
127
78.1%
N
24.9%
628
Non-Majors
Mean
SD
59.1%
T-stat
27.6%
7.188
P-value
< 0.001
Attachment C: 2-Sample T-Test for Average Scores by Sex
N
Mathematics
Overall
441
Female
Mean
SD
55.0%
N
27.3%
312
Male
Mean
SD
72.7%
T-stat
25.7%
-8.991
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 18 of 19
P-value
< 0.001
Online Senior Assessment 2010: Mathematics
Attachment D: Number of Participants Selecting Each Answer for Each Question
Mathematics 1
Answer
TTU
1
7
2
37
3
206
4
136
Mathematics 4
Answer
TTU
1
328
2
42
3
15
4
1
ELSE
6
42
200
121
ELSE
314
40
15
0
Mathematics 2
Answer
TTU
1
285
2
6
3
52
4
43
Mathematics 5
Answer
TTU
1
61
2
108
3
22
4
195
ELSE
263
5
57
44
Mathematics 3
Answer
TTU
ELSE
1
202
185
2
146
147
3
32
28
4
6
9
ELSE
50
120
25
174
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010
Page 19 of 19