Online Senior Assessment 2010: General Report
Table of Contents
EXECUTIVE SUMMARY .......................................................................................................................... 2
INTRODUCTION ........................................................................................................................................ 4
RESULTS .................................................................................................................................................... 8
LIMITATIONS ........................................................................................................................................... 19
APPENDIX ................................................................................................................................................ 21
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 1 of 43
Online Senior Assessment 2010: General Report
EXECUTIVE SUMMARY
In 2010, the Online Senior Assessment was administered during the spring semester.
The final sample size was 755. That means that about 10% of the total population of
senior students (those with >90 hours of credit) at Texas Tech are included in this
analysis. This sample contains more female students than would be expected.
However, the sample seems to represent the senior population well in terms of ethnicity
and college.
The OSA included a total of 32 knowledge questions (i.e., questions that can be
answered either correctly or incorrectly) from the following core curriculum areas:
Humanities, Multicultural, Mathematics, Natural Sciences, and Social and Behavioral
Sciences. For 20 of those questions, students who took their course for core
requirement at Texas Tech (the “TTU” group) performed better than those who took
their course elsewhere (the “ELSE” group). The latter group performed better on the
other 12 questions. The difference was statistically significant at the 0.05 level for only
six of the questions (five questions the TTU group did better with and one question the
Else group did better with).
When averaging the results of the knowledge questions by area first and then
calculating the overall average (i.e., making sure that those areas with more questions
would not weigh more), the average score is 66.9%. The lowest overall score is 25.9%
and the highest score is 97.1%. The same results by area can be split up between
students who took the respective courses for core requirement at TTU and those who
took it elsewhere (TTU vs. ELSE). This could provide some information regarding if
students who took the core at Texas Tech perform better or worse on average than
those who took it elsewhere. Running a 2-sample t-test (a statistical test often used to
check for differences in means of two different groups) shows that the means of those
average scores are not different from each other (not statistically significant at the 0.05
level). The mean scores show that on average participants who took their core at TTU
perform similar to those participants who took their core elsewhere (67.7% for TTU vs.
66.7% for ELSE). A paired t-test was run for only those students who have taken some
classes for core requirements at Tech and some elsewhere (525 students). This allows
us to compare if the same student performs any different depending on where the
course was taken (at TTU or elsewhere). Again, the average is not significantly different
(68.0% for TTU vs.66.9% for ELSE).
The average per area was compared between TTU and ELSE per core area (including
Humanities, Multicultural, Mathematics, Natural Sciences, and the Social and
Behavioral Sciences). The only core area that showed a significant difference (at the
0.05 level) was the Multicultural section. Students who took their Multicultural course at
TTU performed significantly better on the Multicultural section of the OSA than students
who took their Multicultural course elsewhere. Social and Behavioral Sciences and
Mathematics have the lowest average scores while Multicultural and Humanities have
some of the highest average scores. In the individual core area reports the ELSE group
is further broken down to compare the different options for students taking their core
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 2 of 43
Online Senior Assessment 2010: General Report
curriculum requirement outside of Texas Tech (dual credit class, advanced placement
class, CLEP exam, class at another institution). Of the ELSE group, the participants
taking their core curriculum requirement through an advanced placement class
consistently have the highest average core area scores and the participants taking their
core curriculum requirement through a class at another institution consistently have the
lowest average core area scores.
The individual core area reports also compare the core area scores by major, sex, and
some other variables of interest (time to complete OSA, GPA, SAT score, ACT score,
transfer credits, total credits earned, and age). The Natural Sciences is the only core
area for which the participants majoring in that area did significantly better than the
other participants at the 0.05 level. On average, the female participants performed
better on the Humanities section of the OSA and the male participants performed better
on the Mathematics, Natural Sciences, and Social and Behavioral Sciences sections of
the OSA. The overall scores for all of the knowledge questions from the OSA
significantly correlate at the 0.05 level with time to complete OSA, GPA, SAT score,
ACT score, transfer credits, total credits earned, and age. These correlations suggest
that on average students who spend less time to complete the OSA, students with
higher GPA’s, students with higher SAT scores, students with higher ACT scores,
students with fewer transfer credits, and students with more total credits perform better
on the OSA.
Limitations for the study include that there are only a few questions per core area which
may not represent the courses that students took for their core curriculum requirements,
the participants do not have an incentive to do well, and the ELSE group does not
represent a homogenous group for comparison with the TTU group.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 3 of 43
Online Senior Assessment 2010: General Report
INTRODUCTION
The Online Senior Assessment (OSA) was designed in 2008 to assess core-curriculum
knowledge and abilities. In 2010, it was administered for the third time to all graduating
seniors (i.e., students having 90 or more credit hours) between 3/01/2010 and
3/31/2010. Of the 6,977 seniors at Texas Tech during the spring 2010 semester, 846
students participated in the assessment leading to an initial response rate of 12.13%.
Four of the participants were randomly selected to win $1,000 toward tuition and fees.
The instrument has one section for each of the following areas (see Attachment A for
screen shots of the full instrument):
Humanities: 3 self-assessment questions, 4 knowledge questions
Multicultural: 7 knowledge questions
Mathematics: 5 knowledge questions
Natural Sciences: 6 knowledge questions
Technology and Applied Science: 4 agreement questions
Social and Behavioral Sciences: 10 knowledge questions
Some changes were made to the OSA in 2010. The instructions were changed to let
students know that they would not be entered to win the $1,000 if their survey was
incomplete or if it was apparent that they did not put forth an honest effort. Extra
questions were added to each section that asked participants to clarify where they took
their course for each core requirement (e.g., dual credit course, advanced placement
course, CLEP exam, or at another institution). In the Humanities section, some
grammar mistakes were corrected and the last answer option for the seventh question
was changed to make the question more difficult. In the Mathematics section, some of
the wording in all of their questions was changed and the fourth question was replaced
with a new question to help make the section less difficult. In the Social and Behavioral
Sciences section, some of the wording in the first and fourth questions was changed to
make the questions less difficult. The Visual and Performing Arts section was removed
from the assessment. No changes were made to the Multicultural section, the Natural
Sciences section, or the Technology and Applied Science section of the OSA.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 4 of 43
Online Senior Assessment 2010: General Report
Inquisite, the survey software, provides information about when students first started the
assessment and when they submitted it. The following graph shows how much time the
participants spent on the OSA in minutes.
140
Time Spent on OSA
120
Frequency
100
80
60
40
20
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
95
100
105
110
115
120
150
180
210
240
More
0
Time in Minutes
The following table shows descriptive statistics for time spent on the OSA.
Time Spent on OSA
(days: hours: minutes: seconds)
Mean
0:01:22:21
St. Deviation
0:08:36:11
Minimum
0:00:02:25
1st Quartile
0:00:19:41
Median
0:00:27:27
3rd Quartile
0:00:38:50
Maximum
7:01:35:33
The minimum time spent on the assessment was 2 minutes and 25 seconds and the
maximum time was over 7 days. The average time participants spent on the
assessment was 1 hour 22 minutes and 21 seconds. This average is elevated because
of the large outliers on the higher end of time taken to complete the assessment. These
outliers in all likelihood were students who started the assessment and did not get back
to it until a few days later. The median time of 27 minutes and 27 seconds may be a
better reflection of how much time it took most students to take the assessment.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 5 of 43
Online Senior Assessment 2010: General Report
There were 91 participants who took less than 15 minutes to take the assessment. They
were removed before any further analysis was done since it takes about as long just to
read through the instrument. Inquisite forces students to choose an answer before
moving on to the next section, so in all likelihood, these students just clicked on any
answer to get to the end where they could submit their name for the chance to win
$1,000 toward tuition and fees. Eliminating these respondents leads to a final sample
size of 755 students (i.e. 10.8% of the population of TTU seniors).
Given that the final response rate is only about 10%, it is questionable if the sample is
representative of the population. The following graphs show comparisons of the sample
and population by sex, ethnicity, and college to help determine how well the sample
represents the population.
Sample and Population by Sex
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Sample
Population
Female
Male
Sample and Population by Ethnicity
100.0%
80.0%
60.0%
Sample
40.0%
Population
20.0%
0.0%
AI
AS
BL
HI
NR
UN WH
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 6 of 43
Online Senior Assessment 2010: General Report
Sample and Population by College
40.0%
35.0%
30.0%
25.0%
20.0%
15.0%
10.0%
5.0%
0.0%
Sample
Population
AG AR AS BA ED EN HR HS MC VP
Overall, the sample seems to represent the population fairly well in terms of ethnicity
and college. The sample appears to differ from the population of all TTU seniors in
terms of sex, with more female students and fewer male students participating in the
assessment than would be expected from the population.
Before starting every core area section, the participants were asked where they
completed their core requirement for that specific core area. The following table
summarizes the responses. Note that the number of participants selecting each
response adds up to more than the 755 total participants because those that did not
select “I took my core curriculum class at Tech” could select more than one of the other
responses.
TTU
Dual Credit in
High School
Advanced
Placement
CLEP
Exam
Another
Institution
Study
Abroad
Humanities
444
67
51
19
224
NA
Multicultural
558
20
12
1
172
63
Mathematics
386
85
78
21
251
NA
Natural Sciences
453
25
43
7
247
NA
603
25
11
1
123
NA
450
44
30
5
249
NA
482
44
38
9
211
NA
Technology and
Applied Science
Social and
Behavioral Sciences
Average
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 7 of 43
Online Senior Assessment 2010: General Report
RESULTS
One of the main questions the Online Senior Assessment can help answer is if students
who take their classes for core requirement at institutions other than Texas Tech
perform similar to students who took their classes for core requirement at Texas Tech.
In the analysis in this report, we will refer to the students who took their core
requirement courses at Texas Tech as the “TTU” group and we will refer to the students
who have transferred in credits for core requirements as the “ELSE” group.
The data was summarized to show the number of people who had correct and incorrect
answers to each of the 32 knowledge questions from the different areas. A Chi-squared
test for differences in probabilities (a statistical test used to test for differences when the
variables are categorical) was run on each question to test if the difference of
performance between TTU and ELSE is statistically significant at the 0.05 level, i.e. if
there is a relationship between where students took the course for core requirement and
the number of correct answers to the questions (see attachment B for details). The chart
on the next page shows the difference between the percentage of the TTU group that
had the correct answer and the percentage of the ELSE group that had the correct
answer. Positive numbers (i.e., the bars to the right) represent all questions where TTU
had a higher proportion of correct answers (for 20 questions) and negative numbers
(i.e., the bars to the left) represent where ELSE had the higher proportion of correct
answers (for 12 questions). The columns in red show where the difference was
statistically significant at the 0.05 level.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 8 of 43
Online Senior Assessment 2010: General Report
-0.1
-0.05
0
0.05
0.1
Multicultural 4
Multicultural 7
Multicultural 5
Social/Behavioral 1
Multicultural 3
Multicultural 1
Mathematics 5
Mathematics 2
Natural Sciences 6
Humanities 5
Mathematics 3
Social/Behavioral 4
Social/Behavioral 10
Multicultural 2
Social/Behavioral 2
Humanities 6
Social/Behavioral 8
Natural Sciences 4
Natural Sciences 2
Natural Sciences 5
Mathematics 4
Mathematics 1
Multicultural 6
Natural Sciences 3
Social/Behavioral 3
Social/Behavioral 5
Natural Sciences 1
Humanities 4
Social/Behavioral 9
Social/Behavioral 7
Social/Behavioral 6
Humanities 7
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 9 of 43
Online Senior Assessment 2010: General Report
The table below summarizes how students performed overall on the OSA (i.e., the
percentage of correct answers). Only the 32 questions from Humanities, Multicultural,
Mathematics, Natural Sciences, and Social and Behavioral Sciences where one correct
answer exists (i.e., the knowledge questions) were included in this analysis. The selfassessment questions from Humanities and the agreement questions from Technology
and Applied Science were excluded since there is no right or wrong answer. The
average score is 65.4%. The lowest score is 28.1% and the highest score is 96.9%.
Overall Performance
(total percent correct)
N
755
Mean
65.4%
St. Deviation
13.2%
Minimum
28.1%
1st Quartile
56.3%
Median
65.6%
3rd Quartile
75.0%
Maximum
96.9%
The graph below shows the overall performance for all of the participants. It shows that
most students have average scores between 50% and 80%.
Overall Performance
(total percent correct)
30.0%
26.9%
23.6%
25.0%
19.7%
20.0%
15.0%
11.8%
11.8%
10.0%
3.4%
5.0%
0.0%
0.0%
2.4%
0.4%
0.0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 10 of 43
Online Senior Assessment 2010: General Report
Since each area has a different number of questions, the results above are influenced
more by those areas with more questions (e.g., Social and Behavioral Sciences and
Multicultural) and less by those with fewer questions (e.g., Humanities and
Mathematics). The table below provides information on how students performed when
the scores are averaged by area first. The average score is slightly higher (66.9%), the
lowest score is 25.9%, and the highest score is 97.1%.
Overall Performance
(averaged by area)
N
755
Mean
66.9%
St. Deviation
14.0%
Minimum
25.9%
1st Quartile
58.0%
Median
67.8%
3rd Quartile
77.6%
Maximum
97.1%
The results are almost the same when they are averaged by area first. The graph below
shows that the whole distribution seems very similar. It looks like a few more people
have higher scores when the results are displayed this way.
Overall Performance
(averaged by area)
30.0%
27.3%
23.8%
25.0%
20.0%
17.7%
16.6%
15.0%
10.0%
8.1%
5.0%
3.2%
0.0%
0.0%
2.5%
0.8%
0.0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 11 of 43
Online Senior Assessment 2010: General Report
The same results by area can be split up between people who took the respective
courses for core requirement at TTU and those who took it elsewhere (TTU vs. ELSE).
This could provide some information regarding if students who took their core courses
at Texas Tech perform better or worse on average than those who took it elsewhere.
Since it is possible for one and the same student to take one course for one area at
Texas Tech (e.g., Humanities) and another course for another area somewhere else
(e.g., Mathematics), the same student can be included in both groups. The TTU group
averages the scores of those areas where the core requirement was taken at Texas
Tech while the ELSE group averages the scores of those areas where the core
requirement was taken elsewhere. The chart below compares the percentage of correct
answers for TTU and ELSE. The scores seem to be similar for the two groups with a
few more TTU scores between 80% and 90% and a few more ELSE scores between
90% and 100%.
Overall Performance: TTU vs. ELSE
25.0%
20.0%
15.0%
TTU
10.0%
ELSE
5.0%
0.0%
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Running a 2-sample t-test (a statistical test often used to check for differences in means
of two different groups) shows that the means of the TTU group and the ELSE group
are not different from each other (not statistically significant at the 0.05 level). The mean
scores show that on average participants who took their core courses at TTU performed
similarly to those who took their core elsewhere (67.7% for TTU vs. 66.7% for ELSE).
See the information for “Overall” in the table below for the details.
Because participants in the TTU and ELSE group may have taken different classes from
TTU and different classes from elsewhere, a 2-sample t-test was also run comparing
only those participants who took all of their core curriculum classes from TTU and only
those participants who took all of their core curriculum classes elsewhere. This test
also found that on average participants who took all of their core curriculum classes
from TTU perform similarly to those who took all of their core curriculum classes
elsewhere. See the information for “Overall – all classes” in the table below for details.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 12 of 43
Online Senior Assessment 2010: General Report
A paired t-test was also run for only those participants who have taken some classes for
core requirements at Tech and some elsewhere (525 students). This allows us to
compare if the same participant performs any different depending on where the course
was taken (at TTU or elsewhere). Again, there appears to be no difference between the
TTU group and the ELSE group. See the information for “Overall – matched” in the
table below for details.
Core Area
Overall
Overall - all classes
Overall - matched
N
686
160
525
Core at TTU
Mean
SD
67.7%
16.5%
66.8%
14.3%
68.0%
17.1%
N
594
66
525
Core Elsewhere
Mean
SD
66.7%
19.1%
65.3%
15.1%
66.9%
19.7%
T-stat
0.255
0.709
1.126
P-value
0.799
0.479
0.261
The table below compares the differences between TTU and ELSE per core area. The
only core area that shows a significant difference in scores (at the 0.05 level) between
the two groups is the Multicultural core area. The TTU group performed significantly
better than the ELSE group with the Multicultural questions (76.8% correct vs. 71.9%
correct). This may suggest that on average taking a course for the Multicultural core
requirement at TTU (versus taking it elsewhere) helps the students to perform better on
the Multicultural questions.
Core Area
Humanities
Multicultural
Mathematics
Natural Sciences
Social/Behavioral
N
444
558
386
453
450
Core at TTU
Mean
SD
71.3%
28.1%
76.8%
17.6%
63.0%
27.5%
69.8%
20.4%
54.0%
16.7%
N
311
197
369
302
305
Core Elsewhere
Mean
SD
73.2%
25.7%
71.9%
18.8%
61.6%
28.7%
70.3%
21.9%
55.3%
16.8%
T-stat
-0.972
2.772
0.701
-0.354
-1.039
P-value
0.331
0.006
0.483
0.724
0.299
The following chart shows the mean percentage of questions answered correctly with a
95% confidence interval for each of the core areas from the above table, as well as the
average overall score. The mean is shown as the red line and the confidence interval is
depicted by the blue and green lines. If the confidence interval is small, it means that
the scores in this group were fairly close together. If the confidence interval is large, the
variation between the scores in this group was larger. For each area the data is shown
for TTU and ELSE. Social and Behavioral Sciences and Mathematics have the lowest
average scores while Multicultural and Humanities have some of the highest average
scores.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 13 of 43
Online Senior Assessment 2010: General Report
80.0%
75.0%
70.0%
65.0%
Lower 95
60.0%
Mean
Upper 95
55.0%
50.0%
The following table gives a summary of the above chart with information for each core
area section including each section’s position in the OSA, number of knowledge
questions, and the mean percentage of knowledge questions answered correctly for
TTU and ELSE along with the lower and upper 95% confidence interval limits. The
Technology and Applied science section is the fifth section of the OSA and it is
excluded from the table because it does not include any knowledge questions.
Position in
the OSA
Humanities TTU
Humanities ELSE
Multicultural TTU
Multicultural ELSE
Mathematics TTU
Mathematics ELSE
Natural Sciences TTU
Natural Sciences ELSE
Social/Behavioral TTU
Social/Behavioral ELSE
1st
1st
2nd
2nd
3rd
3rd
4th
4th
6th
6th
Number of
Knowledge
Questions
4
4
7
7
5
5
6
6
10
10
Lower Limit
95% C.I.
Mean
Upper Limit
95% C.I.
68.7%
70.4%
75.3%
70.7%
60.3%
58.6%
67.9%
67.8%
52.5%
53.4%
71.3%
73.2%
76.8%
71.9%
63.0%
61.6%
69.8%
70.3%
54.0%
55.3%
73.9%
76.1%
78.4%
75.3%
65.8%
64.5%
71.6%
72.8%
55.6%
57.2%
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 14 of 43
Online Senior Assessment 2010: General Report
The Social and Behavioral Sciences core area was the last area on the OSA and has
the lowest average scores. However, most of the other averages are fairly close
together. It might be that participants lost interest in the very last part of the OSA or that
the participants had more difficulties with the questions from the Social and Behavioral
Sciences than with the questions from the other sections.
The ELSE group can be further divided into the different course options for students
taking their core curriculum requirements outside of Texas Tech: a dual credit class, an
advanced placement class, a CLEP exam, or a class from another institution. The mean
scores of participants using each of these course options for each core area
requirement are compared in the individual core area reports. Within the ELSE group,
the participants who took an advanced placement class for their core requirement
consistently have the highest average score for each core area and the participants who
took a class at another institution consistently have the lowest average score for each
core area. This makes sense considering that students who earn core credit for their
advanced placement need to pass a test, suggesting they were successful in the class.
See the individual core area reports for more details.
Considering that within the ELSE group the participants who took an advanced
placement class for their core requirement consistently have the highest average score
for each core area, a 2-sample t-test was run comparing the TTU group and the overall
average of core area scores for which participants took an advanced placement class to
fulfill the core area requirement. The following table shows the results of this
comparison.
Core
Area
Overall
N
686
Core at TTU
Mean
SD
67.7%
16.5%
Core through Advanced Placement
N
Mean
SD
66
76.0%
21.6%
T-stat
3.033
P-value
0.003
It appears that on average students who take an advanced placement class for core
requirement do better on that core area section of the OSA than students who take a
class at TTU for that core requirement. This again makes sense when considering that
students need to pass an exam to receive advanced placement credit.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 15 of 43
Online Senior Assessment 2010: General Report
The individual core area reports also include comparisons of the participants with a
major in the core area and the participants without a major in the core area. These
comparisons are only included in the core area reports for which there were participants
who could clearly be classified as majors in that core area. The following table gives a
summary of these comparisons. Note that because of the small sample of participants
majoring in Mathematics, the Mathematics section of the OSA was compared using the
participants majoring in Mathematics as well as other majors that require extra
Mathematics courses (i.e., Engineering and Physics).
Humanities Overall
Mathematics Overall
Natural Sciences Overall
Social/Behavioral Overall
N
56
127
79
146
Majors
Mean
75.9%
78.1%
79.3%
53.6%
SD
27.0%
24.9%
20.0%
15.2%
N
699
628
676
609
Non-Majors
Mean
SD
71.8%
27.1%
59.1%
27.6%
68.9%
20.8%
54.8%
17.1%
T-stat
1.093
7.188
4.229
-0.789
P-value
0.275
< 0.001
< 0.001
0.430
Majors performed significantly better than the non-majors in the Natural Sciences and
Mathematics sections. Majors in the Humanities and Social & Behavioral Sciences did
not outperform their non-major counterparts on their sections on average. This could
either be a reflection of the majors or the questions themselves. Please see the
individual core area reports for more details.
The individual core area reports also include comparisons of the female and male
participants for each core area. The following table gives a summary of these
comparisons.
Humanities Overall
Multicultural Overall
Mathematics Overall
Natural Sciences Overall
Social/Behavioral Overall
N
441
441
441
441
441
Female
Mean
73.9%
75.2%
55.0%
68.3%
53.0%
SD
26.1%
17.7%
27.3%
20.1%
15.5%
N
312
312
312
312
312
Male
Mean
69.5%
76.1%
72.7%
72.2%
56.8%
SD
28.3%
18.5%
25.7%
22.1%
18.1%
T-stat
2.226
-0.684
-8.991
-2.513
-3.106
P-value
0.026
0.494
< 0.001
0.012
0.002
The Humanities section is the only core area for which the female participants
performed significantly better than the male participants at the 0.05 level. The male
participants performed significantly better in the Mathematics section, the Natural
Sciences section, and the Social and Behavioral Sciences section at the 0.05 level or
less. Male and female participants performed similarly on the Multicultural section on
average. See the individual core area reports for more details.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 16 of 43
Online Senior Assessment 2010: General Report
The table below shows the correlation between the overall performance for all the
knowledge questions from the OSA with some other variables of interest (p-values for
the correlations are in parenthesis).
Overall
Performance
Correlation
P-value
N
Time
-0.119
(0.001)
755
GPA
0.22
(<0.001)
755
SAT
0.571
(<0.001)
511
ACT
0.541
(<0.001)
388
Transfer
Credits
-0.113
(0.002)
755
Total
Credits
Age
0.161
0.007
(<0.001) (0.848)
755
755
The correlations with time to complete OSA, GPA, SAT score, ACT score, transfer
credits, and total credits earned are significant at the 0.05 level. This suggests that on
average students who take less time to complete the OSA, students with higher GPA’s,
students with higher SAT scores, students with higher ACT scores, students with less
transfer credits, and students with more total credits earned do better on the OSA. The
only variable without a significant correlation is age.
The individual core area reports include regression models for the overall scores for
each core area including the variables discussed above (class taken at TTU, major, sex,
time to complete OSA, GPA, SAT score, ACT score, transfer credits, total credits
earned, and age).The two variables that are consistently found in those models to be
significant predictors of performance for each core area are SAT score and ACT score.
These models suggest that on average students with higher SAT scores and higher
ACT scores do better on each core area section of the OSA. This might suggest that
performance on the OSA is related to general test-taking ability or previous knowledge.
See the individual core area reports for more details from the regression models.
The following tables show the results of regression models for the overall average for
the OSA including time to complete OSA, sex, GPA, transfer credits, total credits
earned, and age as explanatory variables. There are three separate regression models
because not all of the participants have a SAT score and ACT score. Since most
students either have a SAT score or an ACT score, there would be too many missing
values if both scores were included in the same regression model. The first model
excludes both in order to include most respondents in the analysis.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 17 of 43
Online Senior Assessment 2010: General Report
N
753
Variable
Intercept
Time
Sex
GPA
Transfer Credits
Total Credits
Age
Overall Model 1
F
P-value
17.60
< 0.001
Coefficient P-value
0.429 < 0.001
-0.000027
0.002
-0.049 < 0.001
0.060 < 0.001
-0.000016
0.955
0.00070
0.003
0.00086
0.362
This first model excludes SAT and ACT scores to include 753 of the 755 participants.
The model overall is significant at the 0.05 level (R2 = 0.1240). For this model, time to
complete OSA, sex, GPA, and total credits earned are significant predictors at the 0.05
level for the overall average. These predictors suggest that on average students who
take less time to complete the OSA, male students, students with higher GPA’s, and
students with more total credits earned do better on the OSA when the other variables
in the model are held constant.
Overall Model 2
N
511
Variable
Intercept
Time
Sex
GPA
Transfer Credits
Total Credits
Age
SAT
F
P-value
45.89
< 0.001
Coefficient P-value
-0.090
0.123
-0.000020
0.049
-0.036 < 0.001
0.027
0.004
0.00010
0.728
0.00022
0.370
0.0074 < 0.001
0.00047 < 0.001
This second model includes SAT score and excludes ACT score to include 511 of the
755 participants. The model overall is significant at the 0.05 level (R2 = 0.3897). For
this model time to complete OSA, sex, GPA, age, and SAT score are significant
predictors at the 0.05 level for the overall average. These predictors suggest that on
average students who take less time to complete the OSA, male students, students with
higher GPA’s, older students, and students with higher SAT scores do better on the
OSA when the other variables in the model are held constant.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 18 of 43
Online Senior Assessment 2010: General Report
Overall Model 3
N
387
Variable
Intercept
Time
Sex
GPA
Transfer Credits
Total Credits
Age
ACT
F
P-value
31.13
< 0.001
Coefficient P-value
0.067
0.326
-0.000027
0.357
-0.045 < 0.001
0.013
0.271
-0.000066
0.848
0.00014
0.626
0.0056
0.003
0.019 < 0.001
This third model includes ACT score and excludes SAT score to include 387 of the 755
participants. The model overall is significant at the 0.05 level (R2 = 0.3650). For this
model sex, age, and ACT score are significant predictors at the 0.05 level for the overall
average. These predictors suggest that on average male students, older students, and
students with higher ACT scores do better on the OSA when the other variables in the
model are held constant.
LIMITATIONS
The first question in each core area asks students where they took the respective core
course. If they selected anything other than TTU, they could select multiple options. If
they selected TTU, they could only select TTU and nothing else. This assumes that if
students take classes outside of TTU, they can take multiple classes and that if they
take classes at TTU, they can only take one course. This might not necessarily be the
case.
Most students spent between 15 and 45 minutes on the OSA. It is possible that
participants pay more attention towards the beginning than towards the end which might
lead to lower area scores for the core areas who are toward the end. If the sample is
large enough in future years, there could maybe be two versions of the OSA with a
different order of the sections to explore this issue.
A potential limitation is that there are only a few questions per core area and those do
not necessarily represent each subject a student could have taken as part of the core
curriculum. For example, the questions in Natural Sciences seem focused on specific
areas of the natural sciences. A student could have simply taken other classes that are
part of Natural Sciences and performed badly on this assessment.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 19 of 43
Online Senior Assessment 2010: General Report
Another limitation is that students do not necessarily have an incentive to do well on the
assessment. To help encourage students to put forth effort, students were instructed
that they would not be included in the drawing for the $1,000 if they did not complete the
assessment and if it was apparent that they did not put forth an honest effort. This
meant that all of the students who completed the assessment in less than 15 minutes
were excluded from the drawing and the analysis. However, students still could have
taken longer than 15 minutes without trying to do well.
The Technology and Applied Science section of the OSA includes only agreement
statements. While these might give a broader idea of what a student knows, it might be
beneficial to include some knowledge questions in the future.
Another limitation is that the ELSE group does not represent a homogenous group for
comparison with the TTU group. The ELSE group includes students who completed
their core requirements through several different course options. Questions were added
to help clarify where the ELSE group completed each core requirement (i.e., dual credit
course, advanced placement course, CLEP exam, or at another institution).
Comparison of the different subgroups within the ELSE group showed that it is not a
homogeneous group.
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 20 of 43
Online Senior Assessment 2010: General Report
APPENDIX
Attachment A: The Instrument
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 21 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 22 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 23 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 24 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 25 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 26 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 27 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 28 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 29 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 30 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 31 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 32 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 33 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 34 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 35 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 36 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 37 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 38 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 39 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 40 of 43
Online Senior Assessment 2010: General Report
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 41 of 43
Online Senior Assessment 2010: General Report
Attachment B: Chi-Squared Tests for Knowledge Questions
TTU
(N=444)
correct incorrect
290
154
218
226
395
49
363
81
Else
(N=289)
correct incorrect
218
93
145
166
274
37
274
37
Chi Statistic
1.90
0.45
0.13
5.59
Chi Probability
0.1682
0.5029
0.7140
0.0181
Multicultural 1
Multicultural 2
Multicultural 3
Multicultural 4
Multicultural 5
Multicultural 6
Multicultural 7
TTU
(N=500)
correct incorrect
401
157
524
34
536
22
261
297
431
127
414
144
433
125
Else
(N=232)
correct incorrect
134
63
183
14
179
18
74
123
138
59
149
48
135
62
Chi Statistic
1.04
0.25
7.83
5.00
4.05
0.16
6.43
Chi Probability
0.3074
0.6163
0.0051
0.0253
0.0441
0.6897
0.0112
Mathematics 1
Mathematics 2
Mathematics 3
Mathematics 4
Mathematics 5
TTU
(N=386)
correct incorrect
206
180
285
101
202
184
328
58
195
191
Else
(N=348)
correct incorrect
200
169
263
106
185
184
314
55
174
195
Chi Statistic
0.05
0.62
0.36
0.00
0.85
Chi Probability
0.8186
0.4305
0.5462
0.9629
0.3554
Natural Science 1
Natural Science 2
Natural Science 3
Natural Science 4
Natural Science 5
Natural Science 6
TTU
(N=452)
correct incorrect
200
252
210
242
331
121
380
72
360
92
411
41
Else
(N=288)
correct incorrect
148
155
140
163
228
75
253
50
241
62
268
35
Chi Statistic
1.54
0.00
0.38
0.04
0.00
1.23
Chi Probability
0.2142
0.9450
0.5354
0.8341
0.9712
0.2669
Humanities 4
Humanities 5
Humanities 6
Humanities 7
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 42 of 43
Online Senior Assessment 2010: General Report
Social/Behavioral 1
Social/Behavioral 2
Social/Behavioral 3
Social/Behavioral 4
Social/Behavioral 5
Social/Behavioral 6
Social/Behavioral 7
Social/Behavioral 8
Social/Behavioral 9
Social/Behavioral 10
TTU
(N=450)
correct incorrect
330
120
336
114
180
270
99
351
158
292
246
204
164
286
213
237
440
10
265
185
Else
(N=295)
correct incorrect
203
102
223
82
119
186
76
229
101
204
178
127
130
175
163
142
296
9
198
107
Chi Statistic
4.02
0.23
0.07
0.87
0.32
1.01
2.92
2.71
0.39
2.79
Chi Probability
0.0449
0.6332
0.7863
0.3511
0.5707
0.3155
0.0876
0.0995
0.5305
0.0951
Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, June 2010
Page 43 of 43