Online Senior Assessment 2010: Natural Sciences Introduction The fourth component of the OSA contains questions for the Natural Sciences core. The first questions ask the participants about how they fulfilled their Natural Sciences core requirement. The following table shows the number and percentage of participants who selected each response to the first question regarding where students took their core curriculum course. The number of participants selecting each response adds up to more than the 755 total participants because those who did not select “I took all my core curriculum classes in natural science at Tech” could select more than one of the other responses. How did you complete your core curriculum requirement in Natural Science? % of all % of all Response N Responses Participants I took at least one natural science core curriculum 25 3.2% 3.3% class through dual credit in high school. I took at least one advanced placement natural 43 5.5% 5.7% science core curriculum class in high school. Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 1 of 20 Online Senior Assessment 2010: Natural Sciences How did you complete your core curriculum requirement in Natural Science? (Cont.) I took at least one CLEP exam for natural science 7 0.9% 0.9% core curriculum credit. I received transfer core curriculum natural science credit for at least one class that I took at another 247 31.9% 32.7% institution. I took all my core curriculum classes in natural 453 58.5% 60.0% science at Tech. Total Responses 775 100.0% For the analysis in this report the 755 participants are divided into the “TTU” group and the “ELSE” group. The TTU group represents the 453 participants (i.e., 60.0%) who selected “I took all my core curriculum classes in natural science at Tech” and the ELSE group represents the 302 participants (i.e., 40.0%) who selected one or more of the other responses indicating that they took their core curriculum class in the Natural Sciences elsewhere. The following pie chart shows this division of the sample. Natural Sciences 302 (40.0%) TTU 453 (60.0%) ELSE The 302 participants in the ELSE group were also asked if the class they took outside of Tech counted for their core curriculum credit. Of the 302 participants who reported taking a Natural Sciences course elsewhere, 14 (i.e., 4.6%) reported that they did not know if the course counted for their Natural Sciences core curriculum credit and 288 (i.e., 95.4%) reported that the course did count for their Natural Sciences core curriculum credit. The 288 participants who reported that the course taken outside of Tech did count for their Natural Sciences core curriculum credit were also asked which one counted. The following table shows the number and percentage of the 288 participants who selected each response. Which one? Response A dual credit class. An advanced placement class. A CLEP exam. A class I took at another institution. I don't know. Total N % 22 7.6% 29 10.1% 3 1.0% 230 79.9% 4 1.4% 288 100.0% Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 2 of 20 Online Senior Assessment 2010: Natural Sciences The system stores some data for each of the participants and so it was possible to identify the respondents who major in Natural Sciences. The following majors were identified and classified as Natural Sciences majors: Animal Sciences, Biology, Geosciences, Horticultural Turf Grass Science, Microbiology, Physics, and Zoology. There were a total of 79 Natural Science majors in the sample. These participants represent the so-called “experts”. Natural Sciences Majors Major Frequency Percentage Animal Sciences 6 7.6% Biology 31 39.2% Chemistry 9 11.4% Biochemistry 16 20.3% Geosciences 4 5.1% Horticultural Turf Grass Science 2 2.5% Microbiology 6 7.6% Physics 2 2.5% Zoology 3 3.8% Total 79 100.0% Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 3 of 20 Online Senior Assessment 2010: Natural Sciences Results The student learning outcomes for the Natural Sciences are: Demonstrate knowledge of the scientific method and to contrast it with other ways of understanding the world. Demonstrate knowledge of the tools and methods used by scientists to study the natural world. Explain some of the major theories in the Natural Sciences. Describe how Natural Sciences research informs societal issues, including ethics. The first learning outcome aligns with the fourth, fifth, and sixth question. The second learning outcome aligns with the fourth and fifth question. The third learning outcome aligns with the first, second, and third question. The fourth student learning outcome aligns with the second question. The Natural Sciences portion of the assessment contains six knowledge questions. These are shown below as a screenshot from the actual instrument. For analysis purposes, the answers were coded from 1 to 5 (for the questions with 5 answers) or 1 to 4 (for the questions with 4 answers) in the order they appear on the actual instrument. Attachment D shows how many times each answer choice was selected by the different participants for all of the Natural Sciences questions. Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 4 of 20 Online Senior Assessment 2010: Natural Sciences Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 5 of 20 Online Senior Assessment 2010: Natural Sciences Natural Sciences 1: The chart below shows the distributions of answers for the first question for participants who took their class for the Natural Sciences core requirement at TTU (blue) and participants who took their class for the Natural Sciences core requirement elsewhere (red). Answer 4 is the correct choice. It can be seen that almost half of both groups chose the correct answer. The low percentage of correct responses for this question could be because this question requires specific area knowledge that would not be covered in some natural sciences courses. Overall, a few more people in the ELSE group chose the correct answer as compared to the TTU group (49.0% vs. 44.2%). This difference is not statistically significant at the 0.05 level (see attachment A for details). This means that on average students who take their class for the Natural Sciences core requirement elsewhere do not do significantly better with this question than the students who take their Natural Sciences class at TTU. Since the first question aligns with the third learning outcome, this suggests that on average students who take their Natural Sciences course elsewhere meet this learning outcome similar to students who take their course at TTU. Natural Sciences 1 60.0% 49.0% 44.2% 50.0% 40.0% 30.0% 20.0% TTU 20.8%20.9% 20.1% 16.9% ELSE 10.4%11.3% 10.0% 4.6% 2.0% 0.0% 1 2 3 4 5 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 6 of 20 Online Senior Assessment 2010: Natural Sciences Natural Sciences 2: The chart below shows the distributions of answers for the second question for participants who took their class for the Natural Sciences core requirement at TTU (blue) and participants who took their class for the Natural Sciences core requirement elsewhere (red). Answer 4 is again the correct choice. It can be seen that about half of both groups chose the correct answer. The fact that so many people chose the same wrong answer (i.e., answer 1) might indicate a lack of understanding in this area or that the answer choice was confusing. The low percentage of correct responses for this question could be because this question requires specific area knowledge that would not be covered in some natural sciences courses. Overall, a few more people in the TTU group chose the correct answer as compared to the ELSE group (46.6% vs. 46.0%). This difference is not statistically significant at the 0.05 level (see attachment A for details). This means that on average students who take their class for the Natural Sciences core requirement at TTU do not do significantly better with this question than the students who take their Natural Sciences class elsewhere. Since the second question aligns with the third and fourth learning outcomes, this suggests that on average students who take their Natural Sciences course at TTU meet these learning outcomes similar to students who take their course elsewhere. Natural Sciences 2 50.0% 46.6%46.0% 45.0% 40.0% 35.0% 35.3%34.1% 30.0% 25.0% TTU 20.0% ELSE 14.6% 12.8% 15.0% 10.0% 4.0% 3.6% 5.0% 1.3% 1.7% 0.0% 1 2 3 4 5 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 7 of 20 Online Senior Assessment 2010: Natural Sciences Natural Sciences 3: The chart below shows the distributions of answers for the third question for participants who took their class for the Natural Sciences core requirement at TTU (blue) and participants who took their class for the Natural Sciences core requirement elsewhere (red). Answer 1 is the correct choice. It can be seen that a majority of both groups chose the correct answer, with about one fourth of both groups choosing another answer. This question also requires specific area knowledge that would not be covered in some natural sciences courses. Overall, a few more people in the ELSE group chose the correct answer as compared to the TTU group (75.2% vs. 73.3%). This difference is not statistically significant at the 0.05 level (see attachment A for details). This means that on average students who take their class for the Natural Sciences core requirement elsewhere do not do significantly better with this question than the students who take their Natural Sciences class at TTU. Since the third question aligns with the third learning outcome, this suggests that on average students who take their Natural Sciences course elsewhere meet this learning outcome similar to students who take their course at TTU. Natural Sciences 3 80.0% 73.3% 75.2% 70.0% 60.0% 50.0% 40.0% TTU 30.0% ELSE 20.0% 13.7% 14.9% 10.4% 10.0% 7.6% 2.6% 2.3% 0.0% 1 2 3 4 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 8 of 20 Online Senior Assessment 2010: Natural Sciences Natural Sciences 4: The chart below shows the distributions of answers for the fourth question for participants who took their class for the Natural Sciences core requirement at TTU (blue) and participants who took their class for the Natural Sciences core requirement elsewhere (red). Answer 4 is the correct choice. It can be seen that a large majority of both groups chose the correct answer, which may mean that this question is not a good discriminator of knowledge in the Natural Sciences. Overall, a few more people in the TTU group chose the correct answer as compared to the ELSE group (84.1% vs. 83.4%). This difference is not statistically significant at the 0.05 level (see attachment A for details). This means that on average students who take their class for the Natural Sciences core requirement at TTU do not do significantly better with this question than the students who take their Natural Sciences elsewhere. Since the fourth question aligns with the first and second learning outcomes, this suggests that on average students who take their Natural Sciences course at TTU meet these learning outcomes similar to students who take their course elsewhere. Natural Sciences 4 90.0% 84.1% 83.4% 80.0% 70.0% 60.0% 50.0% TTU 40.0% ELSE 30.0% 20.0% 10.0% 2.9% 2.6% 8.6% 7.6% 4.4% 6.3% 2 3 0.0% 1 4 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 9 of 20 Online Senior Assessment 2010: Natural Sciences Natural Sciences 5: The chart below shows the distributions of answers for the fifth question for participants who took their class for the Natural Sciences core requirement at TTU (blue) and participants who took their class for the Natural Sciences core requirement elsewhere (red). Answer 2 is the correct choice. It can be seen that a majority of both groups chose the correct answer. Overall, more people in the TTU group chose the correct answer as compared to the ELSE group (79.7% vs. 79.5%). This difference is not statistically significant at the 0.05 level (see attachment A for details). This means that on average students who take their class for the Natural Sciences core requirement at TTU do not do significantly better with this question than the students who take their Natural Sciences class elsewhere. Since the fifth question aligns with the first and second learning outcomes, this suggests that on average students who take their Natural Sciences course at TTU meet these learning outcomes similar to students who take their course elsewhere. Natural Sciences 5 90.0% 79.7% 79.5% 80.0% 70.0% 60.0% 50.0% TTU 40.0% ELSE 30.0% 20.0% 18.5% 17.5% 10.0% 0.9% 2.0% 0.9% 1.0% 3 4 0.0% 1 2 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 10 of 20 Online Senior Assessment 2010: Natural Sciences Natural Sciences 6: The chart below shows the distributions of answers for the sixth question for participants who took their class for the Natural Sciences core requirement at TTU (blue) and participants who took their class for the Natural Sciences core requirement elsewhere (red). Answer 2 is the correct choice. It can be seen that a large majority of both groups chose the correct answer, which may mean that this question is not a good discriminator of knowledge in the Natural Sciences. Overall, a few more people in the TTU group chose the correct answer as compared to the ELSE group (90.7% vs. 88.7%). This difference is not statistically significant at the 0.05 level (see attachment A for details). This means that on average students who take their class for the Natural Sciences core requirement at TTU do not do significantly better with this question than the students who take their Natural Sciences class elsewhere. Since the sixth question aligns with the first learning outcome, this suggests that on average students who take their Natural Sciences course at TTU meet this learning outcome similar to students who take their course elsewhere. Natural Sciences 6 100.0% 90.7% 88.7% 90.0% 80.0% 70.0% 60.0% 50.0% TTU 40.0% ELSE 30.0% 20.0% 10.0% 5.1% 6.3% 3.3% 3.6% 0.9% 1.3% 3 4 0.0% 1 2 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 11 of 20 Online Senior Assessment 2010: Natural Sciences Natural Sciences Average: The table below compares the differences between TTU and ELSE when the results for all the questions are averaged (e.g., if a student got 3 out of the 6 questions correct, his score will be 3/6 = .50). The mean is almost the same, but slightly higher for students who took their core requirement for the Natural Sciences elsewhere. This difference is not statistically significant at the 0.05 level. This means that on average TTU performs about the same as ELSE in the Natural Sciences. Core at TTU Mean SD N Natural Sciences Overall 453 69.8% Core Elsewhere Mean SD N 20.4% 302 70.3% T-stat 21.9% -0.354 P-value 0.724 The chart below shows the distributions of the average scores for participants who took their class for the Natural Sciences core requirement at TTU (blue) and participants who took their class for the Natural Sciences core requirement elsewhere (red). The distributions are similar, but it looks like the ELSE group had more people with 6 correct answers and that the TTU group had more people with 4 correct answers. Natural Sciences Overall: TTU vs. ELSE 40.0% 35.0% 31.1% 30.0% 33.6% 31.8% 26.8% 25.0% 20.0% 15.6% 14.6% 15.0% 11.3% 10.0% 5.0% 15.6% 1.1%0.3% 4.3% 1.8% 6.6%5.6% 0.0% 0% 16.7% 33.3% 50.0% 66.7% 83.3% 100% Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 12 of 20 TTU ELSE Online Senior Assessment 2010: Natural Sciences The table below shows a comparison of the average scores for the participants selecting each course option within the ELSE group (those selecting more than one course option were excluded from the analysis). The table includes the F value and P value for an analysis of variance comparing the means. Although the table shows information for all four ELSE options, the group with less than 15 participants (CLEP Exam) was excluded from the analysis of variance. Elsewhere Dual Credit Advanced Placement CLEP Exam Another Institution Total N 16 33 4 236 289 Mean 70.8% 79.3% 75.0% 68.7% 70.1% St. Dev. 22.4% 18.6% 9.6% 22.3% 22.0% F-value 3.391 P-value 0.035 Based on the mean, students who took their Natural Sciences course through advanced placement are the highest-performing group and students who took their course at another institution are the lowest performing group. The means are significantly different at the 0.05 level though. This suggests that on average for the students who take their Natural Sciences course elsewhere, which course option they use to take their Natural Sciences course makes a difference in how they perform on the Natural Sciences section of the OSA. Tukey’s method for multiple comparisons was used to find which course option means are significantly different. The following table shows the significant differences at the 0.05 level. Comparison Advanced Placement vs. Another Institution P-value 0.026 The table shows the one difference that was significant at the 0.05 level. This difference suggests that on average students who take their Natural Sciences course through advanced placement do better on the Natural Sciences section of the OSA than students who take their Natural Sciences course at another institution. This outcome makes sense when considering that students who take an AP course need to pass an exam to receive credit, which suggests that they performed well in their AP class. Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 13 of 20 Online Senior Assessment 2010: Natural Sciences The chart below shows the distributions of the average scores for those participants who are majoring in the Natural Sciences (blue) and participants who are not majoring in the Natural Sciences (red). A larger percentage of the Natural Sciences majors answered most or all of the questions correctly. Overall, the majors have a higher average than the non-majors (79.3% vs. 68.9%). This difference is statistically significant at the .05 level (see attachment B for details). This suggests that Natural Science majors perform better on average than non-majors on the Natural Sciences portion of the assessment. Natural Sciences Overall: Majors vs. Non-majors 45.0% 41.8% 40.0% 35.0% 30.8% 31.8% 27.8% 30.0% 25.0% 20.0% 15.8% Majors 17.7% Non-Majors 15.0% 11.2% 6.8% 7.6% 10.0% 5.0% 0.0%0.9% 3.8%2.7% 1.3% 0.0% 0% 16.7% 33.3% 50.0% 66.7% 83.3% 100% Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 14 of 20 Online Senior Assessment 2010: Natural Sciences The chart below shows the distribution of the average scores for the female participants (blue) and the male participants (red). The male participants had higher overall scores than the female participants (mean of 72.2% vs. 68.3%). This is significant at the 0.05 level (see attachment C for details). This suggests that male students do better on average than female students with the Natural Sciences section of the OSA. It appears in the chart below that the significant difference in average scores for the female and male participants is due to more male participants getting 6 out of 6 questions correct and more female participants getting 3 or 4 out of 6 questions correct. Natural Sciences Overall by Sex 33.1% 35.0% 32.7%32.7% 30.0% 24.4% 25.0% 19.2% 20.0% 15.9% 13.8% 15.0% Female Male 10.0% 8.6% 5.9% 6.7% 5.0% 0.7% 1.0% 3.2% 2.2% 0.0% 0% 16.7% 33.3% 50.0% 66.7% 83.3% 100% Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 15 of 20 Online Senior Assessment 2010: Natural Sciences The following table shows the correlations between the overall average for the Natural Sciences questions and time to complete OSA, GPA, SAT score, ACT score, transfer credits, total credits earned, and age (p-values for the correlations are in parentheses). The correlations with GPA, SAT, ACT, and total credits earned are significant at the 0.05 level. This suggests that students with higher GPA’s, higher SAT scores, higher SAT scores, and more total credits earned perform better on average with the Natural Sciences section of the OSA. Some of these correlations, though, are small and more likely to be found statistically significant because of the large sample size. Natural Sciences Overall Transfer Total Time GPA SAT ACT Credits Credits Age Correlation -0.065 0.143 0.334 0.380 -0.050 0.115 0.067 P-value (0.074) (<0.001) (<0.001) (<0.001) (0.171) (0.002) (0.064) N 755 755 511 388 755 755 755 The following tables show the results of regression models for the overall average for the Natural Sciences questions including all of the variables that have been explored in this analysis. There are three separate regression models because not all of the participants have an SAT score and ACT score. Since most students either have an SAT score or an ACT score, there would be too many missing values if both scores were included in the same regression model. The first model excludes both in order to include most respondents in the analysis. Natural Sciences Model 1 N F P-value 753 8.59 < 0.001 Variable Coefficient P-value Intercept 0.381 < 0.001 Time -0.000023 0.090 Sex -0.043 0.004 GPA 0.059 < 0.001 Transfer Credits -0.000018 0.967 Total Credits 0.00096 0.007 Age 0.0038 0.009 Natural Sciences Major 0.097 < 0.001 Natural Sciences class taken at TTU -0.0370 0.040 This first model excludes SAT and ACT score to include 753 of the 755 participants. The model overall is significant at the 0.05 level (R2 = 0.0846). For this model sex, GPA, total credits earned, age, and Natural Sciences major are the significant predictors at the 0.05 level for the overall average for the Natural Sciences questions. These predictors suggest that on average male students, students with higher GPA’s, students with more total credits earned, older students, and students majoring in the Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 16 of 20 Online Senior Assessment 2010: Natural Sciences Natural Sciences do better on the Natural Sciences section of the OSA when the other variables in the model are held constant. Natural Sciences Model 2 N F P-value 511 11.38 < 0.001 Variable Coefficient P-value Intercept 0.017 0.862 Time -0.0000086 0.620 Sex -0.035 0.042 GPA 0.035 0.027 Transfer Credits -0.000042 0.939 Total Credits 0.00037 0.380 Age 0.0082 0.001 Natural Sciences Major 0.089 < 0.001 Natural Sciences class taken at TTU -0.029 0.157 SAT 0.00036 < 0.001 This second model includes SAT score and excludes ACT score to include 511 of the 755 participants. The model overall is significant at the 0.05 level (R2 = 0.1698). For this model sex, GPA, age, Natural Sciences major, and SAT score are the significant predictors at the 0.05 level for the overall average for the Natural Sciences questions. These predictors suggest that on average male students, students with higher GPA’s, older students, students majoring in the Natural Sciences, and students with higher SAT scores do better on the Natural Sciences section of the OSA when the other variables in the model are held constant. Natural Sciences Model 3 N F P-value 387 11.11 < 0.001 Variable Coefficient P-value Intercept 0.021 0.856 Time 0.000019 0.696 Sex -0.059 0.003 GPA 0.034 0.088 Transfer Credits 0.00030 0.619 Total Credits 0.00036 0.449 Age 0.0070 0.024 Natural Sciences Major 0.084 0.008 Natural Sciences class taken at TTU -0.030 0.209 ACT 0.017 < 0.001 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 17 of 20 Online Senior Assessment 2010: Natural Sciences This third model includes ACT score and excludes SAT score to include 387 of the 755 participants. The model overall is significant at the 0.05 level (R2 = 0.2097). For this model sex, age, Natural Sciences major, and ACT score are the significant predictors at the 0.05 level for the overall average for the Natural Sciences questions. These predictors suggests that on average male students, older students, students majoring in the Natural Sciences, and students with higher ACT scores do better on the Natural Sciences section of the OSA when the other variables in the model are held constant. Limitations It is difficult to assess the breadth of knowledge found in the Natural Sciences with six questions. Two of the questions (questions 4 and 6) may have been too simple to discriminate between those with a greater understanding of the Natural Sciences and those with a lesser understanding of the Natural Sciences. Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 18 of 20 Online Senior Assessment 2009: Natural Sciences Attachments Attachment A: Summary of Chi-Square Tests for Questions 1 - 6 Natural Science 1 Natural Science 2 Natural Science 3 Natural Science 4 Natural Science 5 Natural Science 6 TTU (N=452) correct incorrect 200 253 210 243 331 122 380 73 360 93 411 42 Else (N=288) correct incorrect 148 154 140 162 228 74 253 49 241 61 268 34 Chi Statistic 1.72 0.00 0.56 0.00 0.01 0.79 Chi Probability 0.1897 1.0000 0.4559 0.9678 0.9119 0.3741 Attachment B: 2-Sample T-Test for Average Scores of Majors and Non-majors N Natural Sciences Overall 79 Majors Mean SD 79.3% N 20.0% 676 Non-Majors Mean SD 68.9% T-stat 20.8% 4.229 P-value < 0.001 Attachment C: 2-Sample T-Test for Average Scores by Sex N Natural Sciences Overall 441 Female Mean SD 68.3% N 20.1% 312 Male Mean SD 72.2% T-stat 22.1% -2.513 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 19 of 20 P-value 0.012 Online Senior Assessment 2009: Natural Sciences Attachment D: Number of Participants Selecting Each Answer for Each Question Natural Sciences 1 Answer TTU ELSE 1 91 51 2 94 63 3 47 34 4 200 148 5 21 6 Natural Sciences 4 Answer TTU ELSE 1 13 8 2 39 23 3 20 19 4 381 252 Natural Sciences 2 Answer TTU ELSE 1 160 103 2 18 11 3 58 44 4 211 139 5 6 5 Natural Sciences 5 Answer TTU ELSE 1 84 53 2 361 240 3 4 6 4 4 3 Natural Sciences 3 Answer TTU ELSE 1 332 227 2 62 45 3 47 23 4 12 7 Natural Sciences 6 Answer TTU ELSE 1 23 19 2 411 268 3 15 11 4 4 4 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 20 of 20