Online Senior Assessment 2010: Multicultural Introduction The second component of the OSA contains questions for the Multicultural core. The first questions ask the participants about how they fulfilled their Multicultural core requirement. The following table shows the number and percentage of participants who selected each response to the first question regarding where students took their core curriculum course. The number of participants selecting each response adds up to more than the 755 total participants because those who did not select “I took my core curriculum class at Tech” could select more than one of the other responses. How did you complete your multicultural requirement? % of all % of all Response N Responses Participants I took a multicultural class through dual credit in 20 2.4% 2.6% high school. I took an advanced placement multicultural class 12 1.5% 1.6% in high school. 1 0.1% 0.1% I took a CLEP exam for multicultural credit. Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 1 of 19 Online Senior Assessment 2010: Multicultural How did you complete your multicultural requirement? (Cont.) I received transfer multicultural credit for a class 172 20.8% that I took at another institution. 558 67.6% I took my core curriculum class at Tech. I received multicultural credit for having done a study abroad course (either through Texas Tech 63 7.6% University or elsewhere). 826 100.0% Total Responses 22.8% 73.9% 8.3% For the analysis in this report the 755 participants will be divided into the “TTU” group and the “ELSE” group. The TTU group will represent the 558 participants (i.e., 73.9%) who selected “I took my core curriculum class at Tech” and the ELSE group will represent the 197 participants (i.e., 26.1%) who selected one or more of the other responses indicating that they took their multicultural core curriculum class elsewhere. The following pie chart shows this division of the sample. Multicultural 197 (26.1%) TTU ELSE 558 (73.9%) The 197 participants in the ELSE group were also asked if the class they took outside of Tech counted for their core curriculum credit. Of the 197 participants who reported taking a Multicultural course elsewhere, 23 (i.e., 11.7%) reported that they did not know if the course counted for their Multicultural core curriculum credit and 174 (i.e., 88.3%) reported that the course did count for their Multicultural core curriculum credit. The 174 participants who reported the course taken outside of Tech did count for their Multicultural core curriculum credit were also asked which one counted. The following table shows the number and percentage of the 174 participants who selected each response. Which one? Response A dual credit class. An advanced placement class. A CLEP exam. A class I took at another institution. I don't know. Total N % 11 6.3% 10 5.7% 1 0.6% 147 84.5% 5 2.9% 174 100.0% Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 2 of 19 Online Senior Assessment 2010: Multicultural Results The student learning outcomes for the Multicultural core are: Demonstrate awareness and knowledge of cultural differences within one or more distinctive sub-cultures of the United States, or Demonstrate awareness and knowledge of cultural differences within one or more global societies (Outside the U.S). Both student learning outcomes apply to all 7 questions on the OSA. They are all knowledge-type questions where one correct answer exists. A screenshot of the questions is shown below. For analysis purposes, the answers were coded from 1 to 4 in the order they appear on the actual instrument. Attachment C shows how many times each answer choice was selected by the different participants for all of the Multicultural questions. Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 3 of 19 Online Senior Assessment 2010: Multicultural Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 4 of 19 Online Senior Assessment 2010: Multicultural Multicultural 1: The chart below shows the distributions of answers to the first question for participants who took their class for the Multicultural core requirement at TTU (blue) and participants who took their class for the Multicultural core requirement elsewhere (red). Answer 2 is the correct choice. It can be seen that a majority in both groups chose the correct answer. However, there are also quite a few participants in both groups who chose one of the other answers. Overall, a few more people in the TTU group chose the correct answer as compared to the ELSE group (71.9% vs. 68.0%). This difference is not statistically significant at the 0.05 level (see attachment A for details). This means that on average students who take their class for the Multicultural core requirement at TTU do not do better with this question on average than students who take their Multicultural class elsewhere. Multicultural 1 80.0% 71.9% 70.0% 68.0% 60.0% 50.0% 40.0% TTU 30.0% ELSE 24.4% 17.4% 20.0% 10.6% 10.0% 7.6% 0.2% 0.0% 0.0% 1 2 3 4 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 5 of 19 Online Senior Assessment 2010: Multicultural Multicultural 2: The chart below shows the distributions of answers to the second question for participants who took their class for the Multicultural core requirement at TTU (blue) and participants who took their class for the Multicultural core requirement elsewhere (red). Answer 3 is the correct choice. It can be seen that a large majority in both groups chose the correct answer. There are only a few participants in both groups who chose one of the other answers, which may mean that this question is not a good discriminator of Multicultural understanding. A few more people in the TTU group chose the correct answer as compared to the ELSE group (93.9% vs. 92.9%). This difference is not statistically significant at the 0.05 level (see attachment A for details). This means that on average students who take their class for the Multicultural core requirement at TTU do not do better with this question than students who take their Multicultural class elsewhere. Multicultural 2 100.0% 93.9% 92.9% 90.0% 80.0% 70.0% 60.0% 50.0% TTU 40.0% ELSE 30.0% 20.0% 10.0% 0.2% 0.0% 0.2% 1.0% 1 2 5.7% 6.1% 0.0% 3 4 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 6 of 19 Online Senior Assessment 2010: Multicultural Multicultural 3: The chart below shows the distributions of answers to the third question for participants who took their class for the Multicultural core requirement at TTU (blue) and participants who took their class for the Multicultural core requirement elsewhere (red). Answer 1 is the correct choice. It can be seen that a large majority in both groups chose the correct answer. There are only a few participants in both groups who chose one of the other answers, which may mean that this question is not a good discriminator of Multicultural understanding. Overall, a few more people in the TTU group chose the correct answer as compared to the ELSE group (96.1% vs. 90.9%). This difference is statistically significant at the 0.05 level (see attachment A for details). This suggests that on average students who take their class for the Multicultural core requirement at TTU do better with this question than students who take their Multicultural class elsewhere. Multicultural 3 120.0% 100.0% 96.1% 90.9% 80.0% 60.0% TTU ELSE 40.0% 20.0% 1.6% 3.0% 0.2% 1.5% 2.2% 4.6% 2 3 4 0.0% 1 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 7 of 19 Online Senior Assessment 2010: Multicultural Multicultural 4: The chart below shows the distributions of answers to the fourth question for participants who took their class for the Multicultural core requirement at TTU (blue) and participants who took their class for the Multicultural core requirement elsewhere (red). Answer 4 is the correct choice. It can be seen that less than half of both groups chose the correct answer. The fact that so many participants chose the same wrong answer (i.e., answer 2) might indicate a lack of understanding in this area or that the answer choice was confusing. It appears that students probably did not understand the difference between discriminatory action and a stereotype. More people in the TTU group chose the correct answer as compared to the ELSE group (46.8% vs. 37.6%). This difference is statistically significant at the 0.05 level (see attachment A for details). This suggests that on average students who take their class for the Multicultural core requirement at TTU do better with this question than students who take their Multicultural class elsewhere. Multicultural 4 50.0% 46.8% 44.2% 45.0% 39.8% 40.0% 37.6% 35.0% 30.0% 25.0% TTU 20.0% ELSE 14.7% 15.0% 11.3% 10.0% 5.0% 3.6% 2.2% 0.0% 1 2 3 4 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 8 of 19 Online Senior Assessment 2010: Multicultural Multicultural 5: The chart below shows the distributions of answers to the fifth question for participants who took their class for the Multicultural core requirement at TTU (blue) and participants who took their class for the Multicultural core requirement elsewhere (red). Answer 1 is the correct choice. It can be seen that a large majority in both groups chose the correct answer. There are also quite a few participants in both groups who chose one of the other answers. More people in the TTU group chose the correct answer as compared to the ELSE group (77.2% vs. 70.1%). This difference is statistically significant at the 0.05 level (see attachment A for details). This suggests that on average students who take their class for the Multicultural core requirement at TTU do better with this question than students who take their Multicultural class elsewhere. Multicultural 5 90.0% 80.0% 77.2% 70.1% 70.0% 60.0% 50.0% TTU 40.0% ELSE 30.0% 20.0% 3.8% 5.1% 10.0% 7.9% 11.7% 11.1% 13.2% 0.0% 1 2 3 4 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 9 of 19 Online Senior Assessment 2010: Multicultural Multicultural 6: The chart below shows the distributions of answers to the sixth question for participants who took their class for the Multicultural core requirement at TTU (blue) and participants who took their class for the Multicultural core requirement elsewhere (red). Answer 1 is the correct choice. It can be seen that a majority in both groups chose the correct answer. There are also quite a few participants in both groups who chose one of the other answers. More people in the ELSE group chose the correct answer as compared to the TTU group (75.6% vs. 74.2%). This difference is not statistically significant at the 0.05 level (see attachment A for details). This means that on average students who take their class for the Multicultural core requirement elsewhere do not do better with this question than students who take their Multicultural class at TTU. Multicultural 6 80.0% 74.2% 75.6% 70.0% 60.0% 50.0% 40.0% TTU 30.0% ELSE 20.0% 11.1% 10.0% 13.4% 15.7% 8.6% 1.3% 0.0% 0.0% 1 2 3 4 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 10 of 19 Online Senior Assessment 2010: Multicultural Multicultural 7: The chart below shows the distributions of answers to the seventh question for participants who took their class for the Multicultural core requirement at TTU (blue) and participants who took their class for the Multicultural core requirement elsewhere (red). Answer 2 is the correct choice. It can be seen that a majority in both groups chose the correct answer. There are also quite a few participants in both groups who chose one of the other answers. Overall, a few more people in the TTU group chose the correct answer as compared to the ELSE group (77.6% vs. 68.5%). This difference is statistically significant at the 0.05 level (see attachment A for details). This means that on average students who take their class for the Multicultural core requirement at TTU do better with this question than students who take their Multicultural class elsewhere. Multicultural 7 90.0% 77.6% 80.0% 68.5% 70.0% 60.0% 50.0% TTU 40.0% ELSE 30.0% 20.0% 23.9% 15.4% 5.7% 5.1% 10.0% 1.3% 2.5% 0.0% 1 2 3 4 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 11 of 19 Online Senior Assessment 2010: Multicultural Multicultural Average: The table below compares the differences between TTU and ELSE when the results are averaged (e.g., if a student got 5 out of the 7 questions correct, his score will be 5/7 = .71). Participants who took their course for the Multicultural core requirement at Texas Tech have a higher mean than participants who took their course elsewhere. This difference is statistically significant at the 0.05 level. This suggests that on average students who take their Multicultural course at TTU perform better on the Multicultural section of the OSA than students who take their Multicultural course elsewhere. N Multicultural Overall Core at TTU Mean SD 558 76.8% 17.6% N Core Elsewhere Mean SD 197 71.9% 18.8% T-stat 2.772 P-value 0.006 The chart below shows the distributions of the average scores for participants who took their class for the Multicultural core requirement at TTU (blue) and participants who took their class for the Multicultural core requirement elsewhere (red). The distributions are fairly similar. The main difference appears to be that the TTU group had quite a few more participants who answered all of the questions correctly. Multicultural Overall: TTU vs. ELSE 35.0% 31.0% 30.0% 26.2% 29.2% 27.5% 25.0% 20.8% 18.4% 16.8% 20.0% TTU 13.3% 15.0% 10.0% 5.0% 0.0%0.0% 0.2%0.8% 3.5% 1.8% 5.0%5.5% 0.0% 0% 14.3% 28.6% 42.9% 57.1% 71.4% 85.7% 100% Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 12 of 19 ELSE Online Senior Assessment 2010: Multicultural The table below shows a comparison of the average scores for the participants selecting each course option within the ELSE group (those selecting more than one course option were excluded from the analysis). The table includes the F value and P value for an analysis of variance comparing the means. Although the table shows information for all five ELSE options, the groups with less than 15 participants (Dual Credit, Advanced Placement, and CLEP Exam) were excluded from the analysis of variance. Dual Credit Advanced Placement CLEP Exam Another Institution Study Abroad Total N 13 8 1 163 58 243 Mean 72.5% 87.5% 57.1% 70.9% 76.6% 72.8% St. Dev. 18.9% 11.9% . 19.2% 16.2% 18.6% F-value 4.060 P-value 0.045 Based on the mean, students who took their Multicultural course through advanced placement are the highest-performing group and students who took their course at another institution are the lowest-performing group, excluding the one student who took a CLEP exam. The two means included in the analysis of variance are significantly different at the 0.05 level. This suggests that on average for the students who take their Multicultural course elsewhere, which course option they use to take their Multicultural course makes a difference in how they perform on the Multicultural section of the OSA. More specifically this suggests that on average students who take their Multicultural course through study abroad do better on the Multicultural section of the OSA than students who take their Multicultural course at another institution. The group that took their Multicultural course through advanced placement did not have enough participants to include in the analysis of variance, but it appears that they may be the best performing group on average. This outcome makes sense when considering that students who take an AP course need to pass an exam to receive credit, which suggests that they performed well in their AP class. Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 13 of 19 Online Senior Assessment 2010: Multicultural The chart below shows the distribution of the average scores for the female participants (blue) and the male participants (red). The male participants had higher overall scores than the female participants (mean of 76.1% vs. 75.2%). This difference is not significant at the 0.05 level (see attachment B). This means that on average male students do not do better than female students with the Multicultural section of the OSA. Multicultural Overall by Sex 35.0% 29.7% 29.0% 26.9% 26.3% 30.0% 25.0% 21.2% 20.0% 17.9% 17.0% 16.3% 15.0% Female Male 10.0% 5.0% 3.2% 1.3% 1.0% 0.0%0.0% 0.0% 4.8%5.4% 0.0% 0% 14.3% 28.6% 42.9% 57.1% 71.4% 85.7% 100% Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 14 of 19 Online Senior Assessment 2010: Multicultural The following table shows the correlations between the overall average for the Multicultural questions and time to complete OSA, GPA, SAT score, ACT score, transfer credits, total credits earned, and age (p-values for the correlations are in parenthesis). The correlations with GPA, SAT score, ACT score, transfer credits, and total credits earned are all significant at the 0.05 level. These correlations suggest that on average students with higher GPA’s, higher SAT scores, higher ACT scores, less transfer credits, and more total credits do better on the Multicultural section of the OSA. Some of these correlations are small and are more likely to be found statistically significant because of the large sample size. Multicultural Overall Correlation P-value N Time -0.064 (0.079) 755 GPA 0.112 (0.002) 755 SAT 0.307 (<0.001) 511 ACT 0.311 (<0.001) 388 Transfer Credits -0.115 (0.002) 755 Total Credits 0.117 (0.001) 755 Age -0.033 (0.370) 755 The following tables show the results of regression models for the overall average for the Multicultural questions including all of the variables that have been explored in this analysis. There are three separate regression models because not all of the participants have an SAT score and ACT score. Since most students either have an SAT score or an ACT score, there would be too many missing values if both scores were included in the same regression model. The first model excludes both in order to include most respondents in the analysis. Multicultural Model 1 N F P-value 753 4.03 < 0.001 Variable Coefficient P-value Intercept 0.610 < 0.001 Time -0.000020 0.095 Sex -0.014 0.305 GPA 0.036 0.002 Transfer Credits -0.00024 0.544 Total Credits 0.00031 0.324 Age 0.00012 0.925 Multicultural class taken at TTU 0.025 0.143 This first model excludes SAT and ACT score to include 753 of the 755 participants. The model overall is significant at the 0.05 level (R2 = 0.0365). For this model, GPA is the only significant predictor at the 0.05 level for the overall average for the Multicultural questions. This predictor suggests that on average students with higher GPA’s do better on the Multicultural section of the OSA when the other variables in the model are held constant. Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 15 of 19 Online Senior Assessment 2010: Multicultural Multicultural Model 2 N F P-value 511 8.13 < 0.001 Variable Coefficient P-value Intercept 0.243 0.007 Time -0.000016 0.309 Sex -0.010 0.507 GPA 0.028 0.049 Transfer Credits -0.00039 0.419 Total Credits 0.00012 0.758 Age 0.0049 0.032 Multicultural class taken at TTU 0.013 0.536 SAT 0.00029 < 0.001 This second model includes SAT score and excludes ACT score to include 511 of the 755 participants. The model overall is significant at the 0.05 level (R2 = 0.1147). For this model GPA, age, and SAT score are the significant predictors at the 0.05 level for the overall average for the Multicultural questions. These predictors suggest that on average students with higher GPA’s, older students, and students with higher SAT scores do better on the Multicultural section of the OSA when the other variables in the model are held constant. When the SAT score is included, the variable GPA seems to become less important. It is interesting that age becomes more important when the SAT score is included. Multicultural Model 3 N F P-value 387 6.27 < 0.001 Variable Coefficient P-value Intercept 0.308 0.004 Time -0.000059 0.200 Sex -0.032 0.094 GPA 0.023 0.231 Transfer Credits -0.00048 0.392 Total Credits -0.0000047 0.992 Age 0.0046 0.113 Multicultural class taken at TTU 0.0098 0.699 ACT 0.012 < 0.001 This third model includes ACT score and excludes SAT score to include 387 of the 755 participants. The model overall is significant at the 0.05 level (R2 = 0.1172). For this model, ACT score is the only significant predictor at the 0.05 level for the overall average for the Multicultural questions. This predictor suggests that on average Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 16 of 19 Online Senior Assessment 2010: Multicultural students with higher ACT scores do better on the Multicultural section of the OSA when the other variables in the model are held constant. Limitations It is interesting that students who take their Multicultural course at TTU seem to do better on this measure of Multicultural understanding than students who take their Multicultural course elsewhere. It is difficult, though, to measure Multicultural understanding with only seven questions. As mentioned in the results, some of the questions were answered correctly by a large majority of the participants in both groups and may not be good discriminators of Multicultural knowledge. Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 17 of 19 Online Senior Assessment 2010: Multicultural Attachments Attachment A: Summary of Chi-Square Tests for Questions 1 - 7 TTU (N=500) correct incorrect 401 157 524 34 536 22 261 297 431 127 414 144 433 125 Multicultural 1 Multicultural 2 Multicultural 3 Multicultural 4 Multicultural 5 Multicultural 6 Multicultural 7 Else (N=232) correct incorrect 134 63 183 14 179 18 74 123 138 59 149 48 135 62 Chi Statistic 1.04 0.25 7.83 5.00 4.05 0.16 6.43 Chi Probability 0.3074 0.6163 0.0051 0.0253 0.0441 0.6897 0.0112 Attachment B: 2-Sample T-Test for Average Scores by Sex N Multicultural Overall 441 Female Mean SD 75.2% N 17.7% 312 Male Mean SD 76.1% T-stat 18.5% -0.684 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 18 of 19 P-value 0.494 Online Senior Assessment 2010: Multicultural Attachment C: Number of Participants Selecting Each Answer for Each Question Multicultural 1 Answer TTU 1 59 2 401 3 97 4 1 Multicultural 4 Answer TTU 1 12 2 222 3 63 4 261 Multicultural 7 Answer TTU 1 86 2 433 3 32 4 7 ELSE 15 134 48 0 ELSE 7 87 29 74 Multicultural 2 Answer TTU 1 1 2 1 3 524 4 32 Multicultural 5 Answer TTU 1 431 2 21 3 44 4 62 ELSE 0 2 183 12 ELSE 138 10 23 26 Multicultural 3 Answer TTU ELSE 1 536 179 2 9 6 3 1 3 4 12 9 Multicultural 6 Answer TTU ELSE 1 414 149 2 62 17 3 75 31 4 7 0 ELSE 47 135 10 5 Office of Planning and Assessment, Devin DuPree and Sabrina Sattler, July 2010 Page 19 of 19