Online Senior Assessment 2009: Visual and Performing Arts Introduction
The last part of OSA contains the questions for the Visual and Performing Arts (VPA).
The first question asks if students took their VPA course as a core requirement at TTU.
The second question asks in which area the course was taken. The actual questions
are shown below.
Honors
Landscape
Architecture
21
0
26
0
1
0
Office of Planning and Assessment, September 2009 Page 1 of 23 3 41
0
0
TOTAL
Electronic Media &
Communications
24
0
Not Answered
Architecture
88
0
Other
Theater Arts
8 134
0
0
Spanish
Music
VPA_TTU 139
VPA_Else
0
Dance
Art
The following table shows that out of the 704 participants, 485 (i.e., 69%) indicated that
they took their VPA course for core curriculum credit at TTU and 219 (i.e., 31%)
indicated that they took it elsewhere.
0 485
219 219
Online Senior Assessment 2009: Visual and Performing Arts The system stores some data for each of the participants and so it was possible to
identify the respondents who major in Visual and Performing Arts. The following majors
were identified and classified as VPA majors: ARTC, ARTH, ARVS, DAN, MUBA,
MUPF, MUTC, SART, THA, THAA, and THDS. These participants represent the socalled “experts” in the sample. The following table shows that there were a total of 35
VPA majors in the OSA sample. It also displays how many participants were in each of
the major areas.
Data Preparation
Visual and Performing Arts was the only area that asked open-ended questions. See
below for a screenshot of the actual instrument/questions.
Office of Planning and Assessment, September 2009 Page 2 of 23 Online Senior Assessment 2009: Visual and Performing Arts The answers to the three open-ended questions were rated based on their length and
split into the following groups: no answer/incomplete, very short answer, short answer,
and long answer. The following table summarizes how many answers fall into each
group. If “short answer” and “long answer” are considered valid responses, there are a
total of 61% valid responses.
no answer/ incomplete
very short answer
short answer
long answer
Total
183
110
222
243
758
The chart below shows how students who took their course for core requirement at TTU
(“TTU”) and students who took their VPA course for core requirement elsewhere
(“Else”) answered the questions. It is interesting to see that while about two-thirds of the
TTU group has valid responses only about half of “Else” fall into that category. It is
concerning that about 35% of the “Else” group have no answer at all or an incomplete
answer. For the exact split of the two groups between the different answer categories,
please see attachment A.
40%
35%
30%
25%
20%
15%
10%
5%
0%
TTU
Else
Since the questions specifically ask for answers that are 7 – 10 sentences long, only the
243 long answers were chosen to be considered for grading. The overall sample
contains 31% of transfer students and 69% of resident students. Since only 200
answers were to be graded and there were more than 200 long answers it was almost
possible to recreate that split by having all 64 transfer answers and 136 randomly
chosen TTU answers (i.e. 32% and 68%) graded.
Office of Planning and Assessment, September 2009 Page 3 of 23 Online Senior Assessment 2009: Visual and Performing Arts There were two groups (i.e. Red and Blue) with two raters each. Each group received
the answers from 100 participants. That means each rater graded all three of the
answers from 100 participants. Each rater attended a training session where they
learned about what was expected from the students. They also received a scoring
instrument which explains what grades should be given for which kind of answer (see
attachment B for a copy of the actual scoring instrument). The ratings go from 1
(unacceptable) to 5 (superior). The answer to each of the three questions received one
grade per rater. These three grades were averaged to one grade per participant. Since
each participant’s answers were rated by two graders, each participant initially had two
average scores. When these average scores differed by more than 1 point, the answers
were sent to a third rater. In the end, every participant has one score for each of the
three questions that is either an average of the first two raters or an average of all three
raters. Out of 100, there were 52 in one group (i.e., 52% in group “Red”) and 8 (i.e., 8%
in group “Blue”) from the other group that needed to be send to a third rater. Attachment
C shows descriptive statistics and measures of agreement for the two raters of group
“Blue” and attachment D does the same for group “Red”. While in the first group the first
rater has a higher mean score for all three questions, in the second group the second
rater has a higher mean score for all three questions. The difference is much more
pronounced in group “Red” which makes sense considering that 52% of the participants
required a third grader. When assessing acceptable levels of rater agreement, there
are several different accepted choice of measures when dealing with ordinal data (like
the scale from 1 to 5 used here). For this data, both Gamma and Kendall’s Tau seem
appropriate. Kendall’s Tau is not very high for group “Blue” (.57, .50, and .68 for the 3
questions) and at a very similar range for group “Red” (.62, .53, and .68 for the three
questions). Both would indicate only moderate agreement for the two groups. Gamma
might actually be the preferred statistic here though because there are quite a few ties
in the ratings. Group “Blue” (.76, .65, and .86) seems to have somewhat lower
agreement than group “Red” (.82, .70, .86), but overall Gamma indicates a good level of
agreement. However, neither statistic seems to capture the consistently higher ratings
for the second rater in group “Red”. This probably means that while the two raters in
that group have different starting points, they both are very consistent in their ratings
(for example, rater 1 would choose 2, 3, and 4 while rater 2 would choose 3, 4, and 5).
Overall, the agreement between the raters seems acceptable, but there is room for
improvement. This indicates that maybe more examples in the training session could
help define for the raters what to expect from the participants for specific scores.
Results
The student learning outcomes for Visual and Performing Arts are:
- Identify and describe a body of work (individually and collectively) in the creative
arts.
- Explain and differentiate creative works as expressions of values within cultural
and historical contexts.
- Analyze and summarize aesthetic principles that structure creative works.
Office of Planning and Assessment, September 2009 Page 4 of 23 Online Senior Assessment 2009: Visual and Performing Arts They align very closely with the questions on the Online Senior Assessment (excluding
the student rating of the event which acts as an anchor for the third open-ended
question). The first question on the OSA almost states the first student learning
outcome verbatim. The second question aligns with the second learning outcome. The
third question asks the student to affirm their position while demonstrating that different
people have different opinions and valuing of the aesthetics of the work they are
viewing/hearing/experience/etc which aligns with the third student learning outcome.
The chart below shows the distributions of grades received for the first answer for
participants who took their class for VPA core requirement at TTU (bottom) and
participants who took their class for VPA core requirement elsewhere (top). It can be
seen that the group who took the class at TTU has more people in the higher
percentages. Overall, the group “TTU” has a higher average score than the group “Else”
(69% vs 63%). This difference is statistically significant at the .05 level. Please see
attachment E for details. This means that students who take their class for VPA core
requirement at TTU do significantly better than students who take it elsewhere. Since
the first question aligns with the first learning outcome, this means that on average
students who take their VPA course at TTU meet this learning outcome better than
those who take the course elsewhere.
Office of Planning and Assessment, September 2009 Page 5 of 23 Online Senior Assessment 2009: Visual and Performing Arts The chart below shows the distributions of grades received for the second answer for
participants who took their class for VPA core requirement at TTU (bottom) and
participants who took their class for VPA core requirement elsewhere (top). It can be
seen that the group who took the class at TTU again has more people in the higher
percentages. It seems that the “Else” group had more trouble with this question than
with the first question as shown by the many scores below 50%. Overall, the group
“TTU” also has a higher average score than the group “Else” (68% vs 59%). Again, this
difference is statistically significant at the .05 level. Please see attachment F for details.
This means that students who take their class for VPA core requirement at TTU do
significantly better than students who take it elsewhere. Since the second question
aligns with the second learning outcome, this means that on average students who take
their VPA course at TTU meet this learning outcome better than those who take the
course elsewhere.
Office of Planning and Assessment, September 2009 Page 6 of 23 Online Senior Assessment 2009: Visual and Performing Arts The chart below shows the distributions of grades received for the third answer for
participants who took their class for VPA core requirement at TTU (bottom) and
participants who took their class for VPA core requirement elsewhere (top). While “TTU”
still seems to have a few more high scores than “Else”, it is much less clear than in the
first two questions. Overall, the group “TTU” still has a higher average score than the
group “Else” (61% vs 59%). However, this difference is not statistically significant at the
.05 level. Please see attachment G for details. This means that students who take their
class for VPA core requirement at TTU do not do significantly better than students who
take it elsewhere. Since the third question aligns with the third learning outcome, this
means that on average students who take their VPA course at TTU meet this learning
outcome at about the same level than those who take the course elsewhere. It seems
that both groups have trouble with the third question. This could indicate that neither
group has learned to adequately summarize and analyze aesthetic principles. However,
it could also just be that students were not willing to write a long third answer. After all,
the OSA is a fairly long assessment and these questions are the last ones listed.
Office of Planning and Assessment, September 2009 Page 7 of 23 Online Senior Assessment 2009: Visual and Performing Arts The chart below shows the distributions of the average scores for all three answers for
participants who took their class for VPA core requirement at TTU (bottom) and
participants who took their class for VPA core requirement elsewhere (top). The
distributions are fairly similar, but it looks like there is a bigger group in “Else” that
performs just below average and a bigger group that performs just above average.
Overall, “TTU” seems to have more students with higher scores and also has a higher
average score than the group “Else” (66% vs 60%). This difference is statistically
significant at the .05 level. Please see attachment H for details. This means that
students who take their class for VPA core requirement at TTU perform significantly
better in the Visual and Performing Arts than students who take it elsewhere.
Office of Planning and Assessment, September 2009 Page 8 of 23 Online Senior Assessment 2009: Visual and Performing Arts The chart below shows the distributions of the average scores for all three answers for
those participants who are majoring in the VPA area (bottom) and participants who do
not have a major that is part of VPA (top). It is interesting to see that the non-majors
clearly cluster around 60% while most majors have average in the 70% and 80% range.
Overall, the majors have a much higher average than the non-majors (73.55% vs
62.82%). This difference is statistically significant at the .05 level. Please see
attachment I for details. This means that students who major in VPA perform
significantly better than students who major in other areas.
The correlations between the overall average for the VPA questions and GPA, total
transfer credits, total credits earned, and age are all fairly low and none of them are
significant (see attachment J). There also is not a statistically significant difference
between average overall performance between male and female students (see
attachment K).
Office of Planning and Assessment, September 2009 Page 9 of 23 Online Senior Assessment 2009: Visual and Performing Arts Comparison: 2008 vs. 2009
Compared with 2009, the graders/raters in 2008 did a better job. In 2008, there were
three groups that graded answers and about 10% from the first group, 20% from the
second group, and 30% from the third group needed to be re-graded. In 2009, one
group only had 8% that needed to be re-graded but the other group had 52% that
needed to be re-graded. However, the large number of answers that needed to be regraded in 2009 stems largely from one grader consistently giving higher scores. To
prevent this in the future, graders should be made aware that grading too loosely hurts
the overall reliability of the study.
Limitations
When looking at the results, it is important to keep in mind that only the long answers
were considered for grading. Therefore, the grades are much higher than they would
have been if all the incomplete or short answers had been considered as well. This
means that the results here only depict those participants who were willing or able to
answer the questions.
Office of Planning and Assessment, September 2009 Page 10 of 23 Online Senior Assessment 2009: Visual and Performing Arts Attachments
Attachment A: Answer Categories by Group
Row Labels
very short
answer
Count of Did
you take your
VPA course for
core curriculum
credit at TTU?
percentage per group
percentage
of total
percent of
total transfer
12.60%
110
No
32
29.09%
4.22%
Yes
78
70.91%
10.29%
short answer
15.48%
222
No
66
29.73%
8.71%
Yes
156
70.27%
20.58%
64
26.34%
8.44%
179
73.66%
23.61%
long answer
25.98%
30.95%
243
No
Yes
no answer/
incomplete
25.20%
35.52%
183
No
92
50.27%
12.14%
Yes
91
49.73%
12.01%
Grand Total
758
254
total transfer
0.335092348
504
total core here
0.664907652
758
Office of Planning and Assessment, September 2009 Page 11 of 23 percent of
total core
taken here
36.22%
18.06%
Online Senior Assessment 2009: Visual and Performing Arts Attachment B: Scoring Instrument
Constructs of Scoring Instrument (revised 10-21-08)
Content
How clear and concise is the description of the work(s)/event? Does the student . . .
• Clearly describe, using discipline-appropriate terminology, the work(s) and
artist(s), including technical, creative, and/or aesthetic elements that
constituted the work(s)/event?
Context
How effective is the description of the cultural/historical context? Does the student . .
.
• Explain and differentiate the work(s) as expressions of values within cultural
and historical context(s) and how the context relates to the underlying
meaning behind the work(s)/event?
Effectiveness
How well does the student explain & defend their judgment of the event? Does the
student . . .
• Effectively present a convincing argument on behalf of their rating while
offering considered reasons why others may have differing judgments?
Rating Scale of Student Responses
5
Superior
4
Commendable
3
Acceptable
2
Requires
Improvement
1
Unacceptable
Content
Context
Effectiveness
Mean Rating
Office of Planning and Assessment, September 2009 Page 12 of 23 Rating
Online Senior Assessment 2009: Visual and Performing Arts Attachment C: Ratings – Group “Blue”
Office of Planning and Assessment, September 2009 Page 13 of 23 Online Senior Assessment 2009: Visual and Performing Arts Office of Planning and Assessment, September 2009 Page 14 of 23 Online Senior Assessment 2009: Visual and Performing Arts Attachment D: Ratings – Group “Red”
Office of Planning and Assessment, September 2009 Page 15 of 23 Online Senior Assessment 2009: Visual and Performing Arts Office of Planning and Assessment, September 2009 Page 16 of 23 Online Senior Assessment 2009: Visual and Performing Arts Attachment E: 2-Sample T-Test for Question 1 (“TTU” vs. “Else”)
Office of Planning and Assessment, September 2009 Page 17 of 23 Online Senior Assessment 2009: Visual and Performing Arts Attachment F: 2-Sample T-Test for Question 2 (“TTU” vs. “Else”)
Office of Planning and Assessment, September 2009 Page 18 of 23 Online Senior Assessment 2009: Visual and Performing Arts Attachment G: 2-Sample T-test for Question 3 (“TTU” vs. “Else”)
Office of Planning and Assessment, September 2009 Page 19 of 23 Online Senior Assessment 2009: Visual and Performing Arts Attachment H: 2-Sample T-Test for Average Grade (“TTU” vs. “Else”)
Office of Planning and Assessment, September 2009 Page 20 of 23 Online Senior Assessment 2009: Visual and Performing Arts Attachment I: 2-Sample T-Test for Average Grade (“Majors” vs. “Non-Majors”)
Office of Planning and Assessment, September 2009 Page 21 of 23 Online Senior Assessment 2009: Visual and Performing Arts Attachment J: Correlations
Office of Planning and Assessment, September 2009 Page 22 of 23 Online Senior Assessment 2009: Visual and Performing Arts Attachment K: Average Performance and Gender
Office of Planning and Assessment, September 2009 Page 23 of 23