WE GAVE THE TEST NOW WHAT?
ANALYSIS AND REPORTING FOR THE
SOUTH CAROLINA ARTS ASSESSMENT
PROGRAM
Ashlee A. Lewis, Office of Program Evaluation
R. Scot Hockman, SC Department of Education
Intent(s):
To demonstrate the process by which the
SC Arts Assessment Program moves from
test administration to data analysis to
reporting results to schools.
To talk through the continuous
improvement process used for the SCAAP.
SOUTH CAROLINA ARTS
ASSESSMENT PROGRAM (SCAAP)
https://scaap.ed.sc.edu
SCAAP Collaborators



South Carolina Department of Education (SCDE)
 Funding Agency
Office of Program Evaluation (OPE) at USC
 Test And Measurement Specialists
 Logistics
South Carolina Arts Educators
 Content-area Experts
 Capacity building through statewide arts assessment
institutes
Current SCAAP Assessments
Six different assessments in various stages:
 Four Entry Level Assessments
 Music
& Visual Arts (pilot tested in 2002)
 Dance & Theatre (pilot tested in 2005)

Two Middle Level Assessments

Music & Visual Arts (pilot tested in 2008)
Current SCAAP Assessments
All assessments…
 are aligned to SC Academic Standards for Visual &
Performing Arts (2010)
 have two sections
 Multiple-choice/Selected Response Section (45 items)
 Performance Task Section (2 tasks)
SCAAP Administration

Administered each spring
 to
schools that received Distinguished Arts Program (DAP)
grants from the SC Department of Education

Administered at individual schools
 by

appointed test administrators trained by the SCAAP team
Test administrators trained online and in person by the
SCAAP team
SCAAP Participants












2004—66 schools with approximately 5,200 students
2005—51 schools with approximately 3,700 students
2006—70 schools with approximately 4,900 students
2007—81 schools with approximately 5,800 students
2008—56 schools with approximately 4,400 students
2009 – 46 schools with approximately 3,500 students
2010 – 41 schools with approximately 3,740 students
2011 - 45 schools with approximately 3,540 students
2012 – 32 schools with approximately 2,545 students
2013 – 34 schools with approximately 2,763 students
2014 - 34 schools with approximately 2,572
2015 - 41 schools with approximately 3,700 students
(Expected)
Music Test Specifications
Percent covered by Assessment Format
Standard
Overall
Emphasis
Selected Response
Performance Tasks
Standard 1: Performance
25%
-
100%
Standard 2: Creating Music
20%
25%
75%
Standard 3: Music Literacy
25%
100%
-
Standard 4: Critical Response to Music
15%
100%
-
Standard 5: History and Culture
10%
100%
-
5%
100%
-
Standard 6: Connections
Visual Arts Test Specifications
Percent covered by Assessment Format
Standard
Overall
Emphasis
Selected Response
Performance Tasks
Standard 1: Creating Art
25%
40%
60%
Standard 2: Structures and Functions
25%
50%
50%
Standard 3: Exploring Content
10%
100%
-
Standard 4: History and Culture
10%
100%
-
Standard 5: Interpreting Works of Visual Art
25%
25%
75%
5%
100%
-
Standard 6: Connections
MULTIPLE CHOICE/SELECTED RESPONSE
https://scaap.ed.sc.edu
The Assessment Process –
Achieving the Arts Assessment Mission
(Brophy)
Modify and
Improve
Interpret and
Evaluate the Data
Develop and
implement
assessments
Collect Assessment
Data
Assessment
Improve
teaching and
learning
Establish Goals,
and Outcomes
The Assessment Process –
Achieving the Arts Assessment Mission
(Brophy)
Modify and
Improve
Interpret and
Evaluate the Data
Develop and
implement
assessments
Collect Assessment
Data
Assessment
Improve
teaching and
learning
Establish Goals,
and Outcomes
SCAAP Item Bank
Example Music Item
Example Visual Arts Item
Analyses Performed

Reliability indices for test forms
 Cronbach’s

alpha and corrected split-half index
Test form equating
 Using
Item Response Theory (IRT)
 Cross year and cross form test equating
 Empirical reliability based on fitted IRT model



Differential item functioning (DIF) analysis for gender
and ethnicity
Distribution of p-values (percent correct) for items
Discrimination indices for each item
Reliability Estimates
Test
Music
Visual
Arts
Empirical
Reliability
Form
# of Cronbach’s Corrected
Items
Alpha
Split Half
1
45
0.79
0.79
2
45
0.82
0.82
1
45
0.85
0.86
2
45
0.86
0.85
.83
.86
Item Review Process


Convene arts advisors in fall to revise items identified as
problematic.
Review and revise based on:
 P-values
 Discrimination
indices
 Differential Item Functioning (DIF)
 Distribution of distractors


Archive items
Write new items to incorporate into test forms
Improving the Assessment
Each year, item analysis guides the revisions of the
assessment for the following year:
 P-values: Between .20 and .85
 Discrimination index: .19 or higher
 Differential Item Functioning (DIF): All items which
receive a C classification are examined.
PERFORMANCE TASKS
https://scaap.ed.sc.edu
Performance Tasks

Visual Arts
 Compare
and contrast two artworks using art terms
 Create a drawing with given art elements/principles based
on a prompt

Music
 Sing a familiar song
 Improvise an 8-beat rhythm pattern using rhythm syllables
and maintaining a steady beat
Preparing Performance Tasks



Receive tasks from schools (flash drives and booklets)
Scanning visual arts performance tasks
Mass uploading of tasks to website for online rating
Performance Task Benchmarking




A sampling of responses is chosen and “pre-scored” by
members of the SCAAP team based on the rubrics
Arts advisors indicate their agreement or dissent with
existing scores
Discrepancies in scores are discussed and elements of
the rubric are clarified
Validated, benchmarked tasks are used to train and
monitor raters
Rater Training




Rater trainings have been done online and in person,
depending on level of experience
Raters review rubrics and anchor items and review rater
bias types
Interactive practice tests provide feedback on scoring
Raters must pass qualifying test before they begin rating
Rating System




Entire rating system is online
Raters must pass a refresher test after scoring 100 tasks
Monitoring: Seeds are placed randomly throughout unscored tasks
On average, each rater scores 600 tasks over the course
of 3 weeks
Analyses performed


Inter-rater reliability (GENOVA)
Both MC and Performance tasks:
 Correlations
between multiple-choice test forms and
performance tasks for each area.
Inter-rater reliability estimates
Music 2013
Performance
Task
1
(Singing)
2
(Improvisation)
Criteria
Generalizability
Index of
Coefficient
Dependability
Tonal
0.94
0.94
Rhythm
0.84
0.83
Vocal
0.87
0.87
Rhythm
0.87
0.87
Improvisation
0.79
0.79
Inter-rater reliability estimates
Visual Arts 2013
Performance Task
Generalizability
Coefficient
Index of
Dependability
1
(Compare and
Contrast)
0.88
0.88
2
(Drawing)
0.74
0.74
Annual technical report provided to SC Arts
Education Associate and posted online each fall.
Full report available at:
https://scaap.ed.sc.edu
REPORTING
https://scaap.ed.sc.edu
Report Cards




Revisions to report cards made based on teacher
feedback
Report cards generated in collaboration with
programmer
Multiple-choice section of report cards generated and
disseminated prior to end of school year in May
Full report cards including performance tasks results
disseminated the following September.
Sample Report Card
Research on the SCAAP Assessments




Comparing the Dimensionality Structures of Music &
Visual Arts Multiple-Choice Assessments (SCEPUR, 2006)
An Exploratory Study of the Dimensionality Structure of
a Music Multiple-Choice Assessment (AERA, 2006)
Efficacy of a Web-Based Training and Monitoring
Procedure in Scoring Performance Tasks (AERA, 2007)
Raters Characteristics and Performance Scores (AERA,
2008)
Research on the SCAAP Assessments




Rhythm Syllable System and Rhythm Achievement
(AERA 2008)
The Effect of Gender on a Language-related
Theatre Task (SCEPUR, 2009)
Teachers’ Use of Assessment Results (AEA, 2010)
Teachers Making Meaning of Displays of Student
Results (AEA, 2011)
Research Using SCAAP Results




Comparing Arts Achievement to English Language Arts and
Mathematics Achievement in Arts Education Reform Schools
(SCEPUR, 2005)
Evaluating the Program Characteristics of Arts Schools with
Disparate Achievement Levels (SCEPUR, 2006)
Multiyear Evaluation of the Arts Education Reform Effort in
South Carolina (AERA, 2007)
Investigating Arts Programs and Implementation Strategies
for Infusing Arts Into Curriculum (AERA, 2007)
SCAAP Publications
Featured in an assessment textbook:
 Assessing Performance: Designing, Scoring, and
Validating Performance Tasks (Johnson, Penny, &
Gordon, 2008)
Music Assessment Symposium Proceedings:
 Assessment in Music Education: Integrating Curriculum,
Theory, and Practice (Yap & Pearsall, 2007)
Thank you! We welcome your questions!
lewisaa2@mailbox.sc.edu
shockman@ed.sc.gov