Crowd-Sourcing Innovative Practices:
Assessing Integrative Learning
at Large Research Institutions
Mo Noonan Bischof
Assistant Vice Provost
mabischof@wisc.edu
Amy Goodburn
Associate Vice Chancellor
agoodburn1@unl.edu
Nancy Mitchell
Director, Undergraduate Education
nmitchell1@unl.edu
LEAP Integrative Learning
Synthesis and advanced accomplishment
across general and specialized studies
demonstrated through the application of
knowledge, skills, and responsibilities to
new settings and complex problems.
Challenge:
Assessing Integrative Learning
• Can/Should the same assessment tools be used
for assessing within a course, a unit, and/or
institution?
• Does it apply to integrating knowledge and skills
within a discipline, among disciplines, or both?
• How can we align quality improvement levels
while respecting disciplinary purposes & values?
UW-Madison Learning Community
21,615 employees…
2,177 faculty
1,635 instructional academic staff
1,261 research academic staff
5,291 graduate assistants
42,820 students …
29,118 undergraduates
9,183 graduate students
2,774 professional students
1,745 Non-degree students
Annually:
7,400 new undergraduates
29,500 enrolled undergraduates
6,500 Bachelor’s degree graduates
13 academic schools/colleges
distributed responsibility and governance
~500 academic programs, all levels
134 Bachelor’s level degree programs
Annual Degrees
More than 300
200-299
100-199
50-99
1-49
Includes WIX and ELO’s
Institutional-level learning goals, assessments
Program-level learning
goals, assessments
Program-level learning
goals, assessments
Program-level learning
goals, assessments
Program-level learning
goals, assessments
Program-level learning
goals, assessments
Why pilot the AAC&U VALUE
Rubrics?
• Identified gap: institutional level assessment, direct
measure approach
• Evaluates student learning across programs
• Aligns with AAC&U Essential Learning Outcomes
• Aligns with VSA/College Portrait demonstration project
• First pilot project summer 2012, second pilot 2013
• Main Goal: bring faculty across disciplines together to
evaluate student work
AAC&U VALUE Rubric Project
Scorers
Rubrics
• AAC&U VALUE written
communication rubric
• Cohort of 25 faculty
• Cross-disciplinary representation
• Focus on faculty engagement
Artifacts
• “Value-added” approach to
compare first year students
and students near graduation
Written Communication VALUE
Rubric
Selected written communication for ease of
identifying artifacts across disciplines/programs
Dimensions:
• Context and Purpose for Writing
• Content Development
• Genre and Disciplinary Convention
• Sources and Evidence
• Control of Syntax and Mechanics
Artifacts: “Value-added” Approach
• Goal was to collect 350 artifacts at each
level, FYR and NGR
• Identified 52 courses that had high
numbers of FYR and NGR and seemed
likely to have a suitable writing assignment
• 22 courses (41 instructors) had a suitable
assignment and agreed
• Invited 2450 students to submit artifacts
• Collected 451 submissions
Scorers: Faculty Engagement
• 1.5 day workshop in June 2013
Scorers
• Set ground rules
• 3 structured rounds intended to get
faculty familiar with the rubric and to
“test” scorer agreement
• Asked faculty to think beyond their
field/discipline
• Each scorer rated about 40 artifacts
• Discussion revealed challenge with
the 4-point scale and what is
“mastery”
Rubrics
Artifacts
Table 1. Overall Results for All Artifact Scores
Rubric
Dimension
Context
Content
Genre
Sources
Syntax
Student
Group
Nearly
Graduating
First Year
Nearly
Graduating
First Year
Nearly
Graduating
First Year
Nearly
Graduating
First Year
Nearly
Graduating
First Year
# of
Artifacts
213
Mean
Std Dev
Zmw Score
2.95
0.95
3.05*
237
213
2.77
2.79
0.96
4.68*
237
211
2.48
2.69
0.88
2.65*
235
190
2.50
2.61
0.99
1.54
225
213
2.50
2.82
0.84
2.16*
237
2.69
*Zmw score is from the Mann Whitney U-Test. Zmw scores >1.96 indicate that the two groups
are significantly different at p=0.05.
Table 1. Distribution of Combined Scores - Written Communication
Rubric
60.0
51.0
Percent of Scores
50.0
44.4
40.0
30.3
27.3
30.0
22.6
17.5
20.0
10.0
3.8
2.6
0.0
1
2
First-Year Students
3
Nearly Graduating Students
4
Summary Findings
• Percent of nearly graduating students who were judged
proficient or better (a score of 3 or 4 on 4 point scale) on
each of the dimensions was fairly high—ranged from 64%83%. Across all dimensions: 74.7%
• Levels of significant difference between first-year and nearly
graduating students were weak
• Inter-scorer reliability was problematic (“mastery” issue…)
– Overall 67% of scorer pairs showed weak agreement or
– Systematic disagreement
What did we learn?
• Importance of assignment (artifact) development
• Adapt rubric: program mix and/or campus culture
(language, LOs)
• Engagement of faculty = high quality discussions
(ground rules/calibration)
• Next Steps: continue to engage faculty at program
and disciplinary levels
Contact Information
Mo Noonan Bischof, Assistant Vice Provost,
University of Wisconsin-Madison, mabischof@wisc.edu
More about our project: http://apir.wisc.edu/valuerubricproject.htm
University of Nebraska-Lincoln
Research One, Big Ten Conference, Land-Grant
24,000 students
8 independent colleges
Achievement-Centered Education
(ACE)
•
•
•
•
10 Student Learning Outcomes (30 credits)
600 courses across 67 departments
Transferable across 8 colleges
Requires assessment of collected student work
•
•
•
•
UNL Assessment Context
Review of each ACE course on 5-year cycle
Biennial review of all undergrad degree programs
50 disciplinary program accreditations
10-year North Central/HLC accreditation
ACE 10
Generate a creative or scholarly product that
requires broad knowledge, appropriate
technical proficiency, information collection,
synthesis, interpretation, presentation, and
reflection.
HLC Quality Initiative: ACE 10 Project
•
•
•
•
•
25 faculty across colleges meet monthly to
Explore methods and tools for assessing work
Develop a community to share ideas
Connect ACE 10 & degree program assessment
Develop process for creating assessment report
Create team of assessment “ambassadors”
Discussing Assessment Practices
A Common Rubric
disciplinary vs. institutional goals
Inquiry Project Results
• Abandoned idea to pilot a common rubric
• Revised syllabus to focus on processes, not tools
• Developed poster session for public sharing
• Streamlined ACE & program review processes
• Creating process for 5-year ACE program review
Group Discussion
• How do you address
differences across
disciplinary norms and
cultures?
• What strategies can you
use to develop shared
goals and understanding?
• How can program/
disciplinary assessments
inform institutional
assessment and vice
versa?
• What are some effective
practices for supporting
and sustaining faculty
and staff engagement?