Multi-State
Collaborative
to Advance Learning
Outcome Assessment
Pilot Project and PCC
Our Problem
• How can we tell that students who complete the general/transfer
degrees have met the outcomes of those degrees?
• No reliable pattern of course-taking
• No capstone in place
• No exit exam administered
Our Challenge
• NWCCU Recommendation:
• 1. The evaluation committee recommends that the College develop
indicators of achievement for all of the College's core learning
outcomes that are assessable and can be used as a basis for
determining that an established target for student performance levels
has been achieved and that such achievement contributes to
demonstrating mission fulfillment (Standard l.B)
• Figure out the level of achievement you want, and then find a way to
determine how well students meet that
Our Approach
• Has been very SAC-focused
• LDC: How do the core outcomes look in your discipline?
• CTE: How do your outcomes align with core outcomes?
• This is VERY appropriate, and has encouraged conversations
and improvement
Our Hope and Worry
• The Hope:
• that as students make their way through the courses they need to
graduate, they have multiple opportunities to develop and
achieve a level of mastery of the Core Outcomes
• The Worry:
• Will we be able to figure out whether this is really happening ?
Good news:
• Many SACs have found that the best way to assess core
outcomes is to use
• Natural (or at least logical) alignment of discipline content with
core outcomes
• Embedded Assignments (authentic, students will do it, less extra
work for faculty)
• This is a national trend
We are not alone!
• Nine states—Connecticut, Indiana, Kentucky, Massachusetts,
Minnesota, Missouri, Oregon, Rhode Island, and Utah—are
collaborating to develop and pilot a model to use authentic
student work and embedded assignments to assess broad
institutional outcomes
• Statewide teams have been working to develop the model and
pilot
• Active support of SHEEO and AAC&U (and more recently,
Gates Foundation
Key features of the model
• Focuses on broad, institutional, ”essential competency” level
outcomes
• Is designed to reflect cumulative student knowledge and
ability (not focused on the course, instructor or discipline)
• Uses authentic student work derived from embedded
assignments
• Is based on the use of Essential Learning Outcomes and
associated VALUE Rubrics developed by faculty members
under the auspices of AAC&U’s LEAP initiative.
Pilot Project
• Collect student work that can be assessed appropriately for
two “essential competencies,” using the LEAP Value Rubrics
for
• Written Communication
• Quantitative Literacy
• Of students 75% of the way through their degree (Associates
or Bachelor’s)
• To be scored by faculty assembled from all 9 states, CC and
Univ to obtain statewide “score”
Pilot Project
• 6 institutions per state: 3 CCs, 3 Universities
• Artifacts to be collected:
• “Minimum target” is 75 artifacts per outcome per institution
• Ideal is to have “representative” distribution across disciplines
• Students: 75% of the way through their Degree (Associates or
Bachelors)
• Each institution figures out how to reach these targets
Pre-Project Professional Development
• For this to work there must be some instructors who have or
are willing to develop/adapt existing assignments so that
students are likely to demonstrate competency for multiple
elements of the rubric.
• Some Phase I pilot funding will go to states, to be used for
assignment development
Artifact collection
• Institutions remove identifying information (student,
instructor, college, state)
• Artifacts will be coded so the scores can be offered back to
institutions
• Institutions may code internally so results can be used in-house,
back to the instructor if desired
Scoring
• Centralized face-to-face norming and scoring
• Faculty from each of the participating institutions (probably 2,
hopefully 3)
• Team from each state organized by the state project
leadership
Use of Results?
• Campuses may use results however they choose
• States will provide aggregated holistic and analytic scores
among same-sector institutions (e.g., 2-year, 4-year)
• Within-state comparisons may be made with benchmarks
established for same-sector campuses
• Disaggregated comparisons may be made, as possible, by
selected student body and institutional characteristics
• MSC will request aggregated holistic and analytic scores
among same-sector institutions (e.g.,2-year, 4-year)
The Timeline
• Phase I
•
•
•
•
Identification of Institutions: Jan/Feb 2014
Communication and Systems Development
Faculty Professional Development May/June 2014
Artifact Collection: Fall 2014
• Phase II
• Scoring: January 2015
• Results back to states, institutions
Oregon Schools in the Pilot
• University of Oregon
• Oregon State University
• Oregon Tech (OIT)
and
• Portland Community College
• Chemeketa CC
• Southwestern Oregon CC
How will we do this?
• Identify faculty who are “willing to play”
• i.e., who have or are willing to tweak assignments to align with
expectations of the rubric (can be LDC or CTE)
• In Fall, use Banner to identify students in their classes who are
within 75% of degree completion
• The completed assignments from those students will be
collected, coded, redacted if necessary and submitted
• Participating faculty will be offered a chance to be selected for
the multi-state norming/scoring event
A parallel/aligned project for us ?
• We could norm/score the same artifacts as we send forward –
• Compare with multi-state score
• Allows us to involve all participating faculty and others
(depending on need and funding)
• We could norm/score work from all of the students in the
selected classes
• Is there is an difference based on # of credits accumulated?
• If we have SACs participating, we can code that so the results
can be sorted, and they could norm/score, to compare to
Statewide /PCC collection/scoring?
Oregon LO&A Group
• Planning late spring/early summer workshop focused on:
• understanding the rubrics
• developing assignments that work
• Workshop parameters have yet to be determined, but to PCC
participating faculty will be encouraged and supported to
attend
Why I think this is awesome
• Collaboration with faculty from other colleges and universities
• Professional development in…
• Developing assignments
• Scoring with rubrics
• Opportunity to view and evaluate work done by students from
other colleges, universities and states
• What can we learn about our students now?
• Could we do this “in house” to meet NWCCU expectations?
More awesomeness…
• This does not replace the SAC approach, but complements it
• This does not land on SACs as extra work,
• Is voluntary for individual instructors
• SACs may want to “sign on”
• They could use this as their 2014-15 assessment
• It might help those struggling to figure it out
• They might be interested in comparing their results with others
• Quantitative Literacy is not one of our Core Outcome,
• but maybe it should be (it is at MANY colleges and universities
• This would give us a chance to see what that might look like
Questions?
One more thing….
• Suggestions for how to roll out
the “call for participants”?
• FT faculty, PT faculty, also SAC
chairs
• See draft e-mail –
suggestions???