Measuring Student Learning
March 10, 2015
Cathy Sanders
Director of Assessment
Workshop Learning Outcomes
Upon conclusion of the workshop, workshop
participants will be able to:
1) identify and define the elements of UNC Charlotte’s
Student Learning Outcomes Assessment Plan and
Report;
2) develop an analytic rubric;
3) link analytic rubrics and assessment data to
Performance Outcomes; and
4) use assessment data to improve student learning.
Elements of UNC Charlotte’s Student
Learning Outcomes Assessment Plan and
Report
1) Student Learning Outcome
2) Effectiveness Measure
with the scoring rubric
3) Assessment Methodology
4) Performance Outcome
5) Assessment Data
6) Continuous Improvement Plan
Student Learning Outcomes (SLOs)
Are:
competencies (i.e., knowledge, skills,
abilities and behaviors) that students
will be able to demonstrate as a result
of an educational program
SLOs need to be specific and
measurable
Sample SLO That is Not Measurable
Because it is Stated Too Broadly:
“Upon conclusion, workshop
participants will be knowledgeable
about student learning outcomes
assessment.”
Sample Measurable SLOs
Workshop participants will be able to:
1) identify and define the elements of UNC
Charlotte’s Student Learning Outcomes
Assessment Plan and Report;
2) develop an analytic rubric;
3) link analytic rubrics and assessment data to
Performance Outcomes ; and
4) use assessment data to improve student
learning.
An Effectiveness Measure is:
a student artifact, (e.g., exam,
project, paper or presentation) that
will be used to gauge students’
acquisition of the student learning
outcome.
Important Attributes of an Effectiveness
Measure
Validity: Does the Effectiveness Measure assess what it is
supposed to (i.e., the knowledge, skill, ability or behavior
articulated in the SLO)?
Example: An oral presentation would be a valid
Effectiveness Measure for assessing students’ oral
communication skills.
Reliability: Does the Effectiveness Measure consistently assess
the knowledge, skill, ability or behavior of interest?
Analytic rubrics help to facilitate inter-rater reliability
Sample Effectiveness Measure
“Workshop participants will develop an
analytic rubric to demonstrate their ability to
apply their new knowledge of analytic rubrics
acquired in the workshop. The analytic rubric
will model rubrics used by faculty to evaluate
student artifacts and will include a minimum
of a 3 point rating scale and 3 evaluation
criteria with cell descriptors for each criterion
for each level of competency.
Assessment Methodology Is:
A description of the department’s process for assessing the
student learning outcome and reviewing the resulting
assessment data that includes:
1) when and in what course the assessment will be
administered;
2) how the student artifact will be evaluated and by whom;
3) the department’s process for collecting, analyzing and
disseminating assessment data to department faculty will be;
and
4) the department’s process for annually reviewing assessment
data and deciding changes/improvements to make will be.
Sample Assessment Methodology
“Workshop participants will develop an analytic
rubric after the instructor has defined a rubric,
shared a sample rubric and explained the steps in
constructing a rubric. Rubrics will be collected by
the instructor and a Scoring Rubric for Analytic
Rubric Activity will be used to assess workshop
participants’ ability to develop a rubric. The
instructor will summarize and disseminate the
assessment findings to OAA staff who will identify
areas needing improvement and decide changes to
make prior to the next training workshop.”
A Performance Outcome Is:
the percentage of students that will
demonstrate proficiency on the
student learning outcome and the
level of proficiency expected.
Sample Performance Outcome
“80% of workshop participants will score
‘Acceptable (2)’ or higher on the Scoring
Rubric for Analytic Rubric Activity”
Scoring Rubric Defined
A scoring tool that uses a set of evaluation criteria that
are directly tied to the student learning outcome to
assess student learning.
When the content of the rubric is communicated prior
to students completing the work, the grading process is
very clear and transparent to everyone involved.
When scoring rubrics are used, Performance Outcomes
are to be tied to the rubric scale.
Scoring Rubric for Analytic Rubric Activity
Criteria
1 = Needs Improvement
2 = Acceptable
3 = Accomplished
Rating
(1 to 3)
Ability to
develop a rubric
scale
Is unable to develop a
minimum of a 3 point scale
that progresses from least
proficient to most
proficient
Ability to
develop
evaluation
criteria
Is unable to identify at least Is able to identify at least 3
3 relevant evaluation
relevant evaluation criteria
criteria
Ability to develop Cell descriptors are missing
cell descriptors
for some evaluation criteria.
Many of the cell descriptors
are not sufficiently detailed
to differentiate between the
levels of proficiency.
Is able to develop a
minimum of a 3 point scale
that progresses from least
proficient to most proficient
Cell descriptors are present
for all evaluation criteria but
some do not clearly
differentiate between the
levels of proficiency.
Is able to develop a scale
that is greater than 3
points that progresses
from least proficient to
most proficient
Is able to identify more
than 3 relevant
evaluation criteria
Cell descriptors are
present for all evaluation
criteria that clearly
differentiate between the
levels of proficiency.
Steps in Constructing an Analytic
Scoring Rubric
1.
2.
3.
4.
5.
6.
7.
8.
Determine what kind of rubric scale to use (e.g., 3 pt., 5 pt.).
Label each level of proficiency across the top of the rubric (e.g., “(1) Needs
Improvement,” “(2) Acceptable,” “(3) Accomplished”).
List evaluation criteria down the left side of the rubric template.
Write cell descriptors of what the highest level of proficiency “(3)
Accomplished” looks like for each criterion.
Write cell descriptors of what the lowest level of proficiency “(1) Needs
Improvement” looks like for each criterion.
Write cell descriptors of what the mid levels of proficiency look like for each
criterion.
Test it: use the rubric to evaluate student artifacts. Make note of important
evaluation criteria that were omitted, existing criteria to eliminate, and cell
descriptors that need greater specificity.
Revise the rubric to reflect the desired changes.
Analytic Rubric Activity
• Using the blank rubric template provided,
develop an Oral Presentation Rubric with a
minimum of a 3 point rating scale and 3
evaluation criteria with cell descriptors for
each criterion for each level of competency.
Tying it All Together Into a Student
Learning Outcomes Assessment Plan
SLO Assessment Plan
Student Learning Outcome: Workshop participants will be able to:
1) identify and define the elements of UNC Charlotte’s Student
Learning Outcomes Assessment Plan and Report;
2) develop an analytic rubric;
3) link analytic rubrics and assessment data to Performance
Outcomes; and
4) use assessment data to improve student learning.
Effectiveness Measure: Workshop participants will develop an
analytic rubric to demonstrate their ability to apply their new
knowledge of analytic rubrics acquired in the workshop. The analytic
rubric will model rubrics used by faculty to evaluate student artifacts.
and will include a minimum of a 3 point rating scale and 3 evaluation
criteria with cell descriptors for each criterion for each level of
competency.
SLO Assessment Plan
Assessment Methodology: Workshop participants will develop
an analytic rubric after the instructor has defined a rubric,
shared a sample analytic rubric and explained the steps in
constructing a rubric. Rubrics will be collected by the instructor
and a Scoring Rubric for Analytic Rubric Activity will be used to
assess workshop participants’ ability to develop a rubric. The
instructor will summarize and disseminate the assessment
findings to OAA staff who will decide changes to make prior to
the next training workshop.
Performance Outcome: 80% of workshop participants will score
“Acceptable (2)” or higher on the Scoring Rubric for Analytic
Rubric Activity.
Reporting Assessment Data
Report assessment data in the same way
that the Performance Outcome is stated.
PO: 80% of workshop participants will score
“Acceptable (2)” or higher on the Scoring
Rubric for Analytic Rubric Activity
Assessment Data: 89% of workshop
participants scored “Acceptable (2)” or higher
on the Scoring Rubric for Analytic Rubric
Activity
Continuous Improvement Plan
During the annual review of assessment data, department
faculty will:
1. determine whether students are meeting the
Performance Outcome(s) for each Student Learning
Outcome;
2. identify changes that are needed to improve future
student learning;
3. develop the department’s continuous improvement
plan for the upcoming year; and
4. the following year, document in the Student Learning
Outcomes Assessment Plan and Report whether the
changes made improved student learning.
Changes to Consider When Students Are Not
Meeting Performance Outcomes
1.
Change the assessment instrument
– Revise the assessment instrument (i.e., test questions or project
requirements)
– Change to an assessment instrument that measures deeper learning
(e.g. from test questions to a written paper)
– Revise or add rubric evaluation criteria
2.
Change the assessment methodology
– Change what you are assessing (i.e., revise the SLO statement)
– Change when SLOs are assessed (junior year vs senior year)
– Change how the assessment is administered (e.g., videotaped oral
presentation vs oral presentation so that students can watch
afterward to self-assess)
Changes to Consider When Student Are Not
Meeting Performance Outcomes
3.
Change the curriculum
– Revise courses to provide additional coverage in areas where students
did not perform well
– Revise the curriculum to increase students’ exposure to SLOs by
introducing, reinforcing and emphasizing competencies throughout
the curriculum
– Schedule an appt. with CTL curriculum design specialist
4.
Change the pedagogy
– Incorporate more active learning that provides students with
opportunities to apply what they are learning
– Incorporate mini assessments throughout the semester to gauge
earlier whether students are grasping the material and adjust
accordingly
– Incorporate active learning opportunities for students to share/teach
the material they are learning in your classroom
Questions/Discussion