Canvas Outcomes procedures and data collection
Overview:
The objective of Implementing Canvas Outcomes has two main goals. One would be to allow the
collection of data relating outcomes to individual student profiles for the collection of demographic
information. The other would be to streamline and improve the process that instructors use to collect
data for SLO (Student Learning Outcomes).
Although the Canvas system has several reports and tools dedicated to the collection and analysis of
outcome data the current offerings are not directly usable for our SLO process. The main difficulty is that
our SLO system is looking for the number of students in a particular SLO category while Canvas data at
the course level is compiled only by average, median and mode.
Data Collection:
Using the Canvas Administration reports we are able to collect student rubric data at the account level.
The Canvas system is setup to collect outcome data for quiz questions as well. However, the data is
provided for each individual question and not for the entire assessment and would need to be
consolidated. In addition the data does not include information crucial to determining the overall
assessment results. These two factors essentially render the data useless for our purposes.
There is however another approach to outcome data for quizzes that will allow us to gather the data.
Rather than aligning the outcome to a question bank you can attach a rubric to the Quiz and use the
rubric to assess the outcome after the quiz is completed. The disadvantage is that the outcome result is
not automatically calculated along with the quiz results. However there is no additional work as
compared to not using the rubric because the instructor would still need to evaluate each quiz
separately to select an outcome result. The advantage is that the outcome data will be generated as
one result for the entire assessment for each student and will not involve the use of a separate mastery
Percentage. By eliminating the need to calculate a separate percentage for quiz question assessment we
also simplify the assessment matrix and can use the same matrix for all outcomes created at the account
level.
Proposed Procedures:
I would propose that we create all SLO at the account level using a simple Matrix code of:
Not evaluated: The student did not participate in assessment
0
Did not meet expectations: the student did not demonstrate achievement of
the outcome. The student response included significant errors and/or
omitted significant details.
Met Expectations: the student demonstrated achievement of the outcome,
but their response included minor errors or omissions.
Exceeded Expectations: the student demonstrates the outcome without any
errors or omissions in their response.
1
2
3
1. Admins will create the outcomes at the account level using the data from the SLO system.
2. Instructors will need to import the outcome required for this session into the course
3. The assessment to be evaluated can be any graded Assignment, Quiz or Discussion and will need
to have a rubric attached to it. If the assignment already has a rubric the outcome can be add to
the existing rubric otherwise an empty rubric can be created for the outcome. Note that the
outcome should be added to the rubric with the option selected to not add outcome points to
the grade score to avoid interference with existing rubric grading point totals.
4. Once the assessment has been completed by the students the instructor will need to use the
speed grader to access the rubric for ALL students being assessed to select the appropriate
outcome score (see above). It is important that all students including those that did not
participate are given an outcome score.
5. Instructors can use the learning mastery screen or data export to aggregate their results for the
SLO report.
6. At the account level we can run the outcome report and use it to analyze the data across all
outcomes and courses. Data on a specific courses or group of courses could be produced to help
with the SLO report at the discipline or department level.
7. In addition if the outcomes were created at the sub account level instead of at the Main account
level we may be able to directly generate reports at the discipline or department level to better
sub divide the data. I currently do not have access to the sub accounts and cannot currently test
this however; it would be worth looking into the possibility.
8. Once the outcomes are used in a course they cannot be removed from the system. In order to
accommodate changes in the outcomes, new outcomes will need to be created and older ones
archived to a separate location. I would recommend that we create an archive folder at the
account level and move any updated or redundant outcomes that have been used in the past to
that folder.
Conclusion:
It is possible to use the existing reports and tools in canvas to capture student specific outcome data at
the account level without undue burden on the instructors. In fact after the initial setup and training
phase this provides a fairly easy and consistent way to coordinate and collect SLO data across a group of
courses. As this is currently a manual process I think this method could be of benefit to all parties
involved.
I would suggest that we pilot this with a few volunteer instructors to verify the data collection process
and gather instructor feedback.