L21 Steering Committee Minutes
3/18/13
Present: Andy Agar, Ehud Benor, Cathy Cramer, Karen Gocsik, Bob Gross, Alex Hartov, Josh
Kim, Tom Luxon, Jennifer Taxman, Ellen Waite-Franzen, Barbara Knauff, Susan Zaslaw, Pat
Coogan
1. Spring Term Pilot Course Update – Barbara Knauff
Eight faculty have agreed to participate in the Spring Term pilot courses, teaching in BB SP 11 or
Canvas. As decided earlier, the pilot would include a number of smaller writing and rhetoric
courses taught in either Canvas or Blackboard in order to get a closer comparison of the
systems.
Larger courses:
Gross, BIOL 4, Bb (31)
Somoff, RUSS 13, Canvas (40)
Goodman, SOCY 1, Bb (73)
Irwin, ECON 39, Canvas (2 sections, 70)
Writing and Rhetoric:
Gocsik, WRIT 11, Bb (16)
Compton, SPEE 30, Bb (14)
Anguiano, SPEE 20 & 27 , Canvas (2 courses, 20)
Sargent, WRIT 43, Canvas (32)
Question: How advanced are the features that are being used by faculty teaching the pilot
courses? Karen Gocsik (WRIT 11, BB): Plans to use discussion board for peer review, blog,
may try rubrics for students to apply to peer feedback for student presentations. She noted
that she is not having difficulty importing Word documents into the course.
Bob Gross (BIOL 4 BB): Will use discussion board, blog for continual peer review. Reported that
the new Bb interface is like "night and day" compared to SP 7; he is accessing and using tools
that have been there, but not as apparent.
2. Review of Evaluation Activities and Discussion of Feedback Plan – Barbara Knauff
Barbara described evaluation activities during this phase of the LMS review, the intended
audience, and the plan for gathering feedback from each (handout: L21 Evaluation Activities
and Feedback Plan). Activities include mini courses, self-service exploration, Spring Term pilot
courses, Town Hall demos, guided labs, and peer references.
Ehud expressed concern about the order of the rating scales used in the self-service testing
environments. They appear as rating from negative to positive. Is it counter-productive? Are
we testing for positivity or negativity? If possible, the team will adjust the rating scales to
move from positive to negative.
2013-03-18-SC-Meeting-Minutes
Discussion: Are we asking the right questions?
Pilot courses:
Barbara explained that the team will collect feedback from pilot faculty and students at midterm and at the end of the term. Because of the variability in the pilot environment - e.g.,
different courses, different faculty - pilot testing will not produce a one-to-one comparison of
the platforms; rather, user feedback will provide information on different dimensions of
usability. What information does the Steering Committee need?
Discussion: Suggested questions / areas of inquiry for pilot courses
For faculty:
 Did you attempt or think about some change in your course design after evaluating this
tool?
 How well has the system leveraged your thinking about course design?
 Did you find a feature in the LMS that you are testing that encouraged you to try
something in your teaching?
 Get a sense of what tools they used with each system. Level of expertise. Will show
different view and sense of the system.
 Canvas pilots: Describe their experience with system upgrades while teaching the
course.
 Is this system integral to the course, embedded in the course, or is it an add-on?
 The evaluation has many dimensions and it is difficult to ask faculty to compare tools
when both systems are new to them. Are we testing the system or new ways of
teaching the course?
 Attempt to get a sense of the emotional reaction to the LMS. How did they feel?
Prompt an emotional response in open-ended questions at the beginning of the survey.
For students:
 Does the view in your course change your perspective on how you see course?
 Describe any features used in the pilot that you had not used before.
 Does the LMS increase your sense of ownership of your learning experience?
 Do you think using the LMS increased your engagement and learning in this course?
 Does the LMS allow you to see the logical progression of the material in the course?
 Ask about experience with mobile apps.
 Compare systems: Is your experience with the new Bb or Canvas better than your
previous experience with Bb?
 Is this system integral to the course, embedded in the course, or is it an add-on?
 Use approach similar to National Survey of Student Engagement in question design.
 Will analytics be evaluated?
 Focus groups: ask faculty for permission to conduct short, in-class student focus groups
in pilot courses, 30 minutes maximum. Bob and Karen supported this approach.
Schedule toward the end of the term, frame as a teachable moment.
2013-03-18-SC-Meeting-Minutes

Attempt to get a sense of the emotional reaction to the LMS. How did they feel?
Prompt an emotional response in open-ended questions at the beginning of the survey.
Mobile:
Josh asked about evaluation plan for mobile. How will these features be tested?
The committee agreed that broader student representation in testing mobile apps is a
preferable approach.
Discussion on approaches to evaluating mobile:
 In the pilot course orientation, Instructional Designers would be deliberate about
pointing out mobile applications. Ask if students intend to use mobile apps. In
evaluation, ask whether students used mobile apps, what they used it on, describe the
experience.
 Add links to the mobile apps to a visible place on the main page of each system.
 Work mobile activities into the guided labs.
 Develop Kresge training focusing on using mobile devices. Need to consider which
tasks to include on mobile, as not all apply.
Organizations:
Tom asked whether the LMS support of organizations will be tested. Ellen replied that better
sites for organizations are needed, not in an LMS.
Peer references:
Barbara provided a list of peer institutions that might be approached about either Canvas or
Blackboard. What questions should we ask and who should we talk to?
Discussion and suggestions:
 Involve faculty and students. Barbara noted that these individuals may difficult to
contact, but that instructional designers may provide good information on their
experiences with faculty.
 Ask for existing structured data, surveys about implementation.
 Do you have any regrets about your decision?
 Understand what they were using, and what the drivers were for looking at a new
system.
 How easy and how costly was implementation?
 Was there an uptick in LMS use after adopting the new LMS?
 Describe your change process and experiences.
 We need to identify our goals for this process and ask reference if that was their goal.
 Impact on demand for instructional design support.
 SIS integration: Given their systems, how easy was it to integrate. How did it work out in
real life?
2013-03-18-SC-Meeting-Minutes

How easy was it to work with the company? How responsive were they?
Can we make a recommendation before having all the Spring Term Course Pilot feedback?
Susan reviewed the feedback timeline (handout: Feedback Timeline)
Discussion: We will miss student review of materials if early decision; there is significant
student experience toward the end of the course. Grade center data may not be fully utilized.
Cathy indicated that she was hesitant to make decision before a course is over.
Is the extra feedback from the end of the term that much different from the mid-term
feedback? Is the information gathered mid-course and the other information that the team has
gathered compelling enough to wait or not. Do we ask whether we are ready to make a
decision at the May 22 meeting, or want additional information?
A broader conversation is needed about change. More information is needed about the
faculty's willingness to change. The committee agreed that it is very important to have
commitment from faculty that they want to change as often as Bb changes. And, if a
recommendation were made to move to Canvas, do we know how faculty will react to constant
incremental change? It is also important to look at the long-term relationship with the
company and we should not just be evaluating on functionality.
Follow ups
1. Barbara: draft a more definite version of a survey or interview questions and
circulate to the committee by email.
2. Explore the feasibility of offering the Kresge mini course on mobile.
3. Address mobile apps questions to pilot courses.
4. Change order of questions in the pilot course survey
5. Ask faculty whether we could hold a student focus group during class or X hour.
Next Steering Committee meeting: Wednesday, April 10, 10:00 – 11:30
2013-03-18-SC-Meeting-Minutes