The University of Michigan
Measuring Our Impact: The Open.Michigan Initiative
Goals
Measure Open.Michigan’s impact on the University of Michigan (U-M) campus and on the broader open
education landscape, as well as its progress toward overall objectives and mission.
Methods
Phased evaluation project including developing a strategic plan: fall 2010-ongoing.
Qualitative: Surveys (March and April 2011) were administered to U-M community and users of the
Open.Michigan site and resources measuring: awareness, use, support of OER and sharing habits. The response
rates were 3.6% for students (1,452 completed surveys), 4.8% for faculty (196 completed surveys) and 11.3% (136
completed surveys) for staff1. Analysis of existing documentation and data from previous Open.Michigan
activities (including, process evaluation and resource development, 2007-current) and environmental scans of
evaluation in the open education landscape (2005-current) were also conducted.
Quantitative: We gathered and analyzed usage analytics across hosting platforms (Google Analytics established
across three platforms from 2008-current), our published resources and the number of partnerships (U-M and
global), presentations, and amount of institutional support at the University of Michigan.
Results2
Open.Michigan has gained momentum in the larger context of open education, contributing processes (dScribe,
2007), tools (OERca, 2008 and OERbit, 2011) and policy (Code of Best Practices in Fair Use for OpenCourseWare,
2009) to the global development and sustainability of this movement. It is also slowly building institutional buy-in
from the University of Michigan community, including support from the Medical School, the Library, the School
of Information and the MERLOT community. Most of the partnerships and resource co-development result from
collaboration at similar levels of administration, in units such as instructional design teams, publishing groups
and global engagement initiatives. Within the wide scale U-M community there is still relatively little awareness
of the Open.Michigan initiative and of open educational resources (OER) in general. Many respondents to our
survey continued to express confusion about our work and what makes an OER, how copyright laws intersect
with these issues, and how this relates to their individual workflow or academic needs and interests. From this
evaluation process, Open.Michigan developed a strategic vision (http://tinyurl.com/openmichiganvision)
document and is beginning the process of examining its scalable process for gathering and preparing content
for OER publication (dScribe) in summer 2011.
Materials Sharing: The Open.Michigan survey was based on previous OCW/OER evaluations by CTools, (U-M’s
Sakai learning management system, 2007-2010), MIT (2005), Tufts (2010), and TALL (2010). From the results of the
2010 CTools survey (40.7% instructors never heard of OCW, 75.6% students never heard of OCW) Open.Michigan
decided to include questions about how our campus shares materials (what materials, with whom, how and why).
From this we determined that faculty and students alike share individual lecture slides. Faculty share for a
diversity of reasons (see Figure 1) and share lecture slides with others (colleagues and students) 79% of the time
and syllabi 63% of the time. Students share with their classmates through collaborative online sites like Google
1
2
Staff survey results were too low to be fully representative.
Complete results, aggregate data, resources used, and planning documents can be found at: http://tinyurl.com/omevaluation
1
2 Measuring our Impact: The Open.Michigan initiative
docs (40.3%), CTools (43.7%) but email is by far the most used method for sharing material with classmates,
friends (within and outside U-M), faculty and colleagues (1,140 responses).
Impact of OER at U-M: Students, faculty and staff who had heard, created or used OER responded to specific
questions about their use of it and its impact on their academic outcomes (teaching or learning). Of 72 faculty
who indicated they had heard of, used or adapted OER, 24% agreed that using OER had improved their
pedagogical skills (13 of 27 responses) but their responses to other reasons were distributed across the impact
options of the survey. 57% (89 of 156 responses) use OER to study for tests. Very few faculty, staff and students
adapt OER at this point, but the share it and use it as reference material for a variety of reasons. Overwhelmingly
(74% or 115 of 156 responses) students are using OER to improve their overall understanding of a subject (see
Figure 2).
Figure 1. Faculty Response: "I share learning materials with my colleagues to...?"
Figure 2. Student Response: “How has using OER impacted your academic experience?”
Next Steps
Open.Michigan must continue to analyzed the results of these surveys and compare across populations to assess
trends and sharing patterns. We plan to launch surveys addressing the impact of publishing material with
Open.Michigan and to conduct focus groups in the following months (summer and fall 2011). Based on
The University of Michigan
Measuring Our Impact: The Open.Michigan Initiative
preliminary survey results (2011), environmental scans and analysis of activities and outcomes, Open.Michigan will
continue to move forward with the objectives it has set forth in the vision document, including:
1.
2.
3.
4.
Produce more and richer content as OER with the various campus units, improve modularity, instructional
design, and accessibility of U-M OER
Increase the visibility and discoverability of U-M resources through a combination of marketing and metadata
Draw participants from more parts of campus to expand its disciplinary coverage
Ensure OER production is an embedded part of the academic life on campus
Community-based objectives that resulted from our survey responses correlate to these priorities. They include:
•
•
•
•
•
•
Establish Open.Michigan in main channels of U-M communication
Clarify copyright in OER and provide adequate training resources
Consistently define Open.Michigan and OER
Describe why/how to use/create OER
Make short-term improvements to OER
Encourage cultural shifts in learning
3