MIT OpenCourseWare
Program Evaluation Findings
Summary Report
March 2004
For more information, please contact:
Stephen E. Carson
Evaluation Specialist
MIT OpenCourseWare
77 Massachusetts Avenue, 9-251
Cambridge, MA 02139
Phone: 617-253-1250
Fax: 617-253-2115
Email: scarson@mit.edu
For a further explanation of the data contained in this summary, and additional data, please see the complete MIT
OCW Program Evaluation Report, March 2004, available online at
http://ocw.mit.edu/OcwWeb/Global/AboutOCW/evaluation.htm.
• CONTENTS •
I. EXECUTIVE SUMMARY
II. ACCESS FINDINGS
A. SITE TRAFFIC AND FREQUENCY OF VISITS
B. GEOGRAPHIC DISTRIBUTION OF USERS
C. EDUCATIONAL ROLE
D. EDUCATOR PROFILE
D. TECHNICAL CONSIDERATIONS
E. AWARENESS
III. USE FINDINGS
A. CONTENT AREAS
B. EDUCATOR SCENARIOS OF USE
C. SATISFACTION WITH QUALITY OF MATERIAL OFFERED
IV. IMPACT FINDINGS
A. OVERALL IMPACT ON TEACHING AND LEARNING
B. EDUCATOR REUSE OF MIT OCW MATERIALS
C. CONSISTENCY WITH THE MIT BRAND
2
3
3
4
5
6
7
8
9
9
10
11
12
12
13
14
I. Executive Summary
About OpenCourseWare
MIT OpenCourseWare (MIT OCW) is a large-scale, Web-based electronic publishing initiative, accessible on the
Internet at http://ocw.mit.edu.1 Through MIT OCW, MIT makes MIT’s core teaching materials—lecture notes,
problem sets, syllabi, reading lists, simulations, etc.—freely and openly available for non-commercial educational
purposes. MIT OCW publishes those materials in standards-based formats to anyone with access to the Internet.
MIT OCW has a dual mission:
•
Provide free access to virtually all MIT course materials for educators, students, and individual
learners around the world.
•
Create an efficient, standards-based model that other universities may emulate to publish their own
course materials.
Evaluation Design and Execution
In order to understand how well MIT OCW is fulfilling its mission—as well as to establish a thorough and
continuous feedback process that guarantees its improvement over time—MIT OCW has developed a substantial
evaluation program. The evaluation is based on a logic model that focused on understanding specifics in three areas
of user behavior:
•
Access: Who is using MIT OCW?
•
Use: How are they using it and does it meet their needs?
•
Impact: What outcomes result from this use?
The evaluation was undertaken from October 1 through November 30, 2003.2 Data collection employed an
integrated “portfolio approach,” as a combination of methods helped to achieve both breadth and depth in the
evaluation:
•
Web Analytics. Akamai, MIT OCW’s Web hosting and content distribution network provider, captures
aggregate usage data such as page views, object views and user location and offers a more sophisticated
analytic tool called SiteWise, which MIT OCW employed starting November 1, 2003. Data drawn from
these sources are either direct measures, or correlations at a 95% level, between the two systems.
•
Online Intercept Surveys. Between November 6 and 19, the survey tool invited (via a pop-up window) a
random sample of 21,467 MIT OCW visitors to complete an online survey. Of those prompted, 3,573
people began the survey, and 1,220 completed it fully, for a dropout rate of 66% and an overall completion
rate of 5.7%. The sample provides a margin of error of not more than 3%.
•
Supplemental Surveys. A supplemental survey was distributed to 600 individuals around the world with
emphasis on some of the target regions (Latin America, Africa, Asia and Eastern Europe) and roles
(primarily educators and students). 62 respondents started to fill out the survey, and 29 completed it, for a
dropout rate of 53% and overall completion rate of just under 5%. This sample set provides qualitative
insights into the experiences and attitudes of MIT OCW visitors in those target geographies.
•
Interviews. Members of the MIT OCW research team conducted 25 in-depth interviews with participants
drawn from among intercept and supplemental survey respondents who indicated a willingness to
participate and distributed across several target regions (Latin America, Asia, Eastern Europe, North
America) and groups (educators, students and self-learners).
Web analytic, survey, and interview data provide a complementary picture of site Access, Use and Impact, with no
apparent contradictions.
1
Making MIT OCW content available via alternative delivery media is under consideration.
Please note that MIT OCW announced the publication of the 500th course on October 1, with an accompanying flurry of media coverage and
publicity. This led to unusually high levels of site access and usage patterns during that period (particularly an unusually high number of first time
visitors to the site).
2
MIT OCW Program Evaluation Summary Report
March 2004 – Page 2
II. Access Findings
A. Site traffic and frequency of visits
MIT OCW traffic volume is high, and there is a core of repeat visitors.
•
The MIT OCW Web site recorded 728,000 visits between October 1 and November 31, 2003—an
average of almost 12,000 visits per day. (Table 1)
•
Returning visitors accounted for 25% of daily visits in November 2003. (Table 2)
•
More than 95% of users plan to return to the MIT OCW site in the future.
•
Almost 10% of visitors report daily use of the site; 25% at least weekly use; and 40% report more than
10 previous visits to the site.
Table 1
First-Time vs. Repeat Visits — Nov. 1, 2003 to Jan. 30, 2004
80,000
60,000
Repeat Visits
40,000
First-Time Visits
20,000
0
Week 46,
2003
Week 48,
2003
Week 50,
2003
Week 52,
2003
Week 2,
2004
Week 4
2004
Source: Akamai Sitewise
Table 2
Visit
Frequency
Number of Times Previously Visiting
% of Users
Once
2-5 Times
6-10 Times
More than 10
Times
First time
42.3%
N/A
N/A
N/A
N/A
Daily
8.9%
1.6%
3.3%
1.4%
8.8%
Weekly
24.8%
1.4%
10.2%
8.2%
23.0%
Monthly
9.8%
0.1%
6.3%
5.1%
5.1%
Occasionally
(less than once
per month)
14.3%
1.8%
15.1%
4.8%
3.1%
TOTALS
100%
5.0%
34.8%
19.6%
40.1%
Source: Intercept Survey Results
MIT OCW Program Evaluation Summary Report
March 2004 – Page 3
B. Access: Geographic distribution of users
MIT OCW has attracted international attention, with more than half the site traffic coming from
outside North America.
•
45% of visitors come from North America (United States and Canada).
•
Western Europe is second most common point of origin (19%), and East Asia is third with 18%.
•
Middle East and North Africa (1.6%) and Sub-Saharan Africa (0.4%) represent measurable portions of
MIT OCW traffic.
Table 3
Rank
Region
Est. Daily Visitors
% of MIT OCW Traffic
by Region
% of Total Internet
Users By Region3
1
North America
5,352
45.4%
29.6%
2
Western Europe
2,234
19.0%
26.1%
3
East Asia
2,153
18.3%
28.3%
4
Latin America
694
5.9%
5.0%
5
Eastern Europe
465
3.9%
2.0%
6
South Asia
301
2.5%
2.6%
7
Middle East and North Africa
187
1.6%
2.1%
8
Central Asia
165
1.4%
1.2%
9
Pacific
163
1.4%
1.9%
10
Sub-Saharan Africa
53
0.4%
0.9%
11
Caribbean
19
0.2%
0.2%
Source: Akamai SiteWise
3
This column represents an approximate distribution of Internet users by geographical region. Estimates of total number of Internet users from
www.Internetworldstats.com. Underlying usage information comes mainly from data published by Nielsen-NetRatings, ITU, and local NIC and
ISP sources.
MIT OCW Program Evaluation Summary Report
March 2004 – Page 4
C. Access: Educational role
Educators, students and self-learners access the site extensively.
•
Numerically, self-learners predominate, representing 52% of visitors with an average of 6,000 daily
visits. Self-learners are most likely to come from North America (60% of North American visitors).
•
Students represent approximately 31% of visitors or an average of 3,600 daily visits.
•
Educators represent over 13% of the visitors or an average of 1,550 visits per day.
•
The MIT OCW user base is well educated. Almost 70% have a bachelors degree or higher.
Table 4
Role
Middle
All
North Western East Latin Eastern South East & Central
Sub-Saharan
Pacific
Caribbean
Regions America Europe Asia America Europe Asia North Asia
Africa
Africa
Educator
13.1%
8.4%
20.0% 13.8% 18.0%
20.3% 9.0% 33.3% 14.3% 23.5%
20.0%
23.5%
Student
30.9%
26.5%
29.8% 40.7% 30.3%
48.4% 40.3% 25.0% 14.3% 23.5%
40.0%
23.5%
Self-learner 51.6%
59.9%
46.8% 43.4% 49.2%
26.6% 41.8% 41.7% 57.1% 52.9%
20.0%
52.9%
Other
5.2%
3.4%
4.7%
20.0%
0.0%
4.4%
2.1%
2.5%
9.0% 0.0% 14.3% 0.0%
Source: Intercept Survey Responses
MIT OCW Program Evaluation Summary Report
March 2004 – Page 5
D. Access: Educator Profile
Educators represent over 13% of the visitors or an average of around 1550 visits per day.
•
Educators represent a higher percentage of visitors for several regions outside North America, e.g.
Latin America (18%) and Eastern Europe (20%).
•
55% of educators using MIT OCW teach at four-year colleges or the equivalent and their expertise is
most commonly focused in electrical engineering and computer science (26%), or business and
management (14%). (See Tables 5 and 7)
•
Almost 49% of educators using MIT OCW have less than five years teaching experience; the balance
of educator use is distributed across the remaining spectrum of experience levels. (See Table 6)
Table 5
Educator’s Type of Institution
Educator %
Secondary school (e.g., “high school”)
7.6%
2-year college, junior college or the equivalent
8.3%
4-year college or university or the equivalent
54.8%
Graduate or professional school
17.8%
Technical or other trade school
4.5%
Other
7.0%
Source: Intercept Survey Responses
Table 6
Table 7
Educators’
Experience
All Regions
Educator’s Subject Area
% of
Educators
<=5 Years
48.8%
Electrical Engineering and
Computer Science
25.6%
6-10 Years
14.4%
Management
8.8%
11-15 Years
10.0%
Other
8.1%
16-20 Years
10.0%
Mechanical Engineering
6.3%
>20 Years
16.9%
Business
5.0%
Physics
3.8%
Economics
3.8%
Literature
3.8%
Civil and Environmental Engineering
3.1%
Source: Intercept Survey Responses
Source: Intercept Survey Responses
MIT OCW Program Evaluation Summary Report
March 2004 – Page 6
E. Access: Technical considerations
The technical platform for MIT OCW enables high levels of satisfaction with site performance and
other aspects of usability across a wide range of connection types.
•
Over 95% of current MIT OCW users reported they were satisfied with the site performance. (See
Table 9)
•
Over 94% reported satisfaction with the site organization. (See Table 9)
•
Over 97% reported satisfaction with the site visual design and presentation of materials. (See Table 9)
•
The approximately 16% of MIT OCW users accessing the site via dial-up connections reported only
slightly lower satisfaction at 94%. (See Table 10)
Table 9
Strongly Agree
Agree
Neutral
Disagree
Strongly
Disagree
The MIT OCW Web site
is easy to use
38.4%
51.9%
5.2%
2.1%
2.4%
The MIT OCW Web site
is well organized and
easy to navigate
37.2%
50.7%
7.9%
2.0%
2.2%
I’m able to find what I’m
looking for on MIT OCW
24.7%
47.6%
19.0%
5.6%
3.0%
I learned to use the MIT
OCW Web site quickly
46.6%
39.3%
9.6%
2.0%
2.5%
Attributes
Source: Intercept Survey Responses
Table 10
Connection Type
User Satisfaction
Dial-up
94.2%
Cable modem
96.5%
DSL
95.5%
LAN
95.8%
All connection types
95.6%
Source: Intercept Survey Responses
MIT OCW Program Evaluation Summary Report
March 2004 – Page 7
F. Access: Awareness
User awareness of MIT OCW comes via a range of channels.
•
Almost 63% of visitors became aware of MIT OCW via online or offline media articles.
•
Site activity is heavily impacted by media coverage—a peak of usage following the October 2003
global publicity was in excess of 60,000 visits in one day.
•
Over 25% of users report that they became aware of MIT OCW through a colleague, peer, or teacher.
Table 11
Channel
% of Users
Educator
Student
Self-learner
36.5%
31.6%
37.8%
38.1%
26.4%
20.9%
23.6%
30.4%
Colleague or peer
23.0%
34.2%
22.8%
20.5%
Search engine
10.2%
10.8%
10.7%
10.2%
Teacher
2.5%
2.5%
5.1%
0.8%
Online media (online news article,
link, etc.)
Offline media (newspaper,
magazine, television, radio)
Source: Intercept Survey Responses
MIT OCW Program Evaluation Summary Report
March 2004 – Page 8
III. Use
A. Content areas
MIT OCW use is centered on subjects for which MIT is a recognized field leader.
•
Electrical Engineering and Computer Science course sites attract 34% of traffic for users who accessed
specific course materials on MIT OCW while only accounting for 10% of the total courses published.
•
Mathematics, Management, Economics and Physics sites account for an additional 26% while
representing a further 21% of courses published.
Table 12
Rank
MIT Department
# Courses
Published
% of Courses
Published
Dept Total #
Visits
% Of Visits
(Excluding
Global Pages)
1
Electrical Engineering and
Computer Science
55
10.7%
481,062
33.6%
2
Mathematics
18
3.5%
114,678
8.0%
3
Sloan School of Management
47
9.1%
112,795
7.9%
4
Economics
21
4.1%
71,260
5.0%
5
Physics
21
4.1%
67,526
4.7%
6
Mechanical Engineering
20
3.9%
49,113
3.4%
7
Brain and Cognitive Sciences
62
12.1%
47,389
3.3%
8
Aeronautics and Astronautics
16
3.1%
39,250
2.7%
9
Foreign Languages and
Literatures
21
4.1%
38,569
2.7%
10
Architecture
16
3.1%
28,019
2.0%
Source: Akamai SiteWise
MIT OCW Program Evaluation Summary Report
March 2004 – Page 9
B. Use: Educator scenarios of use
Users are largely confirming hypothesized scenarios of use for MIT OCW.
•
Educators most frequently use the site for planning, developing or improving and teaching courses or
classes (44%), and secondarily to enhance their personal knowledge (25%).
•
Students most frequently use the site to find subject matter and materials for use in conjunction with a
course they were currently taking (43%), and secondarily to enhance their personal knowledge (39%).
•
Self-learners overwhelmingly use the site to enhance personal knowledge (80%). No significant new
scenarios of use emerged from the evaluation.
Table 13
All Levels
0-5 years
6-10 years
11-15
years
16-20
years
>20 years
Developing or planning a course(s)
that I am or will be teaching
36.0%
30.3%
28.6%
40.0%
57.1%
45.8%
Learning about subject matter to
enhance my personal knowledge (not
directly related to my teaching)
22.0%
25.0%
28.6%
6.7%
14.3%
20.8%
Learning about subject matter to
enhance my research
11.3%
14.5%
9.5%
6.7%
14.3%
4.2%
Preparing to teach a specific class
10.7%
9.2%
19.0%
20.0%
7.1%
4.2%
Developing or planning a curriculum
for my department/program
10.0%
15.8%
0.0%
6.7%
0.0%
8.3%
Advising students about their course
of study
6.7%
5.3%
4.8%
6.7%
7.1%
12.5%
Planning or developing an
educational Web site or related
educational technology
3.3%
0.0%
9.5%
13.3%
0.0%
4.2%
Faculty Usage Scenario
Source: Intercept Survey Responses
MIT OCW Program Evaluation Summary Report
March 2004 – Page 10
C. Use: Satisfaction with quality of material offered
MIT OCW visitors are largely satisfied with the quality of content on the MIT OCW site.
•
More than 92% of users express high levels of satisfaction (with the overall quality of course materials
published on MIT OCW.
•
79% of users are satisfied with the breadth of subject matter and course areas available on the site,
(currently less than a third of MIT’s overall curriculum).
•
71% of users express satisfaction with depth and completeness of materials for courses available on
MIT OCW.
Table 14
% Satisfaction
% of Educators
% of Students
% of Selflearners
Subject matter and course areas covered
on the MIT OCW Web site
79.4%
73.0%
73.7%
84.6%
Types of course materials for specific
courses
71.0%
73.6%
65.3%
72.8%
Quality of the course materials for specific
courses
92.2%
97.5%
86.7%
93.6%
Content Characteristics
Source: Intercept Survey Responses
MIT OCW Program Evaluation Summary Report
March 2004 – Page 11
IV. Impact
A. Overall impact on teaching and learning
MIT OCW users overwhelmingly are finding that MIT OCW has, or will have significant positive
impact on both teaching and learning activities.
•
Over 80% of all users report either positive impact or extremely positive impact, 18% report moderate
or some positive impact and less than 2% reporting no positive impact.
•
Over 95% of all users report an intention to return to MIT OCW in the future.
•
Over 92% agree that they will recommend MIT OCW to someone else.
Table 15
All Frequency of Visitors
Impact
Extremely/ Positive Impact
Moderately/ Somewhat
Positive Impact
No Positive Impact
Educators
80.40%
18.40%
1.30%
Student
83.60%
15.30%
1.10%
Self-learner
84.80%
13.80%
1.40%
Source: Intercept Survey Results
MIT OCW Program Evaluation Summary Report
March 2004 – Page 12
B. Impact: Educator reuse of MIT OCW materials
Educators report very high impact on teaching practices.
•
Over 97% of educators expressed satisfaction with the quality of the course materials published on the
MIT OCW Web site.
•
Over 47% have reused MIT OCW materials, or plan to in the future. (See Table 16)
•
76% agree that MIT OCW will impact their future teaching practices. (See Table 17)
Table 16
Educators have, or
are planning to,
reuse MIT OCW
materials
Yes
No
Not Sure
Not
applicable
(not currently
teaching)
47.1%
12.1%
37.6%
3.2%
Source: Intercept Survey Responses
Table 17
MIT OCW will have an
impact on teaching practices
Agree
Neutral
Disagree
75.9%
17.6%
8.6%
Source: Intercept Survey Responses
MIT OCW Program Evaluation Summary Report
March 2004 – Page 13
C. Impact: Consistency with the MIT brand
MIT OCW is perceived as being consistent with MIT’s overall brand.
•
Over 84% of users report MIT OCW is consistent or extremely consistent with their perceptions of MIT.
•
Less than 2 percent find MIT OCW inconsistent with the MIT brand.
Table 16
Level of Consistency
Overall
Educator
Student
Self-learner
Extremely consistent
27.7%
26.0%
23.9%
30.3%
Consistent
56.6%
57.8%
58.2%
55.4%
Mixed
14.0%
12.3%
16.3%
13.0%
Somewhat
inconsistent
1.4%
3.9%
1.1%
1.0%
Extremely inconsistent
0.4%
0.0%
0.5%
0.3%
Source: Intercept Survey Responses
MIT OCW Program Evaluation Summary Report
March 2004 – Page 14