OER Research Hub
Evidence Hub Development Brief
Date: 28.08.2014
OER Research Hub
OER Impact Map Development Brief
Contents
1.0 Document Control ......................................................................................................... 3
1.1 Version History .............................................................................................................. 3
1.2 Distribution .................................................................................................................... 3
2. Purpose of document ...................................................................................................... 3
3. Summary .......................................................................................................................... 4
4. About the OER Research Hub project............................................................................ 4
4.1
Project objectives ....................................................................................................... 5
4.2
Project approach: collaboration and openness ....................................................... 6
4.3
Project stakeholders .................................................................................................. 7
4.4
Role of the specified work in the project .................................................................. 8
5. Rationale ......................................................................................................................... 11
6. Specification ...................................................................................................................12
6.1 File Types ......................................................................................................................12
6.2 Information Architecture ..............................................................................................13
6.3 Data and Metadata ........................................................................................................14
6.4 Authentication ..............................................................................................................15
6.5 User Interface ...............................................................................................................15
6.6 Usability & Accessibility ..............................................................................................16
7. Delivery ...........................................................................................................................17
7.1 Phase I – Public Prototyping ......................................................................................17
7.2 Phase II – Iterative Development .................................................................................18
7.3 Phase III - Legacy .........................................................................................................20
1
OER Research Hub
Evidence Hub Development Brief
Date: 28.08.2014
7.4 Resources .....................................................................................................................23
7.5 Liaison and Reporting ..................................................................................................24
7.5 Intellectual Property .....................................................................................................24
7.6 Legacy ...........................................................................................................................24
Appendix A: Background Information on OLnet Evidence Hub .....................................25
Appendix B: Summary of UNESCO OER Mapping Project..............................................29
Appendix C: Wireframe for Survey Data Explorer……………………………………………26
References ..........................................................................................................................32
Author: Dr Robert Farrow, Research Associate
IET
rob.farrow@open.ac.uk
2
OER Research Hub
Evidence Hub Development Brief
Date: 28.08.2014
1.0 Document Control
Document
Identifier:
Class
Deliverable:
Project start
date:
Project
duration:
OER Research Hub Evaluation
Framework
Date due:
OER Evidence Hub WP4
Submission
date:
State:
1 September 2012
Draft
24 months
1.1 Version History
Date
Version
Stage
Summary of changes
28/08/13
v0.1
Initial draft
Initial draft by Rob Farrow
28/8/14
v.1.0
Final draft
Final draft incorporates Phase 2 and Phase 3
plans
1.2 Distribution
Name
Organisation/Role
Author(s):
Rob Farrow
Research Associate
Reviewers:
Martin Weller
Co-Investigator
Patrick McAndrew
Principal Investigator
Approvers:
Patrick McAndrew
Principal Investigator
2. Purpose of document
3
OER Research Hub
Evidence Hub Development Brief
Date: 28.08.2014
The object of this document is to describe the technical work that is to be carried out
for the development of the Evidence Hub (EH) for the OER Research Hub project
(oerresearchhub.org).
3. Summary
The EH will build on the legacy systems developed during the OLnet project to
provide a platform for collaborative research, discussion, and dissemination of OER
news and reports. The value of this approach for focusing research and attracting
collective interest has already been demonstrated both within OLnet (olnet.org) and
outside the OER field, but the system needs to be developed and improved in a
number of ways, including doing more to facilitate engagement with the target
audience(s); a greater range of outputs (including new visualizations) and making the
most of automated functions. There will need to be an agreed workflow for
researchers using the Evidence Hub as well as on-going revision of effective use of
the Evidence Hub as a tool for both research and dissemination.
4. About the OER Research Hub project
The OER Research Hub (OERRH) is a Hewlett Foundation funded project being led
by the Institute of Educational Technology at the Open University (UK)
(http://www.open.ac.uk/iet/main/). The project provides a focus for research,
designed to give answers to the overall question ‘What is the impact of Open
Educational Resources (OER) on learning and teaching practices?’ and identify the
particular influence of openness. To achieve these aims the OERRH team are
working in collaboration with projects across four education sectors (K12, college,
higher education and informal) extending a network of research with shared methods
and shared results.
The project combines:

A targeted collaboration program with existing OER projects;

An international fellowship program;

Networking to make connections; and
4
OER Research Hub
Evidence Hub Development Brief
Date: 28.08.2014

A hub for research data and OER excellence in practice.
These activities are illustrated in Figure 1.
Figure 1: The activities of the OERRH project
4.1 Project objectives
It is aimed that by the end of the OERRH project’s research phase evidence will have
been gathered showing what works and when in terms of the use of OER in diverse
teaching and learning contexts, and that the research team will also have also
established methods and instruments for broader engagement in researching the
impact of openness on learning. The evidence generated via research collaborations
will contribute to this overall aim, whilst providing a legacy for policy change and
practice.
5
OER Research Hub
Evidence Hub Development Brief
Date: 28.08.2014
4.2 Project approach: collaboration and openness
The OERRH project operates on a collaborative rather than partnership model. It
features a dual research approach, combining both remote monitoring of publicly
available data and the development of research instruments which are applied
through a combination of direct and facilitated research.
The core group of collaborating projects is as follows:









The Flipped Learning Network
Gulf of Maine Research Institute
The Open Course Library (OCL) in Washington State
The Community College Consortium for OER (CCCOER)
Connexions/OpenStax College
TESS-India
School of Open/P2PU
Bridge to Success
OpenLearn
The OERRH approach to collaboration combines research with practical assistance
and it is hoped that this will bring benefits to all involved and will also provide an
enhanced flow of research data. The OERRH collaborations cover different sectors
and issues, these include: the opening up of classroom based teaching to open
content; the large-scale decision points implied by open textbooks for community
colleges; the extension of technology beyond textbook through eBook and
simulation; the challenge of teacher training in India; and the ways that OER can
support less formal approaches to learning. By basing good practice on practical
experience and research it is intended that the project will help tackle practical
problems whilst building the evidence bank needed by all.
Meeting the challenges of openness requires research strategies which take account
of the different aspects of OER impact. In the OER world, the way forward is
associated with greater transparency and sharing of educational materials and
research. Similarly, a more useful evidence base should emerge through taking
advantage of working openly and in collaboration as it allows for the pooling of limited
resources and greater scale and efficiency through co-ordinated action.
6
OER Research Hub
Evidence Hub Development Brief
Date: 28.08.2014
4.3 Project stakeholders
The project stakeholders will include (but are not limited to) The Hewlett Foundation;
The Open University; IET; collaborating projects; the OER Research Hub project
team; the OERRH Fellows, OER users and potential users (institutions, educators
and formal/informal learners) and the OER and open education movements. Table 1
provides a fuller analysis of the OERRH stakeholders.
Stakeholder
Interest/Stake
Importance
Project management team Ensuring the provision of the resources that will High
allow the project to meet its objectives.
Project researchers
High
Professional reputation; ensuring that work is
carried out to accepted methodological,
epistemological and ethical standards.
Project principal
High
Professional reputation; ensuring that work is
investigator & cocarried out to accepted methodological,
investigator
epistemological and ethical standards.
The Open University
Opportunities to develop a better understanding Medium
of the impact of OER on their base of teachers
and students; contribution to research profile
aligns with strategic priorities.
Institute of Educational
Medium
As part of the IET research portfolio, OERRH
Technology
contributes to maintaining a reputation for leading
research into openness in education and in
educational technology more generally.
Collaborating projects
A chance to outsource research needs and better Varied
contextualise projects’ own work. Good practice according to
can be identified and disseminated. The
institutional
national/international profile of OER projects can or
be raised.
operational
objectives
7
OER Research Hub
Evidence Hub Development Brief
Date: 28.08.2014
The Hewlett Foundation
The OERRH Fellows
OER users and potential
users (institutions,
educators and
informal/formal learners)
The OER and open
education movements
Getting value for funding input. Ensuring that
project stated objectives, outputs and outcomes
have been achieved. Furthering knowledge of
what works in terms of OER use and
development.
Opportunities for professional development;
networking & collaboration; novel research
opportunities
Improved understanding of OER impact should
facilitate the development of better quality OER
and pedagogies which make use of them.
High
Medium
Varied
Raising the profile of the open education
Medium
movement and of OER generally. Aggregated
research should be of benefit to the movement as
a whole.
Table 1: OERRH project stakeholders
More information about the project, its aspirations and techniques can be found on
the project blog, which is located at http://oerresearchhub.org/.
4.4 Role of the specified work in the project
The key outcomes of the OERRH project will be:

Research: establishing a greater understanding of the priority research
questions on the impact of OER on teaching and learning;

Collaborations: through the research and collaboration program, developing a
shared framework that helps both address these questions and enable their
research, allowing collaborating projects to benefit from development and
advice from the OERRH project team and linked-fellows;

Fellowships: through the fellowship program, enhancing collaboration and
communication across OER projects and providing the space and time for
understanding of OER-related research issues;
8
OER Research Hub
Evidence Hub Development Brief
Date: 28.08.2014

Establishing and developing an on-going, sustainable OER Hub at the OU for
informing decision making in this area.
The OERRH project comprises six work packages:






WP1: Management
WP2: Collaborative Research
WP3: Collaborations and Fellowships
WP4: OER Evidence Hub
WP5: Evaluation
WP6: Dissemination
The EH thus has a central place in the project as an online research database and
communication platform. The EH will function as a research ‘base’ with practical
outcomes that supports the OER and education communities in understanding the
issues and challenges.
A “How to conduct OER research” pack produced first for use in the collaborations
and then as a public, openly licensed document and section of the website. The
open-license research instruments featured in the pack will include schedules for
semi-structured interviews, guidance notes for conducting focus groups (including
hypothesis-related topic guides), a bank of survey questions and guidance for the
design of surveys incorporating these questions, an analysis strategy for analysis
and interpretation of the quantitative and qualitative data collected through the survey
questions, focus groups and interviews, and identification of quantifiable factors
either gathered as secondary data or direct analytics;
The project logic model is shown in Figure 2.
9
OER Research Hub
Evidence Hub Development Brief
Date: 28.08.2014
Figure 2: The OERRH logic model
10
5. Rationale
The OER Evidence Hub will build on the legacy systems developed during the OLnet
project to provide a platform for collaborative research, discussion, and dissemination of
OER news and reports. The concept of the Evidence Hub developed as a response to
the need for technology platforms which could support collaborative knowledge building in
an emerging discourse [http://evidence-hub.net/]. There is an identified need for a web
service where information about OER impact can be retained and recalled but which also
helps users to make sense of research data, offering comparison across cohorts and in
relation to international context. OER research is often fragmentary, or doesn't allow easy
generalisation across contexts OERRH is guided by a series of research hypotheses
which are investigated through both primary and secondary research. Our understanding
of these hypotheses and their validity is to be supported by evidence gathered at the right
level of granularity from a plurality of sources.
The EH needs to be able to present information to different audiences, and to enable
users to navigate information that is relevant to them. This means that it needs to be able
to help people to make sense of complex data and show how it might be relevant to them.
It is both a tool for the research team that provides insights into 'the bigger picture' and a
way for those involved in the open education movement to better understand their own
contexts. OER advocates remain in need of a tool which will enable them to present a
convincing evidence case for policy change and changes in practice.
OERRH will support this by structuring the evidence and making arguments explicit. The
‘researcher pack’ will make OERRH research instruments available openly so that others
can contribute to the evidence base.
By mapping OER evidence and OER initiatives round the world, the EH will help potential
networks of support and collaboration to become aware of of one another. Another
important function for the EH is as a place for disseminating the work of the OER
Research Hub project. This means it will need to be able to accommodate a range of
different types of raw and redacted data as well as multimedia. It must also conform with
the guidance set out in the project Ethics Manual (http://oerresearchhub.org/projectdocuments-archive/reports/oerrh-ethics-manual/ ).
11
Here is a summary of comments generated by the OERRH team about what the EH
should strive to be:
Easily navigable content
Easy answers for funders
Clear evidence relating to OER
impact
A presence that enhances OU
reputation
A bank of resources for others to us
Evidence-informed claims
Guidance for other OER researchers
A catalyst for further research
‘Our story’
A narrative about changes in
OER/OEP
Customised interfaces
Signposting missing evidence
Geographical dataset
A better understanding of each
hypothesis
Information which maps to
hypotheses
Help & evidence on OER issues
Infographics for each collaboration / hypothesis
Table 2: EH Design Guidelines
6. Specification
The platform and/or technical architecture have yet to be decided upon, but a number of
system requirements have been identified (see below).
6.1 File Types
The Evidence Hub should support both stored and embedded content of a range of types
including databases, text, audio and video. HTML pages should be able to accommodate
these. Embedding content should be encouraged where possible, but we will need
storage space at least for any data we add (e.g. survey results) which will need to conform
to data protection policies. We should enable users to sort content in intelligent ways
which suit their strategic needs. We should also ensure that it is possible to filter evidence
by sector (School, Further Education, Higher Education, Informal Learning) and potentially
by stakeholder.
12
6.2 Information Architecture
In OER Research Hub, a series of research hypotheses provide the focal points and
provide a way of establishing the relevance and value of a particular piece of evidence.
They are provided here with a shorthand tag.
a) Use of OER leads to improvement in student performance and satisfaction
[impact]
b) The open aspect of OER creates different usage and adoption patterns than
other online resources [patterns]
c) Open education models lead to more equitable access to education, serving a
broader base of learners than traditional education [access]
d) Use of OER is an effective method for improving retention for at-risk students
[retention]
e) Use of OER leads to critical reflection by educators, with evidence of
improvement in their practice [reflection]
f) OER adoption at an institutional level leads to financial benefits for students
and/or institutions [finance]
g) Informal learners use a variety of indicators when selecting OER [indictors]
h) Informal learners adopt a variety of techniques to compensate for the lack of
formal support, which can be supported in open courses [support]
i) Open education acts as a bridge to formal education, and is complementary, not
competitive, with it [transitions]
j) Participation in OER pilots and programs leads to policy change at institutional
level [policy]
k) Informal means of assessment are motivators to learning with OER
[assessment]
The short form of the tagging system binds relevant content together. The advantage of
doing this in relation to a hypothesis is that it will lets us quickly draw upon a body of
evidence that all relates to that hypothesis even if the content is diverse.
13
6.3 Data and Metadata
Content for evidence nodes at the prototyping stage is to be provided by OERRH
researchers. There are existing data sources which will be used for seeding the EH,
including data currently stored on SurveyMonkey and imports from the legacy EH from the
OLnet project.
Evidence entries themselves are likely to be mostly text but should support embedding of
video, audio and other types of content through HTML. For any given piece of evidence
we expect there to be key metadata which provides the basis for navigating, synthesizing
and presentation of information. For example, it should be mandatory to tag any given
entry with a relevant hypothesis.
Here is the anticipated structure for an evidence record:
Aspect
a
b
Type
Format(s)
Title
Copy
Text
Text/HTML
Radio Buttons
Radio Buttons
Geotagging/GPS
Radio Buttons
g
Relevant Hypotheses
Polarity (positive/negative)
Location
Sector (SchoolK12/College/Higher
Education/Informal)
Link(s)
h
Citation
Text
c
d
e
f
URL
Notes
Likely to be entered
into a simple HTML
editor which
supports embedding
Mandatory
Mandatory
Supports mapping
Further information
for this entry
Academic reference
relevant to this entry
(may or may not
include URL)
Table 3: Structure of Evidence
It may be desirable that some sort of 'featured image' functionality also be included in the
data description. The data model should be able to support entries that are only a few
sentences as well as more detailed 'case study' style entries which might include
multimedia objects or embedded content.
The inclusion of geographical references is important for the user interface that the EH will
be modeled upon (see Appendix B). It should also be possible to add other kinds of
14
entries to the EH, such as OER projects around the world, news highlights, or events.
These may be added at a later date so the system used needs to be open to data which is
unanticipated at this point.
6.4 Authentication
The public prototype version of the EH need not have a fully working system of user
accounts, though this would be desirable if time and budgets allow. Open standards for
authentication which facilitates sign-in through social media accounts (Twitter, Facebook,
Google, Yahoo, Wordpress, etc.) or OpenID are preferable.
We ultimately want signed-in users from outside the team to be able to add evidence o the
database. Using IP addresses or on-device GPS should help us to locate their
contributions on the map and this functionality should remain a possibility regardless of
which platforms or programming languages are used.
Ultimately, administrative privilege should include the facility to edit/delete any aspect of
any entry and moderate comments made, and to create/edit/delete static pages and other
elements of the site.
6.5 User Interface
The site design should be clean, uncluttered, intuitive, and consistent with the project
branding and with the guidelines provided by The Open University. It should use the
existing OERRH logo and branding guidelines which will be supplied to the successful
candidate.
The site will need a few static content pages with information about the project with
contact information. The homepage for the EH should contain some general information
about the project extracted from oerresearchhub.org and possibly displaying the RSS
feed for this site and for our Twitter account (OER_Hub).
Users should be able to browse collected evidence and OER organizations through a
world map similar to Google Maps and preferably which makes use of ‘open’ platforms
such as the OGC (2013) standards. It should also be possible to review the evidence
associated with a particular hypothesis, read case studies, browse by sector, navigate
through tags and/or make use of search box. All views should be working fro the same
data set.
Content will be created by members of the OERRH team which means that users should
be able to create new evidence entries with the structural elements described in 6.3.
15
At some point in the future we may wish to introduce the ability to form interest groups
from users, or to ‘favourite’ particular pieces of evidence. We will also introduce the
functionality for sharing evidence nodes directly to social media.
6.6 Usability & Accessibility
The work should conform to Web Content Accessibility Guidelines 2.0 (W3C, WCAG20).
The OUICE template (http://www.open.ac.uk/webstandards/v3.0/index.php) may provide
some guidance in this respect although the EH design should be consistent with OERRH
rather than Open University branding.
The end-user facing site developed for this project will be tested against the following
platform/browser combinations:
Desktop:
Mac (OS X)
Chrome (latest release)
Mozilla Firefox (latest release)
PC (Windows 7)
Chrome (latest release)
Mozilla Firefox (latest release)
Internet Explorer 9.0
Internet Explorer 8.0
Safari (latest release)
Mobile:
Android
iOS
Chrome on Samsung Galaxy S2
Safari on iPhone 4
Chrome on Nexus 7
Safari on iPad Mini
Chrome on Samsung Galaxy Tab 2
Safari on iPad 2
Table 4: Target Platforms/Browsers
6.7 Analytics
The EH should include calls to a Google Analytics account (details TBC) with all creation,
editing and deletion of records logged in the database. No facility for accessing this log
need be included in this work. The log is intended for potential future expansion of
functionality and for reporting and analysis. The following information should be captured:
 Date and time of the operation
 The operation performed
 The unique reference for that entry
 Relevant user ID
16
7. Delivery
Work on the EH needs to begin immediately in anticipation of upcoming project
dissemination activities which will publicly present research data from the project. It is
expected that there will several phases of development activity with discrete elements.
Phase I refers to the public prototyping of the EH and Phase II comprises distinct
elements of functionality (several of which may be combined in future commissioning).
Development work will take place on local developer machines with regular updates to the
prototype EH which will be hosted on IET servers.
7.1 Phase I – Public Prototyping
Thus far, a basic prototype which aggregates evidence gathered within the project has
been written in JSON (Javascript Object Notation) which shows how our data might be
structured in ways which would support aggregation and database queries. JSON is a
text-based open standard which is designed for human-readable data interchange and is
compatible with a range of programming languages and web browsers. Our experiences
of working from an evidence dump to narratives about each hypothesis shows how the
process might work. The prototype includes information about hypothesis; a summary of
the +ve/-ve evidence count; a list of evidence entries and key ideas.
The first phase of consultancy will build on this in order to deliver the following (including
the creation of any required APIs (Application Programmable Interfaces), GUI (Graphical
User Interfaces) and web interfaces for these tools):
(1) A demonstrator evidence hub
capable of supporting:
 browsing and displaying information about the OERRH hypotheses on a map
 navigation, filtering and presentation of data by variables and hypotheses
including:
 a simple user interface to allow evidence to be entered into the system
 administrative functions
 a summary page for each ( or at least some) hypotheses which presents
aggregated evidence
 embedding of diverse multimedia content
(2) A system of structured files or database content enabling functionality which can
be expanded upon in future
 conversion of OERRH research data into seeded content
 incorporation of imported data about OER projects (e.g. extracted from the OLnet
Evidence Hub)
17
 maintenance of data sets
(3) Data visualisations indicative of project research data, including surveys
 still images suitable for presentation
 prototyping/descriptions of visualisation tools which could be added to the site
(4) Suggest design directions and provide a debriefing meeting with OERRH in
person or remotely to help determine the next stage in the development
(5) Write at least 2 blog posts working with the OERRH workpackage leader to help
brief on the hypothesis hub and provide support to presenting the OERRH hypothesis hub
prototype at the Open Access Unconference (San Jose, CA) and OpenEd 2013 (Park
City, UT) in October-November 2013.
Planned Start Date: 9 September 2013
Planned End Date: 9 November 2013
7.2 Phase II – Iterative Development
The focus for this phase of work is substantive development of site which will enable fuller
dissemination of research data. This has been broken into three distinct sections of work
so that they can be commissioned independently or together.
Phase II.1 (Jan 2014)
(1) Building a system to display and explore survey data that has been generated by
the project. This data is exported from Survey Monkey and can be prepared by the
OERRH research team once a protocol is agreed. Mash-ups of existing and public data
should be supported in future.
(2) Tools for helping users to make sense of survey data. These could include filters
for isolating different variables within the data set and the ability to run simple queries that
will interrogate the data. Data Visualisations that will present survey data might include
showing trends plotting data against a heat map; maps which show results by country or
state; interactive pie/bar charts and pivot tables.
(3) A (front-end) user interface for entering data into the system. A pop-out UI should
facilitate data entry for the evidence, project, location and policy post types. This will
initially be for the use of the OERRH team but will likely be rolled out to other users in the
future. It should include a feature which warns the user when they may be entering
information already in the database by comparing the citation with older posts.
18
(4) Introduction of a new polarity status of 'neutral' to be used exclusively with the
'Uncategorised' evidence category. This will then be used to curate evidence which is not
presently connected with a hypothesis but which may be important for future hypotheses.
(5) Refinement of the citation field which will detect when a URL has been provided and
automatically provide links for these while retaining the option to add a textual academic
citation without automatically converting this to a URL.
Phase II.2 (Feb 2014)
(6) Installation of plugin(s) to support improved integration with social networks:
sign-in, share buttons, community-building (e.g. OpenID, BuddyPress).
(7) Installation of a plugin to facilitate curation through RSS (e.g. FeedWordPress) that
will facilitate queueing of evidence for entry from bookmarking/curation services.
(8) Redesigned pages for the exploration of hypotheses. These pages need to be
structured and developed in such a way as to make it easier for the user to preview and
explore hypotheses with less scrolling and less text immediately visible on screen. A
summary page for each hypothesis comprised of panels showing a mix of visual
representations of data (e.g. hyperbolic tree; scales of balance) as well as textual
summaries (e.g. hypothesis description, total of evidence gathered, featured evidence,
most-shared evidence) could work well for this. These summary pages )(or infographics)
should be print-optimised so that users can generate a quick report on the latest data.
(9) Redesigned pages for the exploration of evidence which help the user to make
sense of the evidence as it is presented according to polarity and/or sector rather than
providing a sorted list of all evidence posts meeting the criteria selected.
(10) A bookmarklet that will facilitate data entry after being added to internet browsers,
providing a way to directly import basic information about a website into the data entry
interface (similar to Scoop.It).
Phase II.3 (Mar 2014)
(11) A method for publishing customised map 'stories' about research data (e.g.
Mapbox) which can be used to present a case relating to a particular sector or hypothesis
by taking the user on a path through a collection of evidence, projects, and/or policies and
providing additional commentary through an additional panel (See
https://www.mapbox.com/tutorial-sherlock/ for an example.) It should also be feasible
that this method be used in relation to scraping tweets and displaying them on a map.
(12) A redeveloped policy map which can show the range of (geographical) influence
that the collected policies have in addition to (potentially multiple). This could work like a
heatmap, showing where there are areas with a lot of active policies. Policies need to be
able to apply to several (or all) sectors simultaneously.
19
(13) Creation of RSS feeds for evidence, project and policy post types.
(14) Re-specification of the CSS active on the site to meet project branding guidelines.
(15) Installation of a plugin for search engine optimization to improve the visibility of
the site. This should be configured so as to make the custom post types available for
search.
In addition and alongside these packages, the developer(s) should write:
(16) Three blog posts (to be written in February, March and April) on the OERRH project
website to document the work being done and the challenges/solutions encountered.
A audit of site usability and accessibility will also be carried out as part of Phase II – this
will be conducted by specialists within IET. This may identify some bugs or fixes which
should be carried out in this phase of development.
Please note that developmental work relating to the Hewlett OER Map project may be
carried out alongside the work described here if the bid id successful. This will be
described in a separate document.
Planned Start Date: Jan 2014
Planned End Date: April 2014
7.3 Phase III – Legacy
This phase of development will address any outstanding issues with previous
development as well as introduce features designed for improving the user experience as
well as improving the long-term sustainability of the site and its services.
(1)
Debugging/finalization of work specified in previous phases of development,
namely:
 Open ID sign-in and sharing from within records
 Specification of the CSS (especially BLOCKQUOTE) to meet project branding
guidelines
 RSS feeds for the different custom post types (evidence, policy, project)
 Buddypress social network plug-in
 Installation of an alternative search plugin
 Ensuring that map pointers don’t bunch so close together that they obscure one
another (e.g. Nairobi)
20
(2) Redevelopment of the interface for exploring OERRH survey data in accordance
with the provided wireframe. The database needs to be replaced with an updated and
cleaned version of the survey data. (See Appendix C.)
(3) Creation of a second ‘Tweetmap’ based on the #oerrhub and #oerrh hashtags and
the @oer_hub handle – the relevant TAGS archives are available from
https://docs.google.com/spreadsheet/ccc?key=0AkiDQSojJ9adGFaQkNpMmVlT2RqbWFYd3pBQjhia1E&usp=drive_web and
https://docs.google.com/spreadsheet/ccc?key=0AkiDQSojJ9adDdMdXJMT3ZvMlByZ0ViakFfOG9fc2c&usp=drive_web#gid=36. This map
and the existing #oermap Tweetmap should be integrated into the native mapping system
rather than hosted on Google maps/fusion tables. It I desirable that it is possible to
manually add entries to this map (URL + custome post type so as to use it for the basis of
a map that show the connections and influence of the OERRH project.
(4) Refinement of the UI to allow users to read full records in overlay style without having
to leave the map view. (See http://www.mira.org.br/ for an example of how this might
work.) Alternatively, the map ‘pop-out’ box can be redesigned to compress the metadata
(Type/Hypothesis/Sector/Polarity/Locale/etc.) into a banner (which could also include
quick links for sharing) or visual representation and expand the box to accommodate the
full text record.
Other UI/display tweaks:
 Colouring the central circle of evidence nodes according to relevant sector
 Legends for all maps
 Installation of the following Wordpress theme(s):
o Forever (https://wordpress.org/themes/forever)
(5) An interface/shortcode for creating new custom post types (or another solution
such as tagging) so that it will be possible for the project team to create new bespoke
maps in the future as data becomes. Four new maps are anticipated as part of this phase
of development:
(a) Building a new map view for hosting data from the TAACCCT grantees
(http://www.doleta.gov/taaccct/pdf/TAACCCT_One_Pagers_All.pdf). This map will
be USA only and the data will provided by one of the OERRH project fellows. This
may involve the creation of a new custom post type but specific metadata
requirements are TBC. There are 57 grantees spread throughout the USA.
(b) Building a new map view based around a series of case studies of notable OER
initiatives (which could be linked to ‘projects’ in the ontology). This map will present
detail on 8-10 initiatives through a structured narrative and may be structured as a
simple blog post with geo locations. (Content provided by project team.)
21
(c) Building a new map for recording MERLOT impact studies; these will include a
simple ontology no special metadata requirements (similar to the OER Policy Map).
Data will be provided by a forthcoming call to the MERLOT user base. It is
expected that there will be up to fifty of these to be displayed on either a global
map or map of North America.
(d) Building a new map for displaying impact stories for the TESS-India project.
This would also have a simple data structure, but one idea that has been discussed
for the India map is the idea of parent/child relationships to show
primary/secondary impacts that have resulted from interventions. It would be good
if there was an intuitive way to explore this (e.g. click to reveal).
Suggested ontology for these maps:
 Name of entry
 Geographical location
 Impact statement
 Sector
 Additional fields as required by map type
There may be a couple of custom fields required for some of these. (E.g., the India map
may include an ‘interaction type’ field.)
Some data collected for (a) – (d) might qualify as evidence in the main map – in which
case it will be added to the OER Evidence Map separately by the project team.
(6)
Full documentation of API end points and facilitation of import/export of site
data though API. Users with appropriate permissions should be able to interface with the
database in such a way as to facilitate the reuse of open data from both the maps and the
survey data explorer. It should also be possible to interface with open data from the
Hewlett OER Map prototypes (MIRA, ISKME, HBZ) and import, publish or cross-reference
this with the OERRH maps.
(7)
Introduction of a digital badging system (e.g. BadgeOS) which would allow
awards to be made for:
 evidence contribution
 voting on evidence entries
 completion of the OERRH School of Open course (begins Sept 2014)
(8)
Suggesting, agreeing upon and implementing technology/ies for presenting
narratives around the OERRH project and its impact (e.g.)
 Aesop Story Engine http://wordpress.org/plugins/aesop-story-engine/
 Scrollkit http://www.scrollkit.com/ (nb. Bought out by Wordpress)
 Killing Lincoln (NatGeo) http://killinglincoln.nationalgeographic.com/
 Bear 71 http://bear71.nfb.ca/#/bear71
22
 Docubase.mit
(9) Installation of Wordpress Ubuntu font and any other necessary plugins.
 Slider https://wordpress.org/plugins/slider-image/
(10) (a) Importing 416 projects (or ‘companies’) from the Little Bird data
by Lumen Learning into OER Projects Map. (Relevant fields are ‘’Full Name’;
‘Description’; ‘Related Website’ and ‘Location’.)
provided
(b) The larger data set also shows influence on Twitter and could
potentially be used for data visualization(s). These should attribute the
data to
Little Bird (http://getlittlebird.com/) and Lumen Learning
(http://lumenlearning.com/).
(11)
Adding ‘Sector’ to the metadata for projects
(11)
Optimization and debugging of final code.
(12) In addition, the developer(s) should write one blog post each month to document
the work being done and record the challenges encountered and solutions that were
adopted.
Planned Start Date: Jul 2014
Planned End Date: Oct 2014
7.4 Resources
The OERRH Development Consultant is required to address the need for an online
implementation giving access to findings linked to the hypotheses used in the OERRH. A
series of consultancies are planned to achieve this in place of the Developer post
indicated in the proposal. This reflects an unsuccessful round of recruitment for that post.
A prototype system is required for demonstration in November 2013. The system
requirements are well specified and data is in place to enable the demonstrator. The initial
consultancy will focus on this task. Further consultancies may follow in order to complete
the development requirements set out in the proposal. The Hypothesis Hub work can also
build on the phase 1 evaluation work that is reporting in September.
In order for debriefing to be effective potential expenses are included to cover travel
to/from MK campus on occasions.
23
Maximum Fee: £6000
Estimated Maximum Expenses: £1000
2008/2009
£Resources made available for subsequent phases of development will be subject to
budgetary review.
7.5 Liaison and Reporting
To ensure clarity of communication, OERRH will provide a single point of contact: Dr. Rob
Farrow (rob.farrow@open.ac.uk) who is a Research Associate on the project and leading
the relevant work package.
Note that at times it may be necessary to liaise and communicate effectively with the
project academic and non-technical staff to manage technical and non-technical project
objectives. The developer is expected to document the development of the software and
ensure that the technical deliverables of projects have gone through appropriate quality
control and quality assurance process.
7.5 Intellectual Property
All work created as part of any consultancy is to be owned by OER Research Hub, and
any code is granted an indefinite non-exclusive license for use by The Open University.
OERRH is committed to a policy of open dissemination, as set out in the project Ethics
Manual. Specifically, all project materials for which it is thought relevant are to be
released openly on a CC-BY Creative Commons licence. This may ultimately include
code and copy generated for the development of the EH.
7.6 Legacy
OER Research Hub is funded until September 2014 and website development is likely to
continue as we work toward project completion. Once the project ends the site could be
able to be managed with a minimum of input and should continue to present the evidence
base constructed during the project while remaining open to new hypotheses and new
data.
24
Appendix A: Background Information on OLnet Evidence Hub
The Open Learning Network (OLnet) was a project based at The Open University’s
Institute of Educational Technology (IET) and Carnegie Mellon University. In addition to
supporting and commissioning research into OER and their impact, OLnet comprised a
fellowship scheme and had an explicit focus on developing software tools which would
facilitate and support collective intelligence and identify potential solutions to the
challenges facing the OER movement.
OLnet’s mission was to ‘to gather evidence and methods about how we can research and
understand ways to learn in a more open world, particularly linked to OER, but also
looking at other influences.’ The idea of gathering evidence for OER impact was thus at
the core of the project and continues to inform work on OER Research Hub. OLnet
included a number of academics involved in different aspects of OER research. The
evidence hub was envisioned as a tool to disseminate their work while linking key aspects
to wider discussions and events that were taking place.
The decision was taken to develop an evidence hub that could accommodate a wide
range of possible forms of evidence but in a way that allows sense-making to take place.
Why might we need to take such an approach to open education? One of the reasons
that it might be appropriate to investigate open education through collective sense-making
might be the fragmentary and partial nature of much of the evidence that is available.
Where evidence does exist, it is often tightly focused on a particular initiative rather than
on open education as a whole.
The designers of the OLnet Evidence Hub used the concept of ‘contested collective
intelligence’ (De Liddo & Buckingham Shum 2010; De Liddo & al 2012). This approach
recognises that most ‘knowledge’ is contested, and different backgrounds, cultural
assumptions and intellectual traditions. It was used to develop a social-semantic web
application for human annotation and knowledge mapping called Cohere
(http://kmi.open.ac.uk/technologies/name/cohere).
Information is classified and assigned to the following nodes:

Issues

Potential Solutions

Research Claims

Evidence

Resources

Organisations/Projects
25
The benefit of breaking information down to this level of granularity is that it can promote
nuanced consideration of the relationships between particular claims or pieces of
evidence. Users can add nodes and make new connections as well as provide argument
or counterargument and vote on the plausibility of claims or solutions. The OLnet
Evidence Hub was thus designed to provide the following functions:

Explore and debate the key challenges for the Open Educational movement.

Allow the community to express the importance of each challenge (challenges can be
promoted or demoted)

Add a description of an OER project, including geographical location

Post, explore and discuss OER related issues and questions

Propose solutions to tackle the major challenges facing Open Education

Share relevant evidence and Web resources for the OER community to contribute to
the evidence base of OER impact on teaching and learning

To make and investigate new claims of OER effectiveness
Cohere allows for some very fine-grained distinctions in textual analysis. Researchers on
OLnet used it to analyse 125 reports from previous Hewlett grantees working with OER.
This thematic analysis was used to seed the Evidence Hub, to create the tagging system,
and to identify a number of ‘issues’ and ‘solutions’ which were used. It also provided
details about organizations which initiated the ‘mapping’ process. However, the full
version of Cohere was not thought to be one which with users would engage, and so a
simplified version was used. Envisaged as a ‘vertical’ version of a horizontal tool, this
version trades nuance in expression for relative ease of use (Buckingham-Shum, 2012).
The knowledge-mapping functions of the OLnet Evidence Hub are largely provided by a
tagging system which can indicate that discrete nodes of information have some common
connection.
26
Fig. 1 – OLnet Evidence Hub Themes/Tags
During the implementation phase a new category of ‘Key Challenges’ was introduced in
order to provide a clearer focus for the information that had been entered. The evidence
model went through a number of revisions over the course of the project. Here is the final
(simplified) data model for the OLnet Evidence Hub:
Fig. 2 – OLnet Evidence Hub Data Model
At the time of writing there are the approximately 1500 records in the OLnet Evidence
Hub, most of which were entered by IET staff:





300+ OER projects and organisations
129 research claims
79 OER issues
323 evidence nodes
553 resource nodes
It’s worth noting that, with the exception of someone expressing a view in a comment, the
information contained within the hub can only be metadata: no materials from the
resources were stored within the hub and only items with a URL can be added as
resources. This arguably lessened the value of the hub for users who want a ‘one stop
shop’ for OER information and engagement.
27
Given the number of possible connections it is possible to make between nodes, the
resulting database is very complex, which resulted in some slow loading times.
The work carried out using the OLnet Evidence Hub resulted in a number of research
outputs. These included two conference presentations (De Liddo et al, 2012; McAndrew
et al, 2012b) and a journal paper (McAndrew et al, 2012a). The research into knowledge
mapping also provided outputs for KMi although these were not necessarily part of OLnet.
(e.g. De Liddo et al, 2011). The key challenges themselves were well received among the
OER community and may have acted as a focal point for discussion and planning.
The website was promoted at a number of high profile events, including Open Ed 2011,
Cambridge 2012 (jointly organised by OCW Consortium and SCORE (Support Centre for
Open Resources in Education), and a number of OU events and webinars (including the
Learn About Fair and OpenOpen: Sharing Research and Practice). OLnet Fellows were
also encouraged to interact with the service. While these promotional activities generated
a respectable level of interest in the site, users never really engaged with the site in
numbers which would authenticate claims to have drawn on collective intelligence, and so
the Hub was primarily seeded with information brought in through email lists, RSS, social
networks and curation services. Perhaps the most effective use of the tool was in
analysing discrete documents, such as the JISC OER impact report or the UNESCO /
Commonwealth of Learning 2010 policy announcement. Summarising lengthy documents
like these is a service which is often valued by those with time pressures, and which may
also have pedagogical benefits for researchers. However, this is a time-intensive task
which requires skills in analysis and communication. Furthermore, the purpose of the hub
is not primarily to provide textual analysis.
Arguably the OLnet Evidence Hub did not meet its aspirations as a dissemination tool.
Very little of the research output of the project is recorded there, and little user activity
which would indicate adoption by the OER community. However, recent site statistics are
more encouraging. The site presently receives around 7-8 visitors a day, most of whom
are new to the site and from either the UK or the USA.
The OLnet Evidence Hub was not subjected to a formal evaluation, though a considerable
amount of user testing took place. Users included IET staff and OLnet fellows as well as
developers from KMi. The Learning and Teaching Development Team also carried out an
audit for accessibility and usability.
Most users found it difficult to orient themselves with the interface, and as these
representative comments from the different evaluation activities show they typically did not
feel inclined to continue using the site.
 The process of data entry was onerous, with multiple forms to complete
 Some users felt unsure what to do when they had linked nodes together
28
 Some felt that there should be additional ways of making sense of the data while
others found the present ontology too complicated
 Some felt that the purpose of the site was unclear: resource? tool? news site?
 Excessive terminology was identified as a barrier to use
 The design was often thought visually unappealing, with too many text lists and
windows
 No clear process for evaluating evidence or claims without large numbers of users
 Some errors in loading maps (possibly Java issues)
 Lack of a definition of ‘evidence’ and other key terms
 Hard to understand the thinking behind the website
 Some 404 errors and slowly loading pages
 Visual representations of arguments add little value
 Not always clear who is making a claim: the writer or the cited source
 Some issues surrounding ownership of materials on the site
The OLnet Evidence Hub was intended to provide a software structure which could
support sense-making discourses around OER. It was supposed to augment everyday
communication mechanisms by crystalizing existing discourse and arranging it in ways
which allow collective intelligence to filter claims while preserving the contested nature of
knowledge. The site has never had a sufficient number of users who acted in the ways
envisioned by the designers, so the approach of the designers has not necessarily been
invalidated.
However, the slow take-up of a tool for which there is a recognised need within the OER
movement indicates that the approach has not been appropriate for the audience we need
to reach. It particular, it seems evident that any user unfriendly interface will not
encourage community engagement, and users need to feel that they are receiving value
from their activity on the site. There also needs to be a clearer sense of what the
objectives of such a service should be.
Appendix B: Summary of UNESCO OER Mapping Project
Undertaken at the same time as the main body of work on the OLnet Evidence Hub – and
working towards a commonly identified need – the UNESCO OER Mapping Project
attempted to record the landscape of OER use to support communication, connection and
collaboration. In 2005 the International Institute for Educational Planning (IIEP) of
UNESCO convened an international OER community with the objective of raising
awareness of OER. The UNESCO/Commonwealth of Learning Chair in OER at
Athabasca University was created with a focus on promoting OER at institutional, national
and international levels.
29
There was a feeling that “[t]he growth of the OER movement over the past decade has
meant that it has become more and more difficult to have an overview of OER initiatives
globally, and even locally. A map of the “OER world” could illuminate its scope, making
initiatives or projects more visible and enabling connection, collaboration and
partnerships” (D’Antoni, 2012).
The process of reflecting on the topic began with a period of collaboration between the AU
OER mapping project and the OLnet project at The Open University (OU), a partner
institution of the UNESCO Chairs in OER. The interaction in the Athabasca University
OER community of almost 900 members was organised over a three-week period with a
suggested focus for each week:
1. What could an OER world map look like?
2. Could a world map be built collaboratively?
3. Reflection and next steps
There was some reflection on what was needed – whether it was a geographic map or a
database of OER projects. In fact, a map is generated from a database, but these
remarks point to the fact that the community may wish to have various representations
and these may be textual or visual. For example, it would be useful to have a listing of
initiatives by country, or by initiative language.
The discussion which took place around this time led to a number of recommendations
about the design of a mapping system. It could be used to:

make visible the scale and geographic range of OER initiatives

broaden the OER scenario beyond the well-known projects and players

trace the life cycles of OER projects over time

identify a global roster of OER experts

enable collaboration and partnerships among OER people and projects

awareness raising to educate those outside the OER community

communication and advocacy with policy and decision makers

inclusive community-building based on a common goal
It was noted that the mapping process should be simple and sustainable with the capacity
to draw on a range of sources. With this in mind, the following ‘essential’ data was
requested for initiatives so that they could be added to the map.
1 OER initiative name
2 OER initiative web site
3 OER initiative working language(s)
30
4 Contact person name
5 Contact person email
6 Latitude and Longitude
More complex metadata requirements were considered, but ultimately rejected in the
discussion as too onerous on the user. It was suggested that case studies be attached to
entries where possible.
Although the UNESCO OER Mapping Project was never formally implemented, though a
prototype version which mapped information from OCWC was used for demonstration
purposes.1 However, the process of consultation led to a number of central
recommendations (D’Antoni, 2012):
 The first point of consensus was that Free and Open Source Software (FOSS) be used
for the mapping exercise – for both the development of a database and for any visual
representations of the data.
 Information should be described by the appropriate metadata, and classified using
open international standards.
 It is important to have a centralised function for final quality control and loading of the
data
 Mapping should combine local data collection with centralised quality control
 Tools should be flexible enough to be adapted by different communities for different
purposes
 Mapping services need to think about how visually impaired users might access
information
Here is what the report concludes about information visualization and its value for an OER
mapping system:
“A visual presentation of OER initiatives worldwide could communicate different stories or
messages, and a well-drawn picture conveys its message largely without words. Maps
can present different combinations of data to give different pictures or perspectives.
Geographic representation of projects by country would present one picture, while
representation by the working language of projects would present another. A timeline
created from information on the launching of projects would indicate, for example, periods
of intense development and highlight new initiatives… The different techniques used in
mapping – such as colour coding or size of marker – all contribute to conveying a
message, and there are many options for presenting data visually.” (D’Antoni, 2012)
1
http://oerworldmap.oerknowledgecloud.org/index.php?lat=20&long=0&zoom=2&filter=1
31
Appendix C: Wireframe for Survey Data Explorer
SECTOR
ROLES
Educators Formal
Students
COUNTRY (Optional)
Informal Learners
[Drop down list]
GENDER
AGE
<15
15-18 19-24 25-34 35-44 45-54 55-64 65-74 75<
DISABILITY
EDUCATION No
for
Vocational
College
Undergraduate Postgrad
QUERY
Select one of the following options: (see below)*
TEACHING & LEARNING PROFILES
OER BEHAVIOURS
CHALLENGES & SOLUTIONS
OER IMPACT
32
GO
[OUTPUT]
CHART
TABLE
Or maybe this works better given the template…
Chart
Table
Notes on Data Explorer







In the database for the survey data explorer
(ALL_Compiled_SurveyData_June14.csv) row 1 indicate filters (those variables
relevant for the checkboxes).
All checkboxes are selected by default – but users can deselect variables to hone
the sample.
Row 2 presents the key areas used for running the queries.
Row 3 gives question wordings.
The interface for selecting query areas could be a panel where you select one, or a
drop down box, or something more creative.
Users will just deselect any variables they need to in order to refine the sample,
then select a query area and press ‘Go’. This will update the output chart(s) and
table(s) as well as the marker map. The other elements are anticipated to be static.
Further queries are generated by changing the
33
*Query
Areas
Here are the areas that comprise each query area. Each contains 2-5 relevant questions
as indicated below. The output should include all of the data for that query area according
to the filters that have been selected/deselected as well as the questions asked. (The
code in parenthesis shows where to look up this data in the file
ALL_Compiled_SurveyData_June14.csv.)
TEACHING AND LEARNING PROFILES
 Subject areas (AG2)
 Internet access (AY2)
 Reasons for using OER (IK2)
OER BEHAVIOURS
 OER use patterns (CK2)
 Types of OER used (CQ2)
 Purpose(s) of OER use (DF2)
 Repositories used (DW2)
 Factors affecting OER selection (GZ2)
CHALLENGES AND SOLUTIONS
 Challenges faced (EN2)
 Support techniques used (HQ2)
IMPACT
 Impact on teaching practice (FD2)
 Impact on learners (learner perspective) (FP2)
 Impact on learners (teacher perspective) (GC2)
 Prediction of future behaviour(s) (ID2)
 Impact on motivation (IV2)
Outputs should include a world map which updates according to the sample that has been
selected as well as simple graphs and tables showing the data relevant to the query that
has been entered.
References
Buckingham-Shum, S. (2012). Evidence Hubs: a new way to interact with idea networks.
Available from http://technologies.kmi.open.ac.uk/cohere/2012/01/27/evidence-hubs-anew-way-to-interact-with-idea-networks/.
34
D’Antoni, S. (2012). A world map of Open Educational Resources initiatives: Can the
global OER community design and build it together? Summary report of an international
conversation: 12 – 30 November 2012.
De Liddo, A., Buckingham-Shum, S., McAndrew, P., & Farrow, R. (2012) “The Open
Education Evidence Hub: A Collective Intelligence Tool for Evidence Based Policy”.
Proceedings of Cambridge 2012: Innovation and Impact – Openly Collaborating to
Enhance Education, OCW Consortium and SCORE, Cambridge, UK, April 16–18 2012,
Milton Keynes, The Open University, pp. 396-404, http://oro.open.ac.uk/33640/.
De Liddo, A., Sandor, A., & Buckingham-Shum, S. (2011). Contested Collective
Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study, CSCW
Journal (21(4-5), pp. 417–448
De Liddo, A; Sándor, Á; Buckingham Shum, S (2012) Workshop 'Collective Intelligence as
Community Discourse and Action, CSCW conference in Seattle, February 11.
2012. http://events.kmi.open.ac.uk/cscw-ci2012/ http://events.kmi.open.ac.uk/cscwci2012/wp-content/uploads/2012/02/CSCW2012-CI-Workshop-final.pdf
De Liddo, A. and Buckingham Shum, S. (2010). Cohere: A prototype for contested
collective intelligence. In: ACM Computer Supported Cooperative Work (CSCW 2010) Workshop: Collective Intelligence In Organizations - Toward a Research Agenda,
February 6-10, 2010, Savannah, Georgia, USA. http://oro.open.ac.uk/19554.
McAndrew, P., Farrow, R., Law, P., & Elliot-Cirigottis, G. (2012a) “Learning the Lessons of
Openness” in Comas-Quinn, A., Fitzgerald, A., & Fairweather, I. (eds.) Journal of
Interactive Media in Education. Special Issue Cambridge OER 2012. Available
fromhttp://jime.open.ac.uk/article/2012-10/html.
McAndrew, P., Farrow, R., Law, P., & Elliot-Cirigottis, G. (2012b) “Learning the Lessons of
Openness”. Proceedings of Cambridge 2012: Innovation and Impact – Openly
Collaborating to Enhance Education, OCW Consortium and SCORE, Cambridge, UK,
April 16–18 2012, Milton Keynes, The Open University, pp. 193-202,
http://oro.open.ac.uk/33640/.
OER Research Hub (2013) OERRH Ethics Manual. Available from
http://oerresearchhub.org/project-documents-archive/reports/oerrh-ethics-manual/.
OER Research Hub (2013) Project Website at http://oerresearchhub.org/.
OGC (2013) Open Geospatial Consortium: KML. Available from
http://www.opengeospatial.org/standards/kml/.
35
Open Learning Network (2013). Evidence Hub for Open Education. Available from
http://ci.olnet.org/.
36