CWSpace
Archiving MIT OpenCourseWare in DSpace
DSpace Federation
2nd User Group Meeting
University of Cambridge
July 7 - 8, 2005
v. 20050704_2215
DSpace: wide adoption as
Institutional Repository
• CONTENT
– Scholarly
materials
– Research or
Education
oriented
AUDIENCE
Humans;
SysAdmins;
Spiders;
Harvesters
July 8, 2005
• FEATURES
– Safe
•
•
•
•
professionally archived
persistent, citable URL
preserved over archival timeframes
policies re: removal, etc.
– Findable
•
•
•
•
•
search (metadata; full-text)
browse
notification e-mails
disseminated metadata (OAI-PMH)
Google and DSpace ("Scholar")
CWSpace, an MIT iCampus project
2
DSpace: new territory as LOR?
Learning Objects Repository
• CONTENT
• FEATURES
– Educational content:
• Teaching, learning,
instruction, assessment
– Courseware materials
– Safe
– Findable
– Emphasis on:
• Learning Objects
– Compound Digital
Objects, Websites…
• Sharing, Re-Use
• Aggregation
• Evaluation
AUDIENCE
Humans; SysAdmins; Spiders ; Harvesters
System-to-System: CLEs; Image gallery tools
July 8, 2005
CWSpace, an MIT iCampus project
3
Project Goal: InterOperability
"To harvest and
digitally archive
OCW learning
objects, and make
them available to
learning management
systems (LMSs) by
using Web Services
interfaces on top of
DSpace."
http://icampus.mit.edu/projects/DSpace.shtml
July 8, 2005
• re: "harvest" - Yes. OCW has
built "Content Exporter”
• re: "archive" - Yes. DSpace
has new IMS-CP ingest
module; course rendering
• re: "learning objects" - No,
but courseware, Yes.
Discovered that from OCW
will be Courses not LOs
• re: "Web Services"...to
"LMSs" - Yes. Initial
version WS simple clients;
also to LMS (SloanSpace)
prototype
CWSpace, an MIT iCampus project
4
InterOperability: the main goal
July 8, 2005
CWSpace, an MIT iCampus project
5
CWSpace: Standards for
Packages & Protocols
• Protocols, APIs
• Packages
– IMS Content
Package
– METS
– MPEG21-DIDL
– XFDU
July 8, 2005
– SOAP & WSDL
– WebDAV
• JSR 170 JCR
– RESTful (XML,
HTTP)
– XML-RPC
CWSpace, an MIT iCampus project
6
Work Activity To Date
Schematic showing various areas of development:
metadata specifications, Web Services, export and
import programming.
July 8, 2005
CWSpace, an MIT iCampus project
7
Metadata for Content Packaging
• METS
– Libraries…
• IMS-CP
– Education…
• MPEG-21 DIDL
– Commercial…
• XFDU
– Aerospace…
July 8, 2005
CWSpace, an MIT iCampus project
8
Package Interchange File (PIF)
• IMS-CP uses a .ZIP file with a
Manifest XML file and all content
files:
http://www.imsglobal.org/content/packaging/cpv1p1p4/imscp_bestv1p1p4.html
July 8, 2005
CWSpace, an MIT iCampus project
9
IMS-CP and OCW
Object Model
July 8, 2005
CWSpace, an MIT iCampus project
IMS-CP
imsmanifest.xml
10
METS
mets.xml
Red arrow:
Logical
organization to
Physical (href)
METS &
IMS-CP
Manifests
Diagrammed
IMS-CP
imsmanifest.xml
Red arrow:
Logical
organization to
Physical (href)
Blue arrow
(METS only):
Descriptive
metadata is
separated from
Logical or
Physical
July 8, 2005
CWSpace, an MIT iCampus project
http://cwspace.mit.edu/docs/ProjectMgt/Reports/SPARC-IR-Workshop/sparc-poster.html
11
XML Elements:
METS & IMS-CP
For those who
like the pointy
angle brackets…
July 8, 2005
CWSpace, an MIT iCampus project
12
METS SIP Profile for DSpace
• Work-in-progress
– MIT CSAIL Publications Archive
• Anticipates DSpace 2.0 possible METS as AIP
• Preservation and Technical Metadata
• Licenses as METS metadata (Deposit; Creative
Commons)
• StructMap raises questions for flat DSpace file
storage (Bundles & fileGrps) (Export re-creation)
• Option for new PluginManager to manage
• DIP work ahead as well; may differ
July 8, 2005
CWSpace, an MIT iCampus project
13
CWSpace
Archiving MIT OpenCourseWare in DSpace
DEMO
(SCREENSHOTS)
OCW Course Rendered
in Three Systems
OCW………….. DSpace ………….. SloanSpace …………………………….
July 8, 2005
CWSpace, an MIT iCampus project
15
Year One Demo: OCW to DSpace
July 8, 2005
CWSpace, an MIT iCampus project
16
OCW Course Website
• Static website
• HTML, PDF,
JPG, XLS
• Akamai:
Multimedia
• LOM XML
• Search;
Feedback;
Tracking
• Copyright
cleared
http://ocw.mit.edu/OcwWeb/Sloan-School-of-Management/15-040Spring2004/CourseHome/index.htm
July 8, 2005
CWSpace, an MIT iCampus project
17
OCW Content Exporter (CE)
• CE generates entire
course website
(rewriting links)
• CE writes
imsmanifest.xml
• CE publishes .ZIP
to web page
• CE can publish
whole dept.
• CE also used:
translation;
professors; etc.
July 8, 2005
CWSpace, an MIT iCampus project
18
Client: IMS-CP DSpace Import Driver
• OCW CE not yet
WS client
• Simple CGI
driver to DSpace
session(),
auth/auth(),
upload(), and
ingest() WS
• 4 params (w-i-p)
RESULTS: Success - Created DSpace item 123456789/75
Using collections='123456789/2’
Started session with token='2-1039ad464ee-7665e5a950ff7ff2'
upload response :: <?xml version="1.0" encoding="UTF-8"?>
<uploadservice><URI>http://rotarran.mit.edu:8080/dspacews/upload/package-21802.zip</URI><size>8309770</size>
</uploadservice>
Ingesting IMSCP package in package-21802.zip
ingest response :: 123456789/75
Session ended"
July 8, 2005
CWSpace, an MIT iCampus project
19
DSpace Item Record for OCW Course
• DSpace info
model maps
“Item” to OCW
“Course”
• Files (all types)
are “Bitstreams”
• Metadata: basic
LOM-2-DC
• Year 2: Further
DSpace dev re:
websites, LOs
July 8, 2005
CWSpace, an MIT iCampus project
20
DSpace Serving Full OCW Course
• Static
website
• HTML, PDF,
JPG, XLS
• Akamai:
Multimedia
• DSpace
search
(Lucene)
• LOM XML
to Dublin
Core (w-i-p)
• OAI
metadata
distribution
• Copyright
cleared,
DSpace, CC
licenses
July 8, 2005
CWSpace, an MIT iCampus project
21
Year Two Demo: Other CLE
(early preview!)
July 8, 2005
CWSpace, an MIT iCampus project
22
SloanSpace Searching, Uploading
OCW Course
PROTOTYPE
• SloanSpace
(dotLRN,
OpenACS) portal
functionality
• Search DSpace
(SRW)
• Retrieve via
(provisional)
DSpace WS
• Directly “Add
Course” to
dotLRN module
for “Learning
Object Repository
System” (LORS)
July 8, 2005
CWSpace, an MIT iCampus project
23
SloanSpace LORS Navigation
for OCW Course
PROTOTYPE
• dotLRN
module, for
IMS-CP
standard
• LOM XML
• (Use of
SCORM
permits
tracking, etc.)
July 8, 2005
CWSpace, an MIT iCampus project
24
SloanSpace LORS Serving Full
OCW Course
PROTOTYPE
• HTML,
PDF, JPG,
XLS
• Akamai:
Multimedia
• Lifecycle
issues in
Year Two
work
July 8, 2005
CWSpace, an MIT iCampus project
25
Year Two Deliverables
• OCW Production
– Operationalize to scale for OCW archiving (100
courses Fall 2005…)
• InterOperability with other CLE/LMSs
– Package: “CWSpace IMS-CP Profile” for SloanSpace,
for Stellar, for Sakai
– Protocol: Web Services use cases, clients for same
• Archived websites
– Improved contextual presentation
• Terminology Extraction Tool (CSAIL)
– Explore integration into DSpace, OCW
July 8, 2005
CWSpace, an MIT iCampus project
26
CWSpace: Planned Work
• DSpace Platform
– Plugin Manager
• Packagers (I/O)
• Crosswalks
– Structured Metadata
– Stackable
Authentication
July 8, 2005
• Protocols
– Lightweight
Network Interface
(LNI)
• SOAP & WSDL
• WebDAV
CWSpace, an MIT iCampus project
27
Planned Work on CWSpace
July 8, 2005
CWSpace, an MIT iCampus project
28
Existing Interfaces to DSpace
•
•
•
•
DSpace Web U/I
SRU/SRW Search
OAI-PMH
Command-line
– DSpace Batch Importer
– Media Filters…
http://wiki.dspace.org/NetworkInterfaces
July 8, 2005
CWSpace, an MIT iCampus project
29
Lightweight Network Interface
• New Proposed Interface(s)
– DSpace “Web Services”, for CWSpace…
• SOAP & WSDL
• WebDAV (Extension to HTTP protocol)
http://wiki.dspace.org/LightweightNetworkInterface
July 8, 2005
CWSpace, an MIT iCampus project
30
A Few (Quick) Thoughts on
Web Services Design
The next few slides provide a (very) brief tour of the topics we are
investigating with our Lightweight Network Interface (LNI)
• Standards-based vs. Custom model
• Abstract vs. Strongly modeled specification
• Technology Approaches (SOAP; REST…)
July 8, 2005
CWSpace, an MIT iCampus project
31
Approaches To Expose Your
Object Model, 1
In selecting an API or protocol or approach to exposing your
object model to a network interface there are two axes to consider.
• ‘X’ axis: Degree of match to Standards
– Custom
• PRO
• CON
More control for service
Harder to interoperate for consumer
– Standards-based
• CON
• PRO
July 8, 2005
Compromises for service
Easier interoperation for consumer
CWSpace, an MIT iCampus project
32
Approaches To Expose Your
Object Model, 2
• ‘Y’ axis: Degree of precision in model
abstraction
– Abstractly Modeled
Open to extension, interpretation, wide application
CON You have a specification, but not implementation
guide
• PRO
•
– Strongly Modeled
• CON
• PRO
July 8, 2005
Compromises for service; line is drawn in the sand (!)
Implementable directions; line is drawn in the sand (!)
CWSpace, an MIT iCampus project
33
Matrix: Expose Object Model
July 8, 2005
CWSpace, an MIT iCampus project
34
Technology Approaches
In then opting for a style of developing the services your Object
Model will provide, there are some technology choices to consider.
• SOAP, WSDL
– Enterprise developers
– Contains hints re: objects, methods
• RESTful (“Representational State Transfer”)
– Developers comfortable with XML markup, HTTP
– Straightforward XML messages over HTTP
• XML-RPC
– Early, simplified spin-off from SOAP (ca. 1999)
• WebDAV (Protocol: extension to HTTP)
– “Resources” and “Properties” works well with Repository
July 8, 2005
CWSpace, an MIT iCampus project
35
Resources, Properties:
WebDAV
DSpace (root)
Community
Collection
Item
Bitstream
Workflow
/mylib/dspace-ni/
/mylib/dspace-ni/dso_1721.1%2F46
/mylib/dspace-ni/dso_1721.1%2F3549
/mylib/dspace-ni/dso_1721.1%2F5543
/mylib/dspace-ni/dso_1721.1%2F5543/bitstream_13
/mylib/dspace-ni/workflow/wf_23
The server only pays attention to the ''last'' element in a path of DSO's, e.g.
http://myserver/DAV/dso_123456789%2F1/dso_123456789%2F4/dso_123456789%2F13
is the same as
http://myserver/DAV/dso_123456789%2F13
http://wiki.dspace.org/LightweightNetworkInterface
July 8, 2005
CWSpace, an MIT iCampus project
36
Resources, Properties:
WebDAV
Collection objects
logo; short_description ; introductory_text ;
sidebar; copyright; default_license; provenance
Item objects
submitter; owning_collection; license;
cc_license; cc_license_rdf;
DAV:getlastmodified
Bitstream objects
DAV:getcontentlength; DAV:getcontenttype;
source; description; format-id; formatdescription; checksum; checksum-algorithm;
sequence-id
July 8, 2005
CWSpace, an MIT iCampus project
37
Example GET URIs
/mylib/dspaceni/dso_1721.1%2F5543?session=1
23xyzzy456&package=org.dspace.
METS
/mylib/dspaceni/dso_1721.1%2F5543/bitstream
/13?session=123xyzzy456
July 8, 2005
CWSpace, an MIT iCampus project
38
Example PUT URI
To add a new item to the collection at handle
1721.1/3549:
PUT /mylib/dspaceni/dso_1721.1%2F3549?session=123xyzz
y456&package=OCW-IMSCP
....package contents...
HTTP/1.1 201 OK
Location:
/mylib/dspace-ni/dso_1721.1%2F5549
....other headers....
July 8, 2005
CWSpace, an MIT iCampus project
39
Features Matrix
• Comparing the Interfaces to DSpace with
the Features they provide
July 8, 2005
CWSpace, an MIT iCampus project
40
Three Things to Take Away
from this talk on the iCampus Project: CWSpace
• We’re working on Packaging Metadata
• We’re working on Web Services
• We’re also working on Archiving Websites
... help us refine the thinking …
July 8, 2005
CWSpace, an MIT iCampus project
41
CWSpace
Archiving MIT OpenCourseWare in DSpace
Thank You. Questions, Comments…
http://cwspace.mit.edu
William Reilly, Larry Stone, MacKenzie Smith—MIT Libraries’ Digital Library
Research Group (DLRG)
Rob Wolfe—MIT Libraries’ Metadata Services Unit
Cec d’Oliveira—MIT OpenCourseWare, Technology
July 8, 2005
CWSpace, an MIT iCampus project
42