The Avalon Media System
Yale University
November 24, 2014
Jon Dunn
Indiana University
Mark Notess
Indiana University
Julie Rudder
Northwestern University
Agenda
Introduction & Overview – Jon Dunn, Indiana
Demo – Julie Rudder, Northwestern
Technical architecture, development process, release
plans – Mark Notess, Indiana
NU implementation and services - Julie
IU implementations and services – Mark
Future directions – Jon
Q & A - All
Avalon Project Objectives
Goal: Create an open source system to enable
libraries and archives to provide online access to
video and audio collections
• Digital audio/video management and delivery system, focused
on needs of libraries and archives
• Follow an agile, open source development model
• Leverage existing technologies, where feasible
• Communicate and market the project broadly to increase
awareness and grow the community of users and developers
Project Overview
• Funded in part by a National Leadership Grant from the Institute of
Museum and Library Services
• Original name: Variations on Video
• Planning grant:
– August 2010 – July 2011
• Implementation grant:
– October 2011 – September 2014 September 2015
• Partnership between libraries at Indiana University and
Northwestern University
Project Partners and Advisors
Project Partners and Advisors
Avalon is a Hydra Solution Bundle
Why Avalon?
• Existing repository systems don’t work well for timebased media
– DSpace, Fedora, Digital Commons, ContentDM, …
– Issues: integration with streaming servers; hard to support
time-based navigation; difficulties in integrating access
control
• Existing media systems don’t work well with
repositories
– Kaltura, Brightcove, Ensemble, ShareStream, …
– Issues: focus on short-term access for teaching and
learning; limited metadata and access control capabilities;
integration with preservation workflows
• Strong desire for community-based solution
Demonstration
Avalon Architecture
Users
Integrations
ILS
LMS,
websites
Authentication
Authorization
Archival
Storage
Avalon Media System
Hydrant Rails App
OmniAuth
CanCan
Fedora
Media Player
(mediaelement.js)
Hydra
Head
Solr
Blacklight
Matterhorn
Rubyhorn
Media Server
 Avalon code – consists of Hydrant Rails App and the Rubyhorn
gem. Other pieces are leveraged.
Search
Browse
View
Ingest
Describe
Manage
Desktop,
Mobile
All Users
Browser,
Drop box
Collection
Managers
Release History
•
•
•
•
•
•
•
0.1 – July 2012
0.5 – October 2012
1.0 – May 2013
2.0 – October 2013
3.0 – May 2014
3.1 – July 2014
3.2 – December 2014 (expected)
Julie Hardesty
Jon Dunn
metadata specialist
project director
Chris Colvard
Mark Notess
lead developer
product owner
Cynthia Ramlo
Leah Lee
lead user experience designer
developer
Brian Keese
Ariadne Rehbein
developer
project assistant
Development Team
Michael Klein
Karen Miller
lead developer
metadata specialist
Stefan Elnabli
Dan Zellner
audiovisual specialist
testing and production support
Claire Stewart
co-project director
Julie Rudder
product owner
Current development process
•
•
•
•
A single, blended team
Agile Scrum process
Code in public GitHub
Face-to-face meetings
at least 2x/year
• Daily standups, IRC
0 – 3.1 Release functionality
End-user functionality
• Faceted discovery for search and browse
• Video and audio playback in browsers or on mobile devices
• Stream-level security
• Ability to embed media player in other websites
• Persistent URLs
• RTMP and HTTP streaming, Flash and HTML5-based player
Collection management
• Interactive or batch uploading of media files and MODS metadata
• Private collection dropboxes
• A hierarchical model for permissions that supports a flexible approach to batch
collections-based content management
• Customized thumbnails by taking a “snapshot” or specifying a timepoint
• Direct import of previously transcoded derivatives
• Avalon transcoding for multiple quality derivatives
0 – 3.1 Release functionality
Integration with other enterprise systems
• Red5 and Adobe media servers
• Local authentication services (CAS, LDAP and others, using OmniAuth)
• Learning management systems, via the Learning Tools Interoperability (LTI)
standard
• Support for LDAP group access control
• Support for persistent URL systems (PURL, Handle)
• Master files: leave, delete, or rename/move after transcoding
Installation
• Easy installation and configuration via a virtual machine image and other
methods
• Fully-transparent integration testing using travis-ci.org - every commit and pull
request triggers a build
• Configuration approach for easier customization where possible
Upcoming releases (Fall/Winter 2014)
Release 3.2 (December)
• Upgrade to latest versions of Hydra, Blacklight, and Bootstrap
• Metadata: adding language, original physical description,
related item links, terms of use
• Let staff select multiple items and publish, unpublish, delete,
set access permissions, assign to collection
Release 3.2.1 (January/February 2015)
• Import descriptive metadata from an Integrated Library System
• Import multiple quality levels of pre-transcoded derivatives
Release 4 (Spring 2015)
• Structural metadata
– Import and store structural
metadata
– Allow users to navigate using
media sections
• Prioritize media in the
transcoding queue
• Bulk update of metadata
• Fedora 4 (testing?)
Beyond Release 4
• Reporting and metrics
• Time limits for access
• IP restricted access
• APIs for ingest, analytics, metadata, etc.
• Fedora 4 migration
• Accessibility (navigability, closed captioning)
• Transcripts/Auxiliary items
• Improved metadata
• Playlists & clip making
• Integration with Wowza and Amazon Cloud Front
• Publish to YouTube or other systems
• Integration with Spotlight exhibits tool
• Internationalization
Follow our progress
• For sprint summaries, read our blog
• For more details and recordings, visit our wiki:
wiki.dlib.indiana.edu/display/VarVideo/Avalon+Media+System
• Avalon Current Sprint page
wiki.dlib.indiana.edu/display/VarVideo/Avalon+Current+Sprint
• Previous Sprints page
wiki.dlib.indiana.edu/display/VarVideo/Previous+Sprints
• Sprint Demos
wiki.dlib.indiana.edu/display/VarVideo/Sprint+Demos
Community and future directions
• Multiple implementations currently underway
– Stanford, UVa, UToronto, UT Austin, …
• Working on new features, including:
– Structural metadata
– Transcripts and captioning
• Starting to deal with code contributions
• Working with Hydra community
– Leverage common features across “heads”
– HydraDAM and Avalon, other preservation-specific
integrations
• Sustainability/governance/business model
• Exploring hosted options
Sustainability and Governance
•
•
•
•
Focus in 2015 with potential funder support
Want to grow the number of implementers
Ongoing costs of software support and development
Potential revenue streams
– Membership/sponsorship
– Income from hosting
– In-kind developer contributions
• Governance and management structure
– Potential alignment with existing 501(c)(3) organization in
higher ed open source community
Julie Rudder j-rudder@northwestern.edu
Digital Initiatives Project Manager
DLF Forum 2014
Use Cases
Course Reserves – restricted access
Archival Collections
MMLC – video management by units
outside the library
Future
Additional units outside the library?
Integration with other campus media solutions?
Better integration with Ares Course Reserves?
Usage so far....
Existing Production Systems
Avalon Web App
Load Balancer
Streaming FMS 2
Streaming FMS 1
VM
Transcoding
Shared SOLR
Dedicated SOLR Core
4 TB dedicated
Hardware
Shared Production Fedora
Fedora
Production Environment
Background
Integrations on Streaming at NUL
• Canvas and Blackboard
Streaming Media since 2001
• Existing handle server
Total Audio-Video Assets Served: 34,146
• LDAP
• Adobe Media Server (secure streaming)
• Related -- Ares Course Reserves (request
mechanism)
Implementation
Pilot – Summer 2013
•
•
•
A few classes support
Two public collections
Local VM infrastructure
Full production – Fall 2014
•
•
•
All new requests via Avalon (no migration)
MMLC launches Unit
NUIT (central data center) infrastructure
AVALON
Retire current infrastructure
by summer 2015
All new digitization
Summer
2014
Summer
2015
2004 – 30,000+ assets served
Retire legacy systems
Staffing
1 DevOps (install, configure, support)
1 Developer (app customization, support)
1 Project Manager (app oversight, integrations oversight)
1 Service Owner (liaise with units & users, workflow, support)
Additional support –
st
1
tier
Business hours
Library Digital
Collection
Staff
24 hour -Canvas helpdesk support
24 HOUR
Central Campus
IT helpdesk
24 HOUR
Canvas
helpdesk
• Avalon at Indiana University
IU Context
•
•
•
•
•
History of audio/video digitization
Legacy systems for delivery
Eight campuses
New demands
Media Digitization and
Preservation Initiative
Legacy Systems
IU Deployment Environment
University IT Services
Library servers
Avalon Web App
Solr, Fedora
Redundant Archive Storage
(HPSS)
Master Files
Shared SAN
Derivatives
4 core, 1GB RAM VM
1GB Disk (VMware)
Transcoding
12 core, 120GB
RAM,8TB dedicated
hardware
IU Enterprise Streaming
Adobe
Media
Server
Integrations
•
•
•
•
•
CAS authentication
Microsoft Active Directory (via LDAP)
PURL resolver
Canvas and Sakai (via LTI)
Scholarly Data Archive (HPSS master file
storage)
• Adobe Media Server (secure streaming)
IU Production Instance
Pilots began Fall
2013
Moved to
production,
October 2014
Archival Film
Omeka Online Exhibit:
World War II Propaganda Frilms
Media Within Archival Collections
Online Publishing:
Ethnomusicology Multimedia
Specialized Avalon Instance: IJCCR
Specialized Avalon Instance:
Kelley School of Business
Future Migrations
• Video e-reserves (in pilot)
• LTI integration with Sakai, Canvas
• Licensed educational video
• LDAP for campus restriction
• Variations: music audio
• Pedagogical, annotation tools
• Media Digitization and Preservation Initiative
Production Staffing
• 10% of a systems administrator (server
management; higher initially)
• 20% of a developer (install, configure,
support; higher initially)
• 20% of project/service manager (app
oversight, workflow planning, work with units
and users, support)
• Does not include collection management or
central IT staffing of shared services
IU Media Digitization and
Preservation Initiative
• mdpi.iu.edu
Avalon in the IU Media Digitization
and Preservation Initiative (MDPI)
• 8.4 petabytes of audio and video data over 5 years
• Avalon serving as key access component
– Integrations:
•
•
•
•
•
Metadata: MARC/ILS, EAD/XTF
Authentication: CAS
Authorization: Active Directory
Learning Management: Canvas, Sakai
Media streaming: Adobe Media Server
• Side-by-side with trusted repository for preservation
– Bit storage in existing hierarchical storage management
system
– Hope to leverage HydraDAM work
– Exploring out-of-region options
Questions?
For more about Avalon
• Website: http://avalonmediasystem.org/
• Avalon e-mail list:
E-mail avalon-l-subscribe@indiana.edu to subscribe
@avalonmediasys
facebook.com/avalonmediasys
Backup Screenshots
Faceted browse via Blacklight
Desktop/Mobile Player
Embedding
LTI (in Canvas)
LTI (in Canvas)
Manage Content
Add item: Upload, Dropbox, or Batch
Dropbox view
Batch Folder
Descriptive Metadata: MODS
Access Control
Permanent URL minted on Publish
Bulk actions