Your Results From Disk to Device
(Preview)
Amit Chourasia
Mona Wong-Barnum
Michael L. Norman
San Diego Supercomputer Center, UCSD
Presented at: XSEDE 13, San Diego, CA, Jul 23, 2013
Stream Encode Explore Disseminate My Experiments
Computation Cycle
Abstract
?
Monitor
?
Analyze & Save
Compute/Ingest
Post Process
Validate
?
Monitoring
Derived
Products
Abstraction
Validation
Direct Access
Direct Sharing
Derived
Products
Not All
Members
Have HPC
Access
Security
Implications
On HPC
Concise derived products
• Plots & Text Documents
• Image Sequences & Videos
Share with the research group
1. Download derived products
2. Share with collaborators
a.
b.
Email snippets
Set up a webpage
– Add content
– Email
Derived Visual Content Survey
Domain/Content
Simulations
Per Year
Time Steps
Per
Simulation
Variables
Images Per
Per
Simulation
Simulation
Astrophysics
5 - 10
10,000
25
Climate
3-5
50,000
Molecular Dynamics
12+
Seismology
Space Weather
Image Size
(pixels)
Movies Per
Simulation
Group Community
Size
Size
10,000
4096x4096
10
10
50
100
10,000
2048x2048+
50
100
1,000
500,000
3
6,000
1600x1200+
2
5
10
3-6
160,000
3
20,000
6000x3000
6
15
30
10+
400
20
10,000
2048x1024+
10
30
2,000
Derived content for a single research group in respective domain
Pitfalls in sharing derived content
Download
Upload
Round Trip
Video
Encoding
Complexity
Web Setup
Process
Duplication
Scalability
Gaps
SeedMe.org
Download
Upload
Direct
Transfer
Video
Encoding
Web Setup
Easy Sharing
Direct Access
Not All Members
Have HPC Access
Process
Duplication
Reusable
Direct Sharing
Security
Implications On
HPC
Scalability
Ubiquitous
Why not use existing tools?
Features
SeedMe
YouTube & Vimeo Flickr
Sharing
Public, Group, Private
Public, Private
Public
Upload Method
Command Line, API, Web
API, Web
API, Web
Meta Data
Rich
Limited
Limited
Download Support
Yes
Vimeo only
No
Copyright
Unencumbered
Encumbered
Encumbered
Still Images
Yes
No
Yes
Periodic Image Update
Yes
No
No
Image Sequence to Video
Yes
No
No
Frame Rate
Arbitrary
29.97 fps
Unknown
Video Resolution
Arbitrary
Upto 4K
Upto HD
Desirable
Limiting
Crippling
SeedMe Architecture
Instant
Processing
Ingestion
(Atomic or
Recurring)
(Text, Images)
Notification
Batch
Processing
(Videos)
Responsive
Content
Access
SeedMe Hardware
Instant
Processing
(Text, Images)
Ingestion
(Atomic or
Recurring)
Responsive
Content
Access
Notification
Batch
Processing
(Videos)
20
gbe
Ingest
Temp
Storage
(10TB)
20
gbe
Web Cluster (2-8 nodes) : Drupal
Mirrored Storage (50 TB) : ZFS
Encode Cluster (16 nodes) :
Apple Compressor + ffmpeg
20
gbe
Phones
Tablets
Desktops
Compressor Encoding Benchmark
12000
10000
2 Cores (Compressor)
6000
1 Cores (fmpeg)
2 Cores (fmpeg)
4000
1 Core (Compressor)
2 Cores (Compressor)
4000
Title (s)
4 Cores (Compressor)
Encoding Magneto
20,000 images @ 838x510 px (2.4 GB)
5000
1 Core (Compressor)
8000
Time (s)
6000
Encoding M8
1800 images @ 4096x2048 px (14 GB)
4 Cores (Compressor)
3000
1 Cores (fmpeg)
2 Cores (fmpeg)
2000
4 Cores (fmpeg)
4 Cores (fmpeg)
2000
1000
0
0
1
2
4
8
1
2
No of Nodes
25000
4
No. of Nodes
4000
Encoding ShakeOut
3800 images @ 3840x2160 px (34 GB)
20000
Encoding Volcano
9300 images @ 310x1450 px (688 MB)
3500
1 Core (Compressor)
1 Cores (Compressor)
3000
2 Cores (Compressor)
2 Cores (Compressor)
2500
15000
4 Cores (Compressor)
1 Cores (fmpeg)
10000
Time (s)
Time (s)
8
4 Cores (Compressor)
2000
1 Cores (fmpeg)
2 Cores (fmpeg)
1500
4 Cores (fmpeg)
1000
2 Cores (fmpeg)
4 Cores (fmpeg)
5000
500
0
0
1
2
4
No. of Nodes
8
1
2
4
No. of Nodes
8
SeedMe Collection
Ticker*
Plots
Image Sequences*
Access: Public, Group, Private
Collaborators
Meta Data
* Recurring: periodic update
Videos
SeedMe: Interaction Process
Create
Collection
Set
Privacy
Set
Metadata
Upload
Content
View
on Web
Update
Create New Collection
Set Access Control (Public, Group, Private)
Specify collaborators* and meta data
Upload/Update content
http/https, ftp, Globus Online
Interactively on Web, Command Line or REST Services
View Collection on Web
SeedMe: Interaction Modality
• Command Line Interface (using curl)
• Web Interface (WYSIWYG)
• RESTful API (integrate your tool chain)
– JSON
{
"email": ”jdoe@sdsc.edu",
"api_key": "XXXXXXXXXXX",
”view_permission": "public",
”viewer_emails": ["a1@a.com", "b1@b.com"],
"title": "simulation XYZ",
"description": "input conditions",
"credits": "Visservices group, SDSC",
"key_values": { “key1”: "v1”, "key2": "v2”},
"expire_date": "2013-05-01",
"tags": [ "x", "y", "z"],
"ticker": ["time step 100", "time step 200"],
Authentication
Permissions
Meta Data
Text Ticker
"files":{
”docs" : ["a.pdf", ”b.txt"],
Doc Files
"plots" : ["a.png", "b.jpg", "c.tiff"],
Plot Files
"image_sequences" : {
"ash": ["ash1.jpg", "ash2.jpg", "ash3.jpg"],
"air": ["air1.jpg", "air2.jpg", "air3.jpg"]
},
"movies": ["m1.mov", "m2.mov »]
}
}
Image Sequences
Videos
Content
Application Scenarios
• Monitor progress with text tickers and images
• Encode and stream videos from image
sequences
• Share & collect visualization results
• Use the collections for education and
outreach
Content accessible on most devices
Work in progress
How it Works
Examples #1 & #2
Periodic update at simulation run time from a Magneto Hydrodynamics Simulation
Courtesy of Mahidhar Tatineni and Homa Karimabadi, UCSD
Future Work
• Production SeedMe Service
• Example Scripts & Documentation
• Globus integration with SeedMe
Acknowledgements
Andrew Ferbert, Michael Dwyer, Kristen Levy & Doug Weimer, SDSC :
Hardware/software setup
Mahidhar Tatineni, SDSC & Homa Karimabadi, UCSD: in-situ tests
Apple Inc. : Provided test hardware/software on loan during project
conceptualization phase
National Science Foundation
This material is based upon work supported by the National Science
Foundation under Grant No. OCI-1235505
Sign up Invitation
For notification and early access
(we won’t spam)
Contact: amit @ sdsc.edu
Persistence?
Move to own
resource
Move to XSEDE
resource
PURGE
Results
Move to
YouTube/Flickr,
Vimeo
Freemium
Move to
Dropbox/Cloud