Big Visualization at Los Alamos
Workshop on Parallel Visualization and Graphics
LAUR-03-7666
Allen McPherson
Los Alamos National Laboratory
October 19, 2003
Outline
Overview of our mission and computing environment
Compute and visualization facilities
Software infrastructure
Production visualization process
Future direction and challenges
LAUR-03-7666
•
•
•
•
•
What we do…
• Mission
– We develop and apply science and technology to…
• Ensure the safety and reliability of the US nuclear stockpile
• Reduce the threat of weapons of mass destruction,
proliferation, and terrorism
• Solve national problems in defense, energy, environment, and
infrastructure
• Goal
LAUR-03-7666
– Creating and integrating core competency for Science Based
Prediction of complex systems linking experiment,
simulation, and theory
Computational Environment
• Current machines
– Q (LANL) 20TF
– White (LLNL) 12TF
– Red (SNL) 3TF
• Next generation
LAUR-03-7666
– Purple (LLNL) 100TF
– Red Storm (SNL) 40TF
– Large computational
clusters for “capacity”
computing
– 1PF (LANL)
Functionality Requirements
• Interactive visualization and analysis of extremely large, time
varying, datasets
– Also, canned multi-display movies
• Visualization of such datasets over large geographic
distance, yet still enabling interactivity
• Support multiple display modalities
LAUR-03-7666
• Same software on all modalities
Generic Physical Simulations
• Simulation of a shallow
water asteroid impact
LAUR-03-7666
• Simulation of a RichtmyerMeshkov instability
experiemnt
Visualization Facilities
Switched analog distribution
Office displays
Collaboratories
Theater
Immersive facilities
LAUR-03-7666
•
•
•
•
•
Switched Analog Distribution
LAUR-03-7666
• Supported over campus-like environment
• Multiple switches: 128x128 & 30x30
Office Visualization
• Stereo CRT through analog switch
– 60 offices
• High-end PC with IBM, et al. highres LCD display
GIG-E connected
25 offices
EnSight client/server model
Users would like stereo here too
LAUR-03-7666
–
–
–
–
Collaboratories
• Delivering visualization facilities
directly into user office areas
• No visualization expert required
• Bright, stereo walls allow normal
lighting
– Two built in SCC
LAUR-03-7666
• Two more non-stereo walls
• Additional office-type equipment
installed for drop-in use
• All driven through switched analog
with our rendering infrastructure
Visualization Theater
•
•
•
•
•
•
Completed April 2003
22’ x 12’ rear-projected power wall
24 bright, stereo projectors
31M pixels
85 seats
Driven through switched analog with SGI
– Will drive with cluster soon
LAUR-03-7666
• Presentation, reviews, training
• Brings visualization performance to major
decision making meetings of the Laboratory
• Used on daily basis for large and small groups
Immersive Environments
• RAVE
– Reconfigurable CAVE
– Full immersion (VR) environment
– Head tracking
LAUR-03-7666
• An experiment in methods for
understanding data
Next-Gen Immersive Environment
Spring/Summer 2004
5 rear-projected surfaces
–
–
–
•
•
•
33 projectors, 43+ M pixels
Bright, digital projectors
This builds on the positive user experiences in the
RAVE:
–
–
•
Front wall: 15’ wide x 12’ high
Floor/Ceiling: 15’ wide x 8’ deep
Side walls: 10’ wide x 12’ high
Immersion helps people understand simulations more
quickly.
They help bridge the comprehension gap between
simulation scientists and experimental scientists and
engineers.
This will be driven with a new SGI 3900 w/ Voyager
graphics.
–
34 commodity graphics pipes
LAUR-03-7666
•
•
Rendering Infrastructure
•
3 SGI Onyx 2000, each with:
– 16 IR pipes (48 pipes total)
– 128 PEs, 128 GB memory, 5+ TB disk
– All available through analog switch
•
High-performance desktop systems are used as a capable rendering
system for moderate-sized problems.
•
A new SGI Onyx 3000 w/ Voyager graphics is being deployed to
support
– additional offices
– the next-generation CAVE system
LANL’s first production, distributed-memory, cluster-rendering system is
currently being deployed for the SCC Theater.
– Next step is to provide distributed-memory cluster-rendering support for
offices & PowerWalls.
LAUR-03-7666
•
Software Environment
• CEI Ensight
– Mainline production visualization tool
• Kitware ParaView and VTK
– Open source, research tool
– Lets us use other lab’s work (e.g. Sandia Ice-T)
– Being deployed for some user functions
• Comparative visualization
• Kong
LAUR-03-7666
– Parallel distributed version of Simian volume renderer
– Collaboration with University of Utah (Joe Kniss)
Distributed EnSight Architecture
GigE
network
ASCI Q Viz Nodes
Switched
Analog
Fiber
Visualization
Rendering Server
EnSight
Server-ofServers
EnSight
Client
EnSight
Server
•
Leave data in place where computed
–
–
•
•
•
Don’t store it twice
Don’t size end-node systems to platform sizes
Uses modest amount of bandwidth
Supports interactive analysis
Supports various display modalities
LAUR-03-7666
PFS
SGI Onyx,
PC Desktops,
or soon: clusters
How is Visualization Used?
• One big example: milepost run on White
–
–
–
–
–
–
Ran for couple of months
650TB of data written (restart files)
364 visualization time steps
85M to 468M unstructured cells per time step
Multiple variables per cell
21TB total visualization data
LAUR-03-7666
• Run at Livermore, visualized at Los Alamos
Big Viz Example
• Visualization is used to…
–
–
–
–
–
–
Understand results and communicate among ourselves
To compare results of previous runs
To ultimately document and present what we did
To debug the codes
To aid with code setup
To do various qualitative and quantitative analyses
LAUR-03-7666
• These are very difficult problems we are trying to understand
• Use large number of visualization and analysis techniques to
answer these questions
• Some of these are billion dollar questions
• Visualization is essential (per the users!)
Big Viz Example
• For this run…
– Users ran many 2D preparatory studies
• Did visualization and analysis with EnSight
– Users ran a small number of 3D tests
• Did analysis and visualization with EnSight
– Ran EnSight visualizations while big 3D code was running
• Not computational steering
• Monitor progress of code
• Get ahead of the curve: problem space is well understood
LAUR-03-7666
– Users spent 6+ months doing viz and analysis on 21TB
Big Viz Example
• Remote visualization to Los Alamos from White
– Obviously can not transfer the data from LLNL
• Or even from Q to the LANL viz machines for that matter
– OC-48 connection
– 50 EnSight servers running on White
• Read and extract geometric data (e.g. iso-surface)
– Geometry sent to LANL and visualized on SGI pipes
– Geometry extraction results in about 5% of the data (avg.)
• So, for ~500M cells, can get 50M triangles (2-1 ratio)
– This works extremely well
LAUR-03-7666
• Users would like it faster though
What’s Next
• Visualization has been extremely successful at LANL
– Has really enabled codes to advance at rapid rate
– Users are pleased with production results
– Visualization presentations really help executives to understand
what we’re doing
• And this in turn helps the Laboratory
• But, users are demanding more and better services
– More rendering resources
– More offices served
– Faster and more useful on large data
LAUR-03-7666
• Research teams continue to work to transition results into
production
What’s Next: Clusters
• Commodity cluster rendering
– All DoE labs have been working on this for years
– Now time to transition to production use
• Initial effort underway at Los Alamos
– 64 node cluster to be use exclusively to drive theater wall
– 24 nVidia 3000G cards, 40 nVidia 5900 cards
• Stereo and genlocked
– Point-to-point from cluster to projector using DVI modems
– CEI developing cluster version EnSight client
• Same client/server paradigm: can viz remote data on cluster!
• Using Chromium
LAUR-03-7666
– Research tools will run on cluster
What’s Next: Digital Video
• We need to serve more offices with all this additional
rendering power
– We will more than double our existing rendering resources
with the 64-node cluster
– We will be adding many more clusters over the next few
years (64-512 nodes)
– Need to devise a way to feed offices, collabs, theaters, etc.
• Large DVI switches not available
LAUR-03-7666
– Lots of work going on in this area, some funded by DOE
– At low costs it may be feasible to dedicate cluster nodes to
offices or displays and simply go point-to-point over DVI
modems
Questions?
• Thanks to…
LAUR-03-7666
– Bob Kares: chief Los Alamos visualization user
– Bob Tomlinson: Los Alamos VIEWS program manager