Scaling Condor on XSEDE for
LIGO
Peter Couvares
Syracuse University
LIGO Scientific Collaboration
Who am I? What is LIGO?
• Former Condor Team member (‘99-’08).
• Now at Syracuse University focused on
distributed computing problems for the LIGO
Scientific Collaboration, and fostering a research
computing community at SU more generally.
• LIGO (the Laser Interferometer GravitationalWave Observatory) is a large scientific
experiment to detect cosmic gravitational waves
and harness them for scientific research.
• http://ligo.org/
The Project
• The Charge:
– demonstrate whether LIGO can effectively utilize
XSEDE resources for its large-scale computing.
(And if not, why?)
• The Challenge:
– LIGO’s existing computing model doesn’t map
perfectly to XSEDE.
Four Talks
•
•
•
•
The political story (NSF)
The cultural story (LIGO + TACC)
The architectural story (what we did)
The technical story (how we did it)
The Political Story
• LIGO plans to buy millions of $$$ of
computers later this year to be ready for the
Advanced LIGO detectors as they come online.
• LIGO has always done most of its computing
“in-house” on dedicated LIGO clusters, with
good results – so we haven’t tried very hard
(at least not lately) to utilize opportunistic
resources we don’t manage ourselves*.
* notable exception = E@H
The Political Story
• Before writing us a check, the NSF wanted to
understand why we only planned to buy our own
private clusters, when some other large NSF projects
are successfully using (or contributing to) shared
resources.
• Given the size of the check, the NSF also probably
wanted to know whether we were doing our
computing sensibly, and weren’t building something
unnecessarily inefficient or eccentric.
• WARNING: this is my speculation based on fourth-hand
accounts of other people’s guesses. I could be wrong.
The Cultural Story
• The NSF asked LIGO to see if it could run some
or all of its large-scale computing work on
XSEDE.
• Stampede was the closest thing to an HTC
cluster in XSEDE, so the NSF told LIGO and
TACC to work together on it.
LIGO View of XSEDE Resources
LIGO View of LIGO Computing
Shotgun Wedding
Shotgun Wedding
• LIGO: “We don’t need a car with 12 cylinders
and molybdenum brakes to commute to work.
These Hyundais we’ve got lined up are fine.”
• TACC: “You need how many cars?!?”
Shotgun Wedding
• LIGO: So, we just need Condor everywhere, no
firewall, a bunch of yum repos and RPMs
installed on all your machines, single-sign on
for our users using their LIGO.ORG credentials,
and the ability to run VMs as jobs.
• TACC: Uh, we don’t normally do any of that
stuff. And no way are you running VM jobs.
Shotgun Wedding
• TACC: Have you optimized your code?
• LIGO: Who do they think we are, amateurs?
Have we optimized our code! Harrumph!
• TACC: Here, look at these FFT results.
• LIGO: Oh. Uh… wow, that’s faster. Nice!
The Cultural Story
• Like any shotgun wedding, neither party were
thrilled to be at the altar under duress.
• But we got to work, and quickly dropped the
grumpiness.
• The TACC staff turn out to be great to work with,
have all kinds of valuable expertise LIGO can use,
and have been extremely helpful.
• Despite the impedance mismatch, together we
succeeded in running a production LIGO
workflow, at scale, on Stampede.
The Architectural Story
Key points of contrast between the LDG and Stampede:
• Central NFS fileservers (LDG) vs. Lustre DFS (Stampede).
• Persistent compute nodes w/state (LDG) vs. transient/stateless
execute nodes (Stampede).
– LDG uses persistent local disk for distributed checkpointing
and Condor logging
• NFS for job input and output, local scratch disks for runtime file
i/o (LDG) vs. Lustre for everything (Stampede).
• Condor batch queue system (LDG) vs. SLURM (Stampede).
• Scientific Linux 6.1 (LDG) vs. CentOS 6.5 (Stampede).
• Software pre-installed in system locations on dedicated resources
(LDG) vs. local builds on shared resources (Stampede).
• Long running jobs (LDG) vs. 48h maximum (Stampede)
15
Design Choice
Make LDG look more like Stampede
or
Make Stampede look more like LDG
Given our experience porting a large body of LDG software and
workflows to new OS platforms and versions, we knew it took
more time than we had, so we started with the latter and
worked back the other way when it was necessary or easier.
16
An LDG Site “Overlay” on Stampede
LDG Overlay on Stampede
• Glide-in Condor pool via SLURM
– Persistent Condor central manager
– Persistent login/submit machine
• Make heavy use of Condor standard universe checkpointing to handle
mismatch between SLURM scheduling policies and long running analysis
jobs with unpredictable runtimes.
• Pre-install LIGO software (RPMs) site-wide on Stampede.
• Use LDR and Globus for data transfer to Lustre via gridftp
• Setup LIGO web services in dedicated VM
– Data discovery service
– LIGO.ORG protected web site to post analysis results
• Enable access to LDG with XSEDE credentials and vice-a-versa
– Stampede an early XSEDE adopter of CILogon
18
The Goal
Validate the ability to transfer simulated aLIGO data from a
LIGO Engineering Run to Stampede and confirm that the CBC
offline detection pipeline can run and generate the same
results on Stampede as the LDG.
•
•
•
•
•
•
•
Select one LDG site (Syracuse) for detailed comparison runs.
Start with the Initial LIGO (iLIGO) pipeline and well understood input data.
Perform correctness and scaling tests
Optimize performance
Switch to the aLIGO pipeline currently being developed
Perform longer running stability tests
In the background allow for other small scale LIGO tests
19
Round 0
Setup systems for testing:
• Install LIGO software including Condor – PASS
– After a few iterations official releases of LIGO software from package
repositories where installed on all Stampede systems.
• Setup VM for LIGO to install and manage web services – PASS
– Took a few iterations including installing extra certificates and mailing
physical security tokens but straightforward.
– Minor change to LIGO web services authentication configuration to
handle different network topology at TACC.
• Setup 10G network transfer of LIGO data via Globus and LDR using
gridftp – PASS
– Took a while to track down a performance issue due to mismatched
MTU but eventually solved.
• Manually support registration of CILogon credentials before XSEDE
deployed that during the test
20
Round 1
Analyze one-day of LIGO data on Stampede
using iLIGO code:
•
•
•
•
21
Condor glide-in via SLURM – PASS
Data transfer via LDR – PASS
Central checkpointing – FAIL
Network firewall issues – FAIL
Round 2
Analyze two weeks of data:
• Solved initial firewall problems – PASS
• Improve security by moving Condor Central
Manager to dedicated host (VM) caused new
firewall problems – FAIL
• Try solving checkpoint scaling by having
parallel checkpoint writing and central resume
– FAIL
22
Round 3
Analyze 6 weeks of data:
• Condor code patch to support parallel checkpoint
save/restore to a shared filesystem without persistent
checkpoint servers – PASS
• Scaling to 9,000 concurrent jobs with synchronous
checkpiont/resume woke up TACC support team at
inconvenient hour – MIX
– >2000 load avg on submit node
• Moved Condor LOCK and LOG files to /dev/shm to reduce
load on Submit machine (temporary solution) – PASS
• Scaling to 25k concurrent jobs hit limit of single submit
machine at 13k jobs – MIX
23
Future
• Submit machine scalability (13k != 25k)
– Several straightforward ways to solve
• Submit fewer but multi-core jobs
• Split work between multiple Submit machines
• Further investigate/enhance Condor Shadow scalability
• Use a factory to manage glide-ins automatically.
• What happens when we don’t have a fortuitous
alignment of OS?
– Virtual Machines (not supported on Stampede)
– Restrict amount of needed software (focus on production
rather than development computing)
– Port necessary packages as opt-in modules
• Enhance LIGO packages to be relocatable as more
appropriate for a shared resource
24
Lessons
• It takes a lot of work to migrate a "big" computing system
to a new environment. Something has to give.
• It can be done.
• Miron might say we “cheated” by statically reproducing
much of our existing environment on Stampede, rather
than bringing it with us – but we had a deadline and it’s a
big first step.
– And the cultural accomplishment inside LIGO may end up being
bigger than the technical accomplishment…
• XSEDE sites like TACC have incredibly valuable expertise –
you should take advantage of it. Not being HPC-focused,
we underappreciated it before this exercise.
Lessons
• Speaking for myself, not LIGO:
• We should have been more optimistic, and more humble,
up front – but we got there.
• The NSF should be more clear about what’s going on when
it arranges this kind of thing, to limit FUD.
• While LIGO must manage its own significant computing
resources for some work (e.g., low-latency analysis,
detector characterization, software development, testing,
and training students), we can use shared resources like
Stampede today for a large fraction of our computing.
• Longer-term, LIGO should develop its “grid plumbing” to
enable more flexible use of other shared resources that
can’t be made to look like LDG sites as easily as Stampede.
Acknowledgements
• Apologies in advance to those I surely forgot…
• LIGO
– Stuart Anderson, Duncan Brown, Kent Blackburn, Josh
Willis, Patrick Brady, many others
• TACC
– Yaakoub El Khamra, Luke Wilson, John Cazes, John
McCalpin, Bill Barth, Nathaniel Mendoza, many others
• Condor
– Greg Thain, Alan De Smet, many others
• NSF
– Faceless bureaucrats who forced us out of our rut!