Birds of a Feather:
Enterprise Condor Use Cases
Jason Stowe
Condor In “Enterprises”
Experience in Production
Environments
What is an ‘Enterprise Environment’?
Companies Only?
Any Organization Using
Condor with
Demanding Users
Organization =>
Groups of Demanding Users
Purchased
Computer Capacity
Need
Computation
Done.
Easily.
On a Deadline.
In-House,
Third Party Applications
without modification
in order to do their jobs
Not just companies
Includes
Research/Gov’t/Academic
environments
A valid definition of
Enterprise Condor Users?
I started off as a
Demanding Condor User
Condor Week 2005
Condor for Movie Pipelines
Condor Week 2006
Web-based Management Tools,
Consulting, and 24/7 Support
A Conversation with Miron
Add new (Enterprise)
Features to Condor
Condor Week 2007
Negotiation Features for
Demanding Groups of Users
So we’ve defined Enterprise, but
Why Condor?
Summarized by Bob Nordlund
In 2007
http://www.cs.wisc.edu/condor/CondorWeek2007/presentations/nordlund_Condor_In_Flight.ppt
"Condor was selected...
-Multi-OS Support
-Simple, Scalable, and Flexible
-Active Community
-Free"
Multi-OS Support
Windows
Many Varieties of Linux
Solaris
And if you’ve got it, Mac OSX
Condor works with most anything
Keeps users’ options open
Easy to Download and Use
Run any executable as a job
Makes parameter sweeps,
monte-carlo simulations easy
Scalable
Many Facilities with
X000-X0000 condor VMs/slots
Flexible
Not just OS/Platform flexible
Start = (NonCondorLoadAvg < 0.5)
Control when jobs run where
Powerful, Command Lines,
Developer APIs
(DRMAA/SOAP)
Condor Community
com·mu·ni·ty
A group of people living … under
the same government.
A group of people having common
interests: the scientific community
… Sharing, participation, and
fellowship.
Support …
Sharing, participation, and
fellowship.
So that’s why Condor in an
Enterprise, but
How are they using it?
Commercial Use Cases
Batch Monte-Carlo Computation
with Windows/Linux/Solaris
Embarrassingly Parallel
From Bob Nordlund
of HartfordLife
@CondorWeek 2007
http://www.cs.wisc.edu/condor/CondorWeek2007/presentations/nordlund_Condor_In_Flight.ppt
Faced a Challenge:
“
-Prepare comprehensive risk analysis report for all products
for investment analysts.
-Run three orders of magnitude more projections through the
system.
-Perform stochastic on stochastic on stochastic analysis.
-2.4 billion individual simulations.
”
IT Responded:
“
Infrastructure:
-500 dual-core, dual-CPU AMD servers
-20 Racks
-Dedicated GigE switches
-Condor pool with HAD
-4 filers with 8TB of storage
-Purchased, installed and configured in 6 weeks
Application
-Java-based liability model
-75x performance gain
-Meta-scheduling application
-In-memory market scenario generation
"
Specifically, it mentions Condor:
“
In production since 2004
...
Dedicated and non-dedicated execute nodes
~1000 Two-socket, dual-core HP x86 servers
~1000 desktops, notebooks
Linux central managers with HAD
Linux and Windows schedulers
Windows execute nodes
”
Using the same Condor installation.
Condor scales
Without changing software,
end-user training, etc.
Just added more hardware.
Workstation Pools –
Finance, Geospatial Imaging
Workstations are only utilized
1/3 of the day, intermittently
Considerable
Latent Compute Capacity
Perfect for Overnight Batch
and Master/Worker jobs
Effective Policies for
Idle Workstations
As mentioned in WatersOnline
In 2007
http://www.watersonline.com/public/showPage.html?page=468641
Some finance institutions have
saved considerable $$$
Running Pools of X000s
Workstations
3rd Party Master Worker Jobs
MatLab, MoSes, TAS, etc.
Condor Can Manage Resources
= Worker Counts
So on a 100 machine pool, with
applications A & B
50
50
Machines Machines
A = 50
B = 50
Lastly
Movies
CORE Feature Animation on Disney
Picture, "The Wild"
Condor for Movies:
75+ Million Jobs
1000+ CPUs (Linux/OSX)
70+ TB storage
From my slide in 2005 Condor Week:
“
1000 2.8 GHz. Processors, Linux 4GB RAM
70-100 Terabytes Several Filers
50 Million Renders so far
(Vanilla Universe)
”
Batch, with many, many
Groups of Demanding Users
But That’s just a few use cases…
Other CondorWeek presenters:
Altera, Micron, Oracle, etc.
You may have new ones…
Talk with Condor Team.
Talk with other users. Talk with me.
Help the community continue to
work well for everyone.
Thank you NeSC.
Questions?
http://www.cyclecomputing.com
jstowe @ cyclecomputing.com