High-Throughput Computing to Enhance Intervisibility Analysis
High-Throughput
Computing (HTC)
• Metacomputing: the network is the
computation resource
• Run many jobs simultaneously on
networked processors e.g. for parameter
sweeps, sensitivity analyses, etc.
• Processors are not dedicated
• Middleware harvests spare compute cycles
Schematic
Coordinator
Starts and monitors
Condor jobs.
Builds CID.
ARC/INFO
Condor
Intervisibility Analysis
•The problem
Complex applications based on
intervisibility are constrained by runtimes.
•Single-processor approaches
Attempt optimisations to fit the
analysis onto one machine.
•Multi-Processor Solution with HTC
Simplify the analysis by generating
visibility viewsheds for all posts in a
Digital Elevation Model (DEM), so
creating a Complete Intervisibility
Database (CID), accessible for many
applications.
Condor Middleware
•Used to manage concurrent ARC/INFO
visibility analyses on networked PCs
•Distributes input data and gathers results
•Options for CPU-time harvesting: control
job start-up and suspension (when
other activity increases on a PC )
•Now supports Windows as well as UNIX
http://www.cs.wisc.edu/condor/
M.J. Mineter1, S. Dowers1, D.R. Caldwell2 and B.M. Gittings1
1:
Geography, School of GeoSciences
University of Edinburgh,
Drummond Street, Edinburgh EH8 9XP, Scotland
Tel +44 131 650 2662
Fax +44 131 650 2524
Email: {mjm, sd, bruce}@geo.ed.ac.uk
2:
U.S. Army Engineer Research and Development Center
Topographic Engineering Center
7701 Telegraph Road, Alexandria, VA, 22315-3864, USA
Tel +1 703-428-6802 X2236
Fax +1 703-428-6425
Email: Douglas.R.Caldwell@erdc.usace.army.mil
Prepared for GeoComputation 2003, 8-10 September 2003
DEM
Zip
Condor
Networked PCs running
Windows
CID
Intermediate Files
Coordinating processor
Design Issues and Results
•Computation time
60 CPU hours harvested in 3.5 hours
using 37 personal computers
•Number of Viewsheds
(336*466) dataset: 156576 viewsheds
•CID File structure
•16 viewsheds are held in one BIL file, for a
4x4 subgrid of observer posts.
•Row of BIL files zipped into one file
•Enables easy random access to viewsheds
•Data volume:
Number of bits= (336*466)2. ~ 2.9GB
Compressed to 3% or 84 Mb by ZIP
•Interactive applications
Java used to access the CID; extensions to
ArcMap using VBA, ArcObjects are loosely
coupled to the Java via shell commands and
pipes
•Metrics
Products descriptive of viewshed
characteristics. See poster by Caldwell et al..
Shaded Relief Image
Prototype DEM Data Set
Southwest Harbor, ME
Source: USGS
336 columns x 466 rows