Geoscience Australia
Software Development at
the Australian Tsunami
Warning System
Background





2004 Boxing Day Tsunami.
Over 230,000 people die in Indian Ocean coastal areas.
In 2005, Australian Government decides to fund a
national Tsunami alert system.
The Joint Australian Tsunami Warning Centre
(JATWC) was born.
Became operational and launched in 2008.
© Commonwealth of Australia (Geoscience Australia) 2011
Overview




The JATWC is a joint mission between Geoscience
Australia (GA)and the Australian Bureau of
Meteorology (BOM).
GA’s mission is to detect Tsunami earthquakes with the
potential to impact Australia, and notify BOM and
other government agencies.
BOM’s mission is to perform Tsunami modelling,
assess impact on Australia (where, how bad), and issue
public and media alerts.
Operates 24/7 year round.
© Commonwealth of Australia (Geoscience Australia) 2011
GA Component - ATWS




The Australian Tsunami Warning System (ATWS) is the
GA component of the JATWC.
Was decided early on to use Antelope as the basic data
flow and processing framework.
Consists of an extensive software stack built on top of
Antelope providing all the capabilities needed to review
and communicate EQ alerts.
Is operated by duty seismologists who work in 12 hour
shifts, with one person on duty at a time.
© Commonwealth of Australia (Geoscience Australia) 2011
ATWS A Closer Look

System is spread across several servers:







DC – seismic data collection (orb2orb, seedlink etc.)
ALERT – event processing (orbassoc, evproc etc.)
IS – post-event review and analysis (dbloc2)
PUB – data dissemination (orb, seedlink, autodrm)
Most of our software runs on ALERT, including our
key Decision Support Tool (DSS) .
The majority of ATWS software is written in Perl, with
some components written in C.
Run on Solaris/SPARC servers.
© Commonwealth of Australia (Geoscience Australia) 2011
ATWS DSS “orbevents”





Main tool used to review automatic events and send out
alerts.
Uses a manual Mwp process to provide magnitude
estimates for EQs with mag > 6.
Is written in C using the Gtk+ graphics toolkit.
Sends both alerts (red) and notifications (green).
Used in conjunction with dbloc2, with the latter used
for more in-depth review, especially for small local
events.
© Commonwealth of Australia (Geoscience Australia) 2011
ATWS DSS “orbevents”
© Commonwealth of Australia (Geoscience Australia) 2011
Manual Mwp Tool
© Commonwealth of Australia (Geoscience Australia) 2011
Other Software

Heads up displays


Integration with Earthquake Catalogue


Web pages that show station latency and epicentres
on a map.
Automatically load automatic and manual solutions
into corporate Oracle database.
Automated alarm system

Watches automatic events and send out emails, rings
phones when triggered.
© Commonwealth of Australia (Geoscience Australia) 2011
Heads up display (EQs)
© Commonwealth of Australia (Geoscience Australia) 2011
Heads up display (latency)
© Commonwealth of Australia (Geoscience Australia) 2011
Process of Improvement


Kent and I identified several areas of improvement that
were critical in supporting a mission-critical system.
These include





Effective use of version control and SW packaging.
Deployment practices and configuration management.
Comprehensive system monitoring.
Planned release process (Agile methodology).
Have largely been implemented, resulting in much
better management of the system.
© Commonwealth of Australia (Geoscience Australia) 2011
Version Control




Some basic questions:
 What is currently running in production?
 What was the last change made?
Important to be able to answer these if serious about
running an operational system.
Version control is an enabler that allows them to be
answered.
Previously code was released, then sometimes checked
into VC as an afterthought.
© Commonwealth of Australia (Geoscience Australia) 2011
Version Control (cont)




Now all code must be checked into VC before it can be
released into production.
Also can see what changed by comparing two previous
release tags (useful for identifying cause of issues).
Supports release management – lets review what
changed before we release to production.
Source code only – maintain knowledge of how to
build, in case one day we need to!
© Commonwealth of Australia (Geoscience Australia) 2011
Version Control (cont)
© Commonwealth of Australia (Geoscience Australia) 2011
Deployment and Configuration





Consolidated configuration module common to all
servers release as a Solaris package.
We use a configuration management tool (Cfengine3)
that “knows” how to copy the right files based on the
current server to the right spot.
Only files that have been changed are updated.
Cfengine can also restart processes if needed.
Also provides documentation of system configuration.
Re-deploying a server is easy.
© Commonwealth of Australia (Geoscience Australia) 2011
System Monitoring






How do you know everything is running OK right now?
Comprehensive System Monitoring is key.
We use Nagios3 to do deep application level monitoring
across all processes on all servers.
Why Nagios? Standard, proven, open-source.
We monitor the status of all application processes,
including processes under rtexec.
How? Generate the Nagios configuration from rtexec
files for each real-time system automatically.
© Commonwealth of Australia (Geoscience Australia) 2011
System Monitoring (cont)
© Commonwealth of Australia (Geoscience Australia) 2011
System Monitoring (cont)
© Commonwealth of Australia (Geoscience Australia) 2011
System Monitoring (cont)
© Commonwealth of Australia (Geoscience Australia) 2011
System Monitoring (cont)
© Commonwealth of Australia (Geoscience Australia) 2011
Development process







Tools and technologies don’t do much alone.
They exist to support a process that ensures they are
effectively put to good use.
A development methodology is this process.
We follow an Agile methodology, loosely based on
Scrum.
Regular, planned release cycle or “sprints”.
Items of work are assigned to sprints.
Unassigned items are used to plan next sprint.
© Commonwealth of Australia (Geoscience Australia) 2011
The Future






Redevelopment of all ATWS software, including new
DSS which will be a major undertaking.
Possibly move to Python as our main development
language.
Migrate to Apple Mac or Linux in next few years.
Array processing.
New magnitude calculators.
Regional responsibilities.
© Commonwealth of Australia (Geoscience Australia) 2011
DSS Redevelopment


New heads up displays.
More information – answer the most common
questions of visitors:






When was the last Earthquake?
Who is on duty?
When was the last alarm/alert sent?
What did the PTWC get for this event?
Less distractions for duty officer.
Extra information sources, such as Twitter.
© Commonwealth of Australia (Geoscience Australia) 2011
DSS Redevelopment (cont)
© Commonwealth of Australia (Geoscience Australia) 2011
DSS Redevelopment (cont)






New “orbevents”.
Much greater functionality.
Seismic awareness, i.e. show detections on map.
Less clutter, focused on primary mission.
Timeline – clear indicator of how long left to make
decision, and when different events occurred (origin,
magnitude, etc).
External data – Twitter, felt reports, external solutions.
Increase situational awareness.
© Commonwealth of Australia (Geoscience Australia) 2011
DSS Redevelopment (cont)
© Commonwealth of Australia (Geoscience Australia) 2011
DSS Redevelopment (cont)
© Commonwealth of Australia (Geoscience Australia) 2011
Python




Perl has been main language of choice.
My background is in Perl.
So why even think about Python?
Actually lots of reasons, but 2 main ones:
Finding good Perl developers is hard. Really hard.
No-one wants to learn Perl – existing Perl
developers want to learn something else!
 Lack of bindings to modern Graphics Toolkits.

© Commonwealth of Australia (Geoscience Australia) 2011
Python (cont)





We expect the new DSS to rely heavily on advanced
graphics manipulation that need canvas like
functionality.
Gtk+, does not have a native canvas element.
An extension, Goocanvas, is available, but Perl bindings
are incomplete compared to Python.
Also, most modern cross-platform Toolkit available,
Qt4, has no Perl bindings but excellent Python
bindings.
Tk? Possibility, but seems like a step backwards.
© Commonwealth of Australia (Geoscience Australia) 2011
Python (cont)

Other benefits:







Simply, easy to read and maintain.
Can focus on the problem, not the language.
Becoming the worlds de facto standard scripting language,
bindings to just about everything.
Great, active worldwide community. Particularly in scientific
communities, including seismic ones (obspy).
Large developer community within GA.
Amazing range of first-rate libraries: numpy, scipy, matplotlib,
obspy, PyQt4, PIL, and many more…
No (or little) dependency hell.
© Commonwealth of Australia (Geoscience Australia) 2011
Python (cont)

Issues?






Antelope bindings, unclear future.
Multi-threading – works but issues (the (in)famous GIL).
Staff retraining, recruitment
Overall we think that the benefits speak for themselves.
Have already started prototyping application for the
new DSS in Python using Kent’s contributed Python
bindings for Antelope.
Settling on the Qt graphics framework in combination
with Python.
© Commonwealth of Australia (Geoscience Australia) 2011
Python (cont)
© Commonwealth of Australia (Geoscience Australia) 2011
Mac vs. Linux





Big question running through the Antelope community.
Antelope on Solaris has been EOL’d (2013), future
Linux support uncertain.
We have a corporate preference for Linux, as it plays
well in an Enterprise environment.
Lack of Mac expertise, and questions remain about
how Mac’s can operate in a data-centre context.
Redundant power supplies, LDAP, SAN, Lights-out
management (LOM), virtualization, updates.
© Commonwealth of Australia (Geoscience Australia) 2011
Magnitude Calculators





We currently have customized Ml, Mb, Ms (Msx)
modules running under orbevproc.
We also use the standard AutoMwp calculator.
CMT process works outside of orbevproc (trexerpt).
Want to develop MT, Theta, MCoda, WPhase, etc.
New orbevproc model for running these modules:



Let orbevproc gather, filter and process waveforms.
Write to SAC files, manipulate SAC headers.
Call external “black-box” module to run against SAC files.
© Commonwealth of Australia (Geoscience Australia) 2011
Questions?
© Commonwealth of Australia (Geoscience Australia) 2011