eCensus 2011: A case study
IBM Smarter Planet – eCensus
Rod Hennegan
Lead Architect eCensus 2011 & 2006
Global Business Services
IBM Australia
eCensus 2006 awards
• Event
– eCensus 5 weeks
– Australian open 2 weeks
• Mass scale
– 2011 eCensus
– 2011 Australian Open
• 10 Million users
• Peak (final)
2
27
/0
28 7/2
/0 00
6
29 7/2
/0 00
6
30 7/2
0
/0 0
6
31 7/2
/0 00
7 6
1/ /20
08 06
2/ /20
08 06
3/ /20
08 06
4/ /20
08 06
5/ /20
08 06
6/ /20
08 06
7/ /20
08 06
8/ /20
08 06
9/ /20
0 06
10 8/2
/0 00
6
11 8/2
/0 00
6
12 8/2
/0 00
6
13 8/2
/0 00
6
14 8/2
/0 00
6
8
/
15 2
/0 00
6
16 8/2
/0 00
6
8
/
17 2
/0 00
6
18 8/2
/0 00
6
8
/
19 2
/0 00
6
20 8/2
/0 00
6
21 8/2
/0 00
6
22 8/2
/0 00
6
23 8/2
/0 00
6
24 8/2
/0 00
6
25 8/2
/0 00
6
26 8/2
/0 00
6
27 8/2
/0 00
6
28 8/2
/0 00
6
29 8/2
/0 00
6
30 8/2
Mass Scale Internet Event
% of total submissions per day
50%
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
• 4.8 million users (50% of dwellings in Australia)
• Peak (Census night)
eCensus and Open/Olympics
• eCensus
– Collection system
– Private data
• Tennis Open/Olympics
– Dissemination
– Public data (Broadcast)
eCensus
eCensus
SSL encrypted tunnel
Internet
Private
data
eCensus
Respondent
Servers xN
Open/Olympics
Australian Open/Olympics
15
1
Internet
Public data
Fan, Viewer
3
Caching
layer
Servers xN
Census night - very public event
“Call me cynical but I was just waiting for eCensus to fall over in a
pathetic heap….”
Personal BLOG of
• Census night
– High visibility (Census night)
– Peak load period
• System outage on Census night
– Shhhhh, maybe no-one will notice 
– ABS and IBM reputations
• Key areas
–
–
–
–
4
Hardware/Software/System failures
Malicious attack
Human error
Impact on other systems/clients
Computer World Journalist
Rod Gedda 9 Aug 2006
The ABS – IBM relationship
IBM
(Data collection & delivery)
• Contract
– Application development
– Create hosting environment
– Testing
•
•
•
•
System testing
Usability
Web security testing
Performance & Availability
– Hosting the event
•
•
5
Operations management
Security operations
ABS
(Data analysis & dissemination)
• Contract
– Requirements & Questions
– User Acceptance testing
– 3rd Party testing
•
•
•
•
Usability
Security (application & hosting)
Performance
Accessibility (Vision Aust.)
– Manage event
•
•
Field force management
Supervision of IBM access
Security/Privacy of Census data
• Aust. Government (PSM/ISM)
• Census and Statistics Act 1905
– Encryption on all network links
– Employed under the act to access raw census data
• Encrypted when written to persistent storage (database)
– Custom security solution (Hardware security modules)
– Decrypted (visible) only to
» ABS after delivery
» Respondent (application processing a respondent request)
– ABS legal review (post 2006)
• Access under the act = plain text, encrypted, or physical
• No IBM Staff access to privileged accounts
– Except under direct ABS supervision
– Supervision process remote supervision of remote staff
6
Security/Privacy of Census data
• Enforcement
– Application source reviewed 3rd party
• Ensure implementation of these mechanisms
• Prove production version = code reviewed
– Extensive cutover process
• All privileged accounts passed to ABS
• Secure racks locked by ABS
– Post production data deletion process
• IBM still responsible operations
– Extensive monitoring and automated failover
– Automation of required functions (stop/start etc)
– Supervised access
7
eCensus application
• Web Browser based – why ?
– Multi-platform
• Run on browser on virtually any operating system
– Removes install process support
– Zero footprint solution
• No plug-ins, no requirement to install Java, Active X controls
• eCensus implementation
– Two distinct user interfaces optimised for
• User experience
• Accessibility – (W3C Accessibility AA)
8
eCensus application
• Implementation approaches
– Use a Web Forms product
• XForms standards based (Others like Adobe forms)
• Pixel perfect replication of a paper form
– Custom Web application
• Build and present web pages containing questions
• 2006 eCensus solution
• Stateless application (0 MB per session at server)
– Generic Questionnaire engine
• Questionnaire definition in XML
• Application interprets definition
• Requires 1/3 server hardware of 2006 solution
9
Generic Questionnaire engine
• XML definition (Abstraction of a questionnaire)
–
–
–
–
–
–
–
Questionnaire –> Forms –> Questions -> Responses
Question response types (Text, Date, Address, Selection)
Question sequencing (Conditions based on responses)
Use of response content
Define business rules for responses/across questions
Tabular presentation for Text and Select questions
Consistent presentation through use of CSS
• Support virtually any internet questionnaire
– Population Census, Agricultural Census
– ABS strategic surveying solution
10
Using the eCensus intellectual property
• How can we utilise the eCensus technology
– ABS provides surveying services
– eCensus application and technology
• eCensus is not a product
• ABS - rights to the IP for the application
– Contact Paul Lowe (paul.lowe@abs.gov.au)
• IBM - eCensus application & all related services
– Commercial: Robert Howdin (rhowdin@au1.ibm.com)
– Technical: Rod Hennegan (rod.hennegan@au1.ibm.com)
• IBM - Smarter planet initiative
– Innovative solutions to business opportunities
– Services, COTS/custom software, middleware, hardware
• Innovating for 100 years – its what we do and we do it well
11
eCensus 2006 Awards
• 2007 Computer World Honours Laureate
– Honouring Those Who Use Information Technology to
Benefit Society
http://www.cwhonors.org/viewCaseStudy.asp?NominationID=312
 Winner 2007 Australian Excellence in
e-Government Awards
http://www.finance.gov.au/publications/excellence-in-e-governmentawards-2007-finalist-case-studies
12