National Data Network - summary of experiences
Introduction
Development of the National Data Network (NDN) commenced in 2004. The Australian
Bureau of Statistics (ABS) was the lead agency in developing the NDN in collaboration with,
and advice from, other parties including Professor Fiona Stanley from the Australian
Research Alliance for Children and Youth (ARACY). In July 2008 the ABS made the
decision to cease further development on the network. This report gives a brief overview of
the project and the lessons learnt.
Background
The NDN vision at the beginning of the project was "the best utilisation of, and return on
investment from survey and administrative information resources by:

providing users (e.g. planners, researchers, policy analysts, project managers,
evaluators) with an improved means of finding and accessing information resources that
aren't publicly available, and

providing custodians of information resources with a means of giving greater visibility
and wider access to their information resources, while continuing to give close attention
to access and use conditions."
The aim of the NDN was to develop low cost infrastructure that would provide a distributed
library of data holdings relevant to policy analysis and research. These data holdings would
remain held and controlled by their custodian organisations. A key element of this was that
software would need to be installed in other agencies.
The Australian Statistician convened the NDN Interim Governing Board to govern the
development of the network. Membership of the board consisted of agencies who
participated in developing the NDN. Governance of the NDN also included input from the
National Statistical Service (NSS) Board and the Children and Youth Statistical Portal
Reference Group. The NSS Board comprised ABS senior managers who provided a
strategic focus for NSS activities, of which NDN was an important component. The CYSP
Reference Group provided guidance on the development of the portal and involved various
organisations in the field of children and youth policy development. The Children and Youth
Statistical Portal is discussed in more detail later in this report.
While the ABS took a leadership role in the NDN project, it did not intend to have exclusive
ownership. It was expected that other agencies would contribute to the development as the
project evolved. In reality there was limited participation from other agencies (in terms of
making resources available on the network) and the cost of development was met wholly by
the ABS.
Experiences
The first prototype of the NDN was developed in July 2004, followed by the first public
release in August 2005. In the first few years the focus was on agreed metadata principles
and the development of search and discovery metadata that would be consistent with other
networks and systems.
In January 2006 the NDN website was launched and in September 2006 an updated
version of NDN software was released. In September the Address Coder service was also
introduced as an example of a web service that could be made available through the NDN.
In the first few years the take up from other agencies in publishing resources to the NDN
was very low. The belief at the time was that the benefit to agencies to participate would
become increasingly obvious as the project progressed. Many agencies "sat back" to see
what would happen before committing to the project.
In mid 2006 the Children and Youth Statistical Portal (CYSP) was recognised as a useful
project for the pilot phase of NDN development. In October 2006 a portal prototype was
released to a limited group to gather feedback and suggestions for the upcoming pilot portal
release.
In July 2007 the pilot phase of the NDN commenced. As well as CYSP, the Victorian Child
and Monitoring System (VCAMS) was also identified as a pilot project to help demonstrate
the value proposition of the NDN to users and custodians.
Children and Youth Statistical Portal (CYSP)
A pilot version of the CYSP was launched in October 2007. The aim of the portal
was to become a single source for detailed information resources on children and
youth, and to improve both the visibility and accessibility of these resources. The
portal would allow custodians to register and expose their information resources,
providing a foundation for best possible evidence based policy decisions, and to
allow the best use of these resources.
The pilot completed in June 2008, and was evaluated with the main findings listed
below.

There were limited agency resources made available through the portal (the vast
majority were ABS resources). No "non-public" resources could be loaded (due
to security issues) and so all of the resources were already publicly available
elsewhere. This lack of "critical mass" made it difficult to demonstrate a clear
business benefit to custodians and users and as a result usage was low.

Due to resource and time constraints, the portal lacked extensive design features
and functionality. For example, plans to have links to 'children and youth' related
organisations, newsletters and events calendars, or to enable the portal to store
and play video/audio files were not implemented. Navigation was also
considered poor (e.g. it was not possible to get back to the portal home page
from the portal forum.)

There was a focus on demonstrating technical features rather than the business
benefits. For example there was too much emphasis on how the NDN was
structured and not enough about how clients could use and derive benefit from
the site.

The prototype used meant that code deficiencies in the design were not
addressed through the pilot.

There were many difficulties for custodians in installing the software.

The metadata template was considered overly complex. The explanatory text
was seen as incomplete and confusing. Some users also thought more
mandatory fields were needed (most of the fields were voluntary resulting in lack
of detailed metadata for published resources).
Victorian Child and Adolescent Monitoring System
The Victorian Government is developing a comprehensive, across government,
monitoring system whose aim is to monitor and report on the safety, health,
development, learning and well-being of children and young people in Victoria. The
project is being managed by the Statewide Outcomes for Children Division (SOCD)
in the Victorian Department of Education and Early Childhood Development
(DEECD). This system is known as the Victorian Child and Adolescent Monitoring
System (VCAMS), and is intended to underpin planning for improvement at a
program, local government and statewide level, as well as inform research and
evaluation to generate new evidence on effectiveness. A key component of VCAMS
is to develop an interactive web-based system where data can be made directly
available to government, researchers and the public.
The data to be available through VCAMS will be based on an Outcomes Framework
(comprised of 35 outcomes and 150 indicators), related to children's health, safety,
learning, development and well-being. A number of Indicators are used to measure
each Outcome in the Framework. For example, healthy weight (outcome) is
measured by children who are overweight and children who are obese (indicators).
The system will be able to provide access to and query options for data on all the
indicators. Data for VCAMS will be derived predominantly from across-government
administrative systems and new data collections being implemented by the SOCD.
In October 2007 SOCD agreed to undertake a pilot project with the ABS to explore
the potential benefits and issues of using an NDN based solution to efficiently and
securely make VCAMS data available to users through a web-based interface. The
outcomes of the pilot demonstrated the capability of the NDN to operate as part of
an end to end solution for VCAMS. There were significant time savings for staff and
users were able to obtain community profiles from an electronic, query capable
system. However, the short time-frames and difficulties associated with establishing
new technologies in departments meant that the solution was entirely developed
using ABS infrastructure, rather than the IT infrastructure of Victorian departments.
As a result, some key issues were uncovered - the need to develop detailed data
sharing agreements and the need to work out IT requirements for housing 'Nodes' in
departments.
Following the pilot, in-principle support was given for the use of an NDN type
solution. An interdepartmental working group has been established to manage the
project and oversee the transition to a production system using an NDN approach to
securely transport data. The project is still in progress and the ABS is providing
technical and statistical advice.
The project provided some valuable lessons for future collaborative projects. These
are listed below.

Value of 'executive level' champion and engagement, which flowed down to
engagement at an operational level.

Importance of who will host the project, especially a cross government project
(ownership, agency neutrality etc).

Value of an out-posted officer. The placement of an ABS officer within SOCD:
 enabled ABS to engage with the business and better understand the
business problem, which was essential to ensuring engagement with
SOCD and determining ABS opportunities
 engendered trust with SOCD
 facilitated access to senior staff, and allowed SOCD to feel comfortable
that their needs were being represented
 engendered a strong sense of working with SOCD to achieve common
outcomes, rather than working for them, and allowed opportunities for
deeper statistical engagement to be taken as they arose.

Being able to demonstrate the business benefit for SOCD (and the data
providers)

Understanding the governance arrangements
 The governance arrangements for SOCD and VCAMS were quite
complex, and involved relationships with working groups and committees
across a range of Victorian agencies.
 Understanding these arrangements and the IT governance allowed
strategies for influencing the outcomes to be developed with the help of
SOCD.

Being flexible and responsive
 As our understanding of the environment developed, the approach
changed significantly.

Working to a different/longer timeframe than expected
 Achieving outcomes in this project sometimes took longer than expected
which forced us to re-focus our expectations, planning and scheduling.
Collaborative undertakings require give and take from both parties and an
understanding of each party's operating framework.
Challenges
In late 2007 a number of security issues were discovered with the NDN and work
commenced on preventing "hacking" attempts to access the network. This work hindered
further development of the CYSP as most resources were put towards rectifying the security
concerns. Further investigations revealed a number of areas requiring further work,
including development of technical documentation and improving the consistency in the
implementation of business functions. This work, and experience over the last four years,
highlighted that development work up to that time had in fact been done in a demonstration
mode and that significant work was required to develop a full production capability. These
developments, as well the experience with the CYSP, indicated the time was right for stocktake and review.
Early in 2008 the ABS conducted three reviews covering the operational, architectural and
strategic aspects of the NDN developments. These reviews showed that developing IT
infrastructure and interfacing with existing IT infrastructure of other organisations is a highly
complex process and requires significant and specific investments in the open software
development environment. There have also been significant advances in technology
applications since the NDN project commenced. Further development of the functionality of
the NDN would require considerable time and financial investment around issues such as:
 security of access
 metadata services
 availability of web services
 user take-up
 better processes to sufficiently maintain and organise data/information.
The ABS did not receive any financial support for the NDN project. Funding for
development of the NDN came wholly from within the ABS budget. The ABS financial
position does not support it investing alone on the scale needed to see the project through
to a robust production state.
Useful learnings
There have been valuable lessons learnt and some excellent collaborations established
during the development work on the NDN. The pilot projects have shown that sharing and
linking data across governments is appropriate and there is increasing demand for it.
However, there are key learnings that need to be kept in mind for future projects. These are
discussed below.
Ensure the project is driven by business requirements and not by technology. The
project team needs to be aware of the bigger picture - why are we doing this? To
some extent the NDN project was drawn into the technology at the expense of
understanding the business drivers and needs of users.
Related to this, there needs to be strong demonstrable business value (buy in) for
agencies to collaborate and contribute resources. This involves learning more about
how potential collaborators work and their needs. If there is 'executive level' support
and engagement then there is more likelihood of engagement at the operational
level. In the case of CYSP, there were quite a few very interested stakeholders who
'signed up' to be participants in the pilot. Despite the enthusiasm, limited resources
were made available through the portal, few non-ABS resources were accessed,
and the 'partners' did not use the system. In retrospect, the identified business
needs lacked rigour and hence the 'buy-in' was not as committed as it could have
been.
Keep the scope of the project manageable. If the scope is too ambitious then it can
be difficult to define and therefore too difficult to manage. The ambitions of the
NDN, while aspirational, needed to be established as a set of incremental goals.
There needs to be strong project management, in particular regular reviews on
where you are up to and being realistic about what can and can't be achieved. In the
case of the NDN, this translated to recognising earlier in the development process
that substantial work was required to progress from a prototype to a full production
system.
Use an approach that keeps future adaptability in mind. In terms of IT don't push for
adding functionality at the expense of future adaptability. The technical environment
can change very quickly and the project needs to be able to adapt to this. With the
NDN, 'Rich Client' was used as the user interface. This software was too closely
linked to the rest of NDN software so that whenever updates were made to the NDN,
Rich Client also needed to be updated. As well as resulting in extra work for
developers, updated software had to be sent out to all users who then needed to reinstall the software.
An architecture that requires installing software in different IT environments (even
"open source" software) is likely to have difficulties and high support costs. There
were many technical problems that could not be resolved. The ABS was unable to
replicate many problems within its own IT environment that were experienced in
other IT environments.
Be aware that appropriate governance arrangements and IT environment are
needed to support open source development and collaboration.
Where to from here?
The ABS is committed to an Australia where consistent, high quality, accurate information
on the economy, society and the environment are easily accessible at any time. The
benefits to the community from improved decision making would be far reaching while a
streamlined system would also reduce the collection burden placed on businesses and
households.
Advances in technology have provided infrastructure to greatly improve the information
base of the nation with enhanced abilities to collect, store, process and 'communicate'
information. However, a national statistical service requires more than just technology,
although it is an important component. It also requires the widespread use of data
standards, frameworks and classifications, increase statistical capability across government
and inter-agency cooperation and data-sharing.
The Australian Government has demonstrated a keen interest in the use of evidence to
support policy development and evaluation. While there is a wide range of statistics being
produced across all jurisdictions through statistical collections, administrative and
transactional databases to meet this demand, it is clear that to get the best value from these
sources requires a coordinated approach to national statistics.
The Australian Bureau of Statistics has responded to this challenge through an initiative
titled the National Statistical Service.
The National Statistical Service
The National Statistical Service (NSS) is the community of government agencies building a
rich statistical picture for a better informed Australia. The specific objectives of the NSS are
to:

deliver a high quality, up-to-date, comprehensive, coherent statistical picture of the
economy, society and the environment to assist and encourage informed decision
making, research and discussion within governments and the wider community;

provide a world class official statistical service that retains the confidence and trust of
the Australian society as both providers to, and users of, the resulting official statistics;

maximise the use for official statistical purposes of data available within government
administrative systems by government agencies;

minimise the burden of statistical reporting at all levels of the Australian community;

document and retain as an enduring national resource key statistical outputs and their
underlying data sources.
The ABS is continuing to work with some of the NDN collaborators and other agencies. As
examples, the ABS is currently involved with the following projects:
 The development of a digital rights management framework with the
Queensland University of Technology OAKLaw Project and the Queensland
Office of Economic and Statistical Research
 Continuing work with the Victorian Department of Education and Early
Childhood Development on VCAMS
 Development of web service feed for Census data with the WA Landgate
Shared Land Information Platform (SLIP) Project
 Providing assistance to the Department of Human Services with developing a
formal proposal for a Commonwealth Social Spatial Data Initiative Platform.