EVALUATION STRATEGY
2015-2019
INDUSTRY.GOV.AU
Further information
For information on other department initiatives please see the department’s website at:
www.industry.gov.au/OCE
For more information or to comment on this publication please contact:
Martine Rodgers
Evaluation Unit
Department of Industry and Science
GPO Box 9839
CANBERRA ACT 2601
Telephone: +61 2 6213 7194
The views expressed in this publication are those of the authors and do not necessarily reflect those
of the Australian Government or the Department of Industry and Science.
© Commonwealth of Australia 2015
ISBN: 978-1-925092-67-7 (print)
ISBN: 978-1-925092-68-4 (online)
This work is copyright. Apart from use under Copyright Act 1968, no part may be reproduced or
altered by any process without prior written permission from the Australian Government. Requests
and inquiries concerning reproduction and rights should be addressed to
chiefeconomist@industry.gov.au. For more information on Office of the Chief Economist research
papers please access the department’s website at: www.industry.gov.au/OCE.
Creative Commons Licence
With the exception of the Coat of Arms, this publication is licensed under a Creative Commons
Attribution 3.0 Australia Licence.
Creative Commons Attribution 3.0 Australia Licence is a standard form license agreement that allows
you to copy, distribute, transmit and adapt this publication provided that you attribute the work. A
summary of the licence terms is available from http://creativecommons.org/licenses/by/3.0/au/deed.en.
The full licence terms are available from http://creativecommons.org/licenses/by/3.0/au/legalcode.
The Commonwealth’s preference is that you attribute this publication (and any material sourced from
it) using the following wording:
Source: Licensed from the Commonwealth of Australia under a Creative Commons Attribution 3.0
Australia Licence. The Commonwealth of Australia does not necessarily endorse the content
of this publication.
Contents
Introduction
1
Performance measurement and reporting
2
Impact on evaluation activity
3
What is evaluation?
4
Good evaluation practices – a consistent approach to evaluation
5
Governance
6
Executive Board
6
Steering Committees and Reference Groups
6
Evaluation and Audit—what is the difference?
6
Assurance and Audit Committee
6
Internal Audit role
7
Evaluation Unit role
7
Planning for evaluation across the policy and programme lifecycle
8
Evaluation types
8
Evaluation readiness
9
Prioritising evaluation effort
10
A four-year evaluation plan
10
Alterations to the evaluation plan
10
Scaling evaluation
11
Responsibility for conducting evaluations
11
Evaluation approach
12
Lessons learned — taking advantage of completed evaluations
13
Building capacity and capability
14
Fostering a culture of evaluative thinking
14
Building evaluation capability
14
Supporting guidance material
15
Measures of success
16
Evaluation maturity
16
Reviewing the Evaluation Strategy
18
APPENDIX A – Glossary
APPENDIX B – Departmental guidance materials
19
Error! Bookmark not defined.
Introduction
This Evaluation Strategy provides a framework to guide the consistent, robust and transparent
evaluation and performance measurement of programmes and policies in the department.
Evaluations, reviews and performance monitoring provide assurance that policies and programmes
are delivering outcomes as intended, performance is tracked—allowing for correction to occur—and
inform future policy and programme design. As Australia is called to adapt to changing economic and
policy environments, the evidence gained from evaluations and other forms of performance
measurement and assessment support the decision-making of government.
For government, and this department, the continual questioning of how we are performing is a critical
part of good performance management and accountability. We need to know:

Have we achieved what we set out to do?

How are we progressing in achieving the department’s strategic objectives?

Could we have done things better?

Should we continue to do this or do something else?
Through asking these types of questions we gain an understanding of what works and what doesn’t
work and why, what is being done well and what is not, what should be pursued and what should not.
This knowledge can improve the design and implementation of effective interventions.
The Public Governance, Performance and Accountability Act 2013 (PGPA Act) establishes a core set
of obligations that apply to all Commonwealth entities. The Enhanced Commonwealth Performance
Framework brings an increase in external scrutiny, and introduces new requirements for strategic
planning, measuring and assessing performance, evaluation and reporting.
Reflecting the department’s response to the PGPA Act, the principles outlined in this Evaluation
Strategy (the Strategy) will strengthen evaluation and performance measurement capacity in the
department and support building a culture of evaluative thinking, ultimately leading to better resource
allocation and decision-making and the evolution of programmes.
This Strategy:

Outlines the department’s approach to performance measurement and reporting, according to
good evaluation practice

Establishes a protocol for policy and programme areas to plan for evaluation across the
lifecycle of a programme

Introduces a strategic, risk-based, whole-of-department approach to prioritising evaluation
effort, and illustrates how evaluations may be scaled based on the value, impact and risk
profile of a programme

Describes how evaluation findings can be used for better decision-making

Describes how the department is building evaluation capability and a culture of continuous
improvement

Outlines how the department will measure its progress in implementing this Strategy.
This Strategy is not intended to be a complete guide to evaluation and performance measurement. It
is supported by a range of internal and external resources including:
1

the department’s comprehensive guidance material and templates for planning and
conducting an evaluation

the department’s Performance Measurement and Reporting Framework

the Department of Finance Enhanced Commonwealth Performance Framework

the Australian National Audit Office Best Practice Guide—Successful Implementation of
Policy Initiatives.
Performance measurement and reporting
The department’s performance measurement and reporting framework supports the implementation
of the Enhanced Commonwealth Performance Framework (the Framework) under the PGPA Act. 1
The Framework enables Commonwealth entities to devise the necessary links between their
performance information and their external reporting. Entities are encouraged to adopt performance
measurement methodologies that better assess the results of activities and articulate their
performance story. The framework introduces a more transparent and cohesive form of performance
reporting related to the activities of an entity in achieving its purpose.
Under the PGPA Act Commonwealth entities will be required to produce and publish annually:
A Corporate Plan, which sets out the purpose of the entity and the method of
measuring and assessing the entity’s performance in achieving its purpose.
An Annual Performance Statement, which measures and assesses the entity’s
performance in achieving its purpose in the reporting period. This statement is
published as part of the Annual Report.
There is a strong emphasis on performance monitoring, evaluation and reporting. Further, the
Framework establishes a clear cycle of planning, measuring, evaluation and reporting of results to
Parliament, ministers and the public. The revised reporting framework allows flexibility in using
various data sources to better assess the results of government programmes. Performance measures
are tailored specifically for each programme so that they reflect its design, including its specified
objectives, inputs, outputs and outcomes.
Traditional Key Performance Indicators (KPIs) can be complemented with other measures which
provide a more meaningful link between public resources used and the results delivered. These may
include benchmarking, surveys, peer reviews, and comprehensive evaluations.
Given the department’s high level of complexity, a hierarchy has been adopted to measure and report
on performance. Performance measures are identified and reported on at the level of the ‘Activity’,
‘Strategic Objective’ and ‘Entity’, reflecting the outputs, impacts and outcomes of the department’s
activities. This approach means that performance monitoring and assessment is undertaken at the
appropriate levels, where useful performance measures can be developed and quality data is
available.
1
Information in this section is based on guidance available as at 5 June 2015. Department of Finance
(2015), Enhanced Commonwealth Performance Framework, Public Management Reform Agenda,
webpage viewed 5 June 2015, https://cfar.govspace.gov.au/legislation-pgpa-act/#transitional.
2
Figure 1: Reporting hierarchy
█
Vision (outcomes)
Enabling growth and productivity for globally
competitive industries
█
Strategic objectives/purposes (impact)
Supporting science and commercialisation
Growing business investment and improving capability
█
Activity (intermediate output)
Business research, development and commercialisation
Economic transition
█
Component (output)
Entrepreneurs’ Infrastructure Programme
Cooperative Research Centres
The department sets out its vision and four strategic objectives in the Strategic Plan 2015-19.
Impact on evaluation activity
Good performance information will draw on multiple sources that offer different perspectives on the
achievement of a programme’s objectives. The performance story of a programme is likely to be best
supported through a diverse set of measures.
Evaluations provide a balanced performance story through their incorporation of programme logic
tools, and assessment against set criteria. They provide meaningful information and evidence on a
component’s aim and purpose in terms of its effectiveness and efficiency and the activities that
focussed on that purpose. They provide an opportunity to look beyond performance monitoring and
reporting and consider how well the programme is achieving its outcomes.
The department responds to growing demand for evidence-based analyses of policy and programme
impacts by applying robust research and analytical methods, both quantitative and qualitative, to
determine and isolate what works in industry and science policies and programmes.
3
What is evaluation?
Evaluation is an essential part of policy development and programme management. The continual
questioning of what we are trying to achieve and how we are performing enables us to learn and
improve what we do, ensuring that decision-making is informed by the best available evidence.
Policy and programme evaluation involves collecting, analysing, interpreting and communicating
information about the performance of government policies and programmes, in order to inform
decision-making and support the evolution of programmes.
Evaluation helps to answer questions such as:

Is the policy contributing to the intended outcomes or any unintended outcomes?

Are there better ways of achieving these outcomes?

What has been the impact of the programme?

Is the policy still aligned with government priorities, particularly in light of changing
circumstances?

Should the current programme be expanded, contracted or discontinued?

Is there a case to establish new programmes?

Can resources be allocated more efficiently by modifying a programme or mix of
programmes?2
Evaluation is integral to continual improvement. It is a not a one-off, or ‘tick the box’ exercise.
Evaluation supports:
Evidence-based
Policy Development
Public
Accountability
Learning
Performance
Reporting
2

evidence-based policy development and decision-making

a stronger basis for informing government priorities

more efficient resource allocation.

the public accountability requirements of programme sponsors and
governments

the department’s risk-management processes, helping to encourage greater
public trust in government.

shared learning to improve policy development and programme design and
delivery

a culture of organisational learning within the department.

the analysis and assessment of balanced and meaningful performance
information to report on progress in achieving strategic outcomes

an enhanced ability to achieve government priorities.
Davis G & Bridgman P (2004), Australian Policy Handbook, Allen & Unwin, Sydney, pp.130-131.
4
Good evaluation practices – a consistent approach to evaluation
If evaluations are to be valuable to decision-makers across government, consistency in approach
and planning are required. Evaluations should be conducted to a standard that ensures the
information is credible and evidence-based.
The table below outlines the key principles used to guide evaluation in this department.3
Evaluation Principles:
1.
2.
3.
Integrated
Fit for purpose
Evidence-based

Evaluation is core business for the department and is not simply a
compliance activity

Evaluation planning is undertaken at the New Policy Proposal Stage
or early in the design of programmes

Evaluation results are communicated widely and inform decisionmaking and policy development.

The scale of effort and resources allocated to an evaluation are
proportional to the value, impact, strategic importance and risk profile
of a programme

The evaluation method is selected according to the programme
lifecycle, feasibility of the method, availability of data and value for
money.

The department applies robust research and analytical methods to
assess impact and outcomes

Collectors of administrative data strive to attain baseline
measurements and trend data in forms that are relatable to external
data sets.
4.
Specific and timely

Evaluation planning is guided by the timing of critical decisions to
ensure sufficient bodies of evidence are available when needed.
5.
Transparent

All evaluation reports are communicated internally unless there are
strong reasons to limit circulation

The department will move towards publishing more content
externally to strengthen public confidence and support public debate.

Evaluation governance bodies have some independence from the
responsible policy and programme areas

Typically, evaluators will have some independence from the
responsible programme and policy areas.
6.
3
5
Independent
Adapted from Department of the Environment (2015), Evaluation Policy, Canberra, p.7.
Governance
Executive Board
The Executive Board is responsible for oversight of the department’s evaluation activity, including:

Endorsing the department’s annual evaluation plan

Agreeing the evaluations to be listed and reported on in public documents, including the
Corporate Plan and Annual Performance Statement

Observing progress against the annual evaluation plan

Noting emerging themes from completed evaluations and audits to help inform future policy
and programme design

Observing information sharing between the Audit Committee and Evaluation Unit regarding
evaluation activity within the department, including the implementation of major
recommendations

Considering whether to publish evaluation reports or summaries to support greater
transparency of the department’s performance and dissemination of learning as noted in the
National Commission of Audit.
Steering Committees and Reference Groups
Evaluations conducted by the department are overseen by a Steering Committee or Reference
Group. A Steering Committee is more appropriate for larger or more complex evaluations, involving
multiple stakeholder groups or layers of government. Steering Committees increase transparency,
cross-fertilisation and independence. A Steering Committee gives direction to the team conducting the
evaluation and ensures the evaluation is of an acceptable quality and is independent. While it is
beneficial to have the knowledge of policy and programme area representatives, there should be a
level of independence in the membership of Steering Committees.
A Reference Group differs from a Steering Committee in that it is a sounding board for ideas, but
does not formally approve or direct the evaluation. The Chair of a Reference Group should be
independent from the policy and programme areas.
Evaluation and Audit—what is the difference?
The roles of evaluators and auditors are quite different but they have similar outputs—both inform
performance reporting and address public accountability requirements. There are strategic linkages
and synergies between the two functions. When planning an evaluation or audit, and identifying
lessons learned to inform policy development and programme delivery, evaluation and audit activity
should be considered and coordinated. The responsibilities of the Assurance and Audit Committee
and the role of Internal Audit and the Evaluation Unit are outlined below to help identify who does
what.
Assurance and Audit Committee
The Assurance and Audit Committee—established in accordance with the PGPA Act—provides
independent advice and assurance to the Executive on the appropriateness of the department’s
accountability and control framework, independently verifying and safeguarding the integrity of the
department’s financial and performance reporting.
The Annual Audit Plan provides an overview of the delivery of Internal Audit services, which include
General audits, ICT audits, Management Initiated Reviews and Assurance Advisory Services.
6
Internal Audit role
Internal audit provides an independent and objective assurance and advisory service to:

provide assurance to the Secretary that the department's financial and operational
controls designed to manage the organisation's risks and achieve the department's
objectives are operating in an efficient, effective and ethical manner

assist the Executive and senior managers in the effective discharge of their
responsibilities

improve the effectiveness of risk management, control and governance including
business performance

advise the Assurance and Audit Committee regarding the efficient, effective and
ethical operation of the department.
Evaluation Unit role
The Evaluation Unit are the authoritative source of advice on evaluation. The Unit’s role is to help
build capability in performance measurement and evaluation, and promote a culture of evaluative
thinking and continual improvement across the department. The Unit has responsibility for developing
and reporting progress against the department’s Evaluation Plan.
The Evaluation Unit are available to provide expert advice and guidance to divisions in planning and
conducting evaluations. They conduct evaluations on behalf of divisions as directed by the Executive
Board and can provide expertise as members of Steering Committees and Reference Groups.
The Evaluation Unit is responsible for…

Providing expert advice and guidance to line areas in planning and conducting
evaluations

Strengthening line areas’ capability to conduct evaluations

Developing new tools to support line areas and increase the rigour of evaluations
across the department

Conducting or participating in evaluations of selected high-priority evaluations

Providing advice as members of Steering Committees and Reference Groups

Promoting evaluation thinking across the department and sharing report findings

Developing the department’s four-year Evaluation Plan and reporting progress
against the Plan to the Executive Board.
The responsibilities of the Evaluation Unit are further described on pages 12 and 15.
7
Planning for evaluation across the policy and programme lifecycle
The decision to conduct an evaluation is strategic rather than simply routine. Decision-makers need
to give considerable thought to what they want an evaluation to address and when an evaluation
should occur.
Evaluation activity has different purposes at different points in the programme lifecycle. All policy and
programme areas need to consider evaluation requirements from the early policy and programme
design stage (ideally this occurs at the New Policy Proposal stage and in conjunction with the
Evaluation Unit). Planning for evaluation at this early stage helps to identify what questions need to be
answered and when, so that meaningful data can be collected to measure the programme’s outcomes
and impact.
Evaluation types
The department uses four key types of evaluation, grouping them under the categories of formative
(as the initiative is getting underway) and summative (once results can be observed). 4
Formative
Prospective evaluation—ensures that the programme is prepared for robust evaluations from the
outset using a programme logic model to refine the proposal and determine performance measures
and data sources.5
Post-commencement—also known as a post implementation review, these evaluations ‘check in’
soon after the programme has begun. This type of evaluation focuses on initial implementation,
allowing decision-makers to identify early issues regarding programme administration and delivery
and take corrective action.
Summative
Monitoring—these evaluations draw on performance information to monitor the programme’s
progress. They are usually suited to programmes which are at a “business as usual” patch in the
programme lifecycle, and the primary audience are people internal to the programme. The findings of
a monitoring evaluation provide an indication of performance, contributing to the measurement of the
department’s strategic objectives; a basis for future reviews; and an opportunity to test the
programme’s data sources to see whether they are providing the required performance information.
Impact—Impact evaluations are usually larger and more complex evaluations which allow for
assessment of a programme’s performance. Where possible they would test this against a
‘counterfactual’: they seek to compare programme outcomes with a prediction of what would have
happened in the absence of the programme, and may include research about programme alternatives
to allow comparison of results. They may involve a cost-effectiveness or cost-benefit analysis.
This type of evaluation tends to have more of an external focus—the primary audience are people
external to the programme. Impact evaluations usually aspire to find objectively verifiable results,
contributing to the measurement of the department’s strategic objectives and outcome.
4
5
Refer to glossary at Appendix A.
United States General Accounting Office (1990), Prospective Evaluation Methods, Washington D.C.
8
Figure 2: Evaluation stages in the programme and policy lifecycle
Evidence from
evaluations informs policy
and programme design
beyond the programme
being evaluated.
Policy development &
Prospective
evaluation
Programme Logic
Impact
Evaluations
Evaluation
Planning
Post-Commencement
Evaluation
Monitoring
Evaluations
Programme logic is used at the policy and programme development stage to clarify
the intent of a programme and assist in the development of performance measures,
key evaluation questions and data sources, including establishing baseline data.
One year into a programme a post-commencement evaluation can be conducted to
examine whether the programme is operating as intended.
At the 12 – 18 month mark the data collection and performance measures of a
programme can be tested through a monitoring evaluation, to see if the indicators
and data are presenting sufficient information to assess effectiveness and impact of
the programme. This would also give an assessment of performance against the short
term performance indicators.
Depending on the nature of the programme, after three to five years an impact
evaluation can be conducted using an appropriate methodology.
Evaluation readiness
To help policy and programme areas to plan for evaluation and performance monitoring, all
programme areas will complete a Performance Measurement and Reporting Template, or similar
documentation. The information will inform the policy or programme Evaluation Lifecycle. Completing
these documents and instilling a performance-measurement mindset will ensure the department’s
programmes are ‘evaluation ready’, i.e. key evaluation questions have been identified, and there is
enough evidence and data to enable an evaluation to be conducted.
9
Prioritising evaluation effort
A four-year evaluation plan
The department has a strategic, risk-based, whole-of-department approach to prioritising evaluation
effort to provide greater oversight in the allocation of resources to evaluation activity across the
department.
As required under the PGPA Act, the department’s Evaluation Plan covers a four-year period (over
the forward estimates). Elements of the Plan will be published externally by the department, including
in the Corporate Plan and Annual Performance Statement.
While noting the many benefits of evaluations, it is not feasible, cost-effective or appropriate to fully
evaluate every programme and policy.
The department prioritises evaluation effort and resourcing based on the criteria:

Total funding allocated for the programme

Internal priority (importance to the department’s and Australian Government’s goals)

External priority (importance to external stakeholders)

Overall risk rating of the programme

Track record (previous evaluation, strength of performance monitoring, lessons learnt).
The department’s Evaluation Plan (published internally) is developed in consultation with divisions
using the above criteria as a guide to how and when evaluations should be conducted. To reduce
duplication and leverage effort, the department takes account of audit and research activity when
developing its Evaluation Plan, adopting a whole-of-department approach to evaluation planning.
The Evaluation Plan uses tiers to identify evaluations of highest priority and strategic importance. See
table on page 12.

Tier one—evaluations of highest priority and strategic importance

Tier two—second order of strategic importance and may include methodologies to test the
performance monitoring of the programme.

Tiers three A & three B—programmes of lesser strategic importance, low risk, or a terminated
programme. Tier three A includes single payment grants.
Alterations to the evaluation plan
The Evaluation Plan is not cast in stone. The department recognises that circumstances change as
months and years pass. It is important that ongoing intelligence about programmes’ evolution, and
adjustments in Government policy-setting, inform the Evaluation Plan. Generally, the Plan is updated
annually but alterations can be made at any time.
10
Scaling evaluation
The scale of an evaluation should be proportionate to the size, significance and risk profile of the
programme (sometimes referred to as ‘fit for purpose’). This means that evaluative effort and
resources should not be expended beyond what is required to satisfy public accountability and the
needs of decision-makers.
The selection of an evaluation method should also take into account the programme lifecycle and
feasibility of the method, the availability of data and value for money. Evaluations should be
appropriate to the particulars of a given programme; they are not a ‘one size fits all’ arrangement.
Options could include light touch/desk reviews, grouping a suite of programmes into one evaluation,
sectoral reviews and research projects that focus on the effectiveness and impact elements of
programme performance.
The increased accountability and scrutiny under the PGPA Act further reinforces the critical role of the
Evaluation Unit as the authoritative source for guidance on evaluation. To ensure evaluative effort and
resources are used to maximum benefit, it is a requirement that divisions consult with the Evaluation
Unit when determining an appropriate scope and methodology for their evaluations.
Responsibility for conducting evaluations
Priority, scale and methodology will inform who will conduct an evaluation. Subject-matter or technical
expertise should also be considered, as should resource availability, time and cost. Options include:
an individual or small team; working collaboratively across divisions or other agencies; engaging the
Evaluation Unit to conduct the evaluation; working in partnership with the Evaluation Unit; or engaging
an external consultant or academic.
The policy owner is accountable for managing the evaluation with assistance from the programme
delivery team.
There should be a level of independence from the areas responsible for policy and programme
delivery. For evaluations of lesser strategic importance or terminated programmes this could be
through an independent member on a Steering Committee or Reference Group. Seconding
individuals from divisions separate from the policy and delivery areas is a viable option to provide
some independence, build capability and alleviate resourcing constraints.
11
Evaluation approach
The table below outlines the fundamental issues which should be considered in determining the scale
of an evaluation. There may also be times where Cabinet or central agencies determine the type of
evaluation and when it should be conducted. The Evaluation Unit can provide advice and guidance on
planning and conducting evaluations across all of the tiers.
Characteristics
of Programme
Likely
characteristics
of Evaluation
Tier One
Tier Two
Tiers Three A & Three B

Significant funding

Moderate funding


Highest risk

Medium risk
Relatively small
funding or single
payment grants

Strategically
significant


Low risk

May be Flagship
programme
New or untried
programme that
requires testing of
assumptions and or
data

Lesser strategic
importance

High public profile and
expectations

Not widely publicised

Similar to other
programmes that have
been subject to
evaluation activity

Informal process

Can be completed
internally

Limited data
requirements

Low resource
allocation

Limited consultation

Low profile release

Politically significant

Formal process

Extensive consultation



Evaluation Unit
role
High resource
allocation
Central agencies may
be involved
Wide public release

Medium level of
strategic importance

Moderate public
profile and
expectations

Greater level of data
collection and analysis

Multiple evaluation
points during the
development and
implementation

Regular process
reporting
Evaluation Unit should be
consulted on
development of
methodology/terms of
reference.
Evaluation Unit should be
consulted on
development of
methodology/terms of
reference.
Evaluation Unit should be
consulted on
development of
methodology/terms of
reference.
Evaluation Unit
involvement either
conducting evaluation or
represented on Steering
Committee.
Could be conducted by or
in partnership with the
Evaluation Unit or by
independent
person/team.
Reference Group should
have a level of
independence.
Independent evaluator
could be internal or
external to the
department.
Evaluator could be
Internal or external to the
department.
12
Lessons learned — taking advantage of completed evaluations
Policy making is a process of continuous learning, rather than a series of one-off, unrelated
decisions. Effective use of organisational knowledge in policy development enables policy makers to
learn from previous successes and failures to develop better policy. Policy and programme
evaluations provide the evidence base to inform best practice expenditure of public funding and the
development of policy. 6
Evaluations increase understanding about the impact of government policy, programmes, regulations
and processes, and are just one of the sources of performance information that help the department
to assess whether it is achieving its strategic priorities. Along with research and audit findings, the
outcomes from evaluations are a valuable resource; they support evidence-based policy and the
continual improvement and evolution of programmes.
Organisational learning uses past experiences to improve policy, recognising that the government
may repeatedly deal with similar problems. Developing a culture of organisational learning can make
an organisation more responsive to the changes in its environment and facilitate adaptation to these
changes. 7
It is expected that evaluation findings will be communicated widely across the department, particularly
to inform decision-making, with resulting recommendations acted upon routinely. It is also expected
that evaluation findings and emerging trends are captured, reported and communicated, and brought
to the attention of the Executive Board as appropriate.
To improve the sharing of evaluation findings and make them accessible across the department, all
evaluations commissioned or undertaken by the department will be accessible internally through the
Research Management System. The System provides significant insight to the approaches used to
design policy and implement departmental programmes.
6
7
Department of Industry and Science (2014), Policy Development Toolkit, Canberra.
Ibid.
13
Building capacity and capability
Building capacity and capability in performance measurement and evaluation is not limited to
technical skills and knowledge. Performance measurement and evaluation needs to be integrated
into the way we work and think.
Fostering a culture of evaluative thinking
As we are called to adapt to changing economic and policy environments, measuring how we are
performing and providing credible evidence becomes paramount. This cannot be achieved without a
shift to a culture of evaluative thinking and continuous improvement.
Organisational culture significantly influences the success of evaluation activity and requires strong
leadership. This department is building a supportive culture, led from the Executive, that encourages
self-reflection, values results and innovation, looks for better ways of doing things, shares knowledge
and learns from mistakes.
Without such a culture, evaluation is likely to be resisted, perceived as a threat rather than an
opportunity and treated as a compliance exercise.
To develop a culture of evaluative thinking the department requires:

a clear vision for evaluation and continuous improvement

clear responsibilities and expectations to empower staff, along with appropriate training and
guidance material

knowledge-sharing and tolerance for mistakes to encourage learning and improve
performance

a culture of reward to showcase effective evaluations

support for the outcomes of robust evaluation to build trust, welcoming the identification of
problems or weaknesses.8
Building evaluation capability
A culture of evaluative thinking and capability building go hand in hand—both are required to achieve
a high level of evaluation maturity within a high performing organisation.
Conducting an evaluation requires significant knowledge, skill and experience. The department is
committed to building performance measurement and evaluation capability and technical skills to
support staff in planning and conducting evaluations, and undertaking performance monitoring.
Learning continues for staff in the Evaluation Unit and across the department in specialised evaluation
techniques and methods. The Evaluation Unit is made up of evaluation professionals who are
members of the Australasian Evaluation Society (AES) and other professional organisations.
The role and responsibilities of the Evaluation Unit include building capability through providing expert
advice and guidance, and ensuring the department is meeting its external reporting accountabilities.
8
ACT Government (2010), Evaluation Policy and Guidelines, Canberra.
14
In addition to the points noted on page 7:
The Evaluation Unit can advise on the…

Development of programme and policy logics

Conduct of programme and policy evaluations
(prospective, post-commencement, monitoring and impact evaluations)

Preparation of:

evaluation lifecycles
(to guide evaluations throughout a programme’s lifecycle)

evaluation plans
(to guide a single evaluation)

Development of programme level performance measures (KPIs)

Application of departmental evaluation principles.
Supporting guidance material
The Evaluation Unit have developed comprehensive guidance material to support on-the-job learning.
The topics covered range from planning for an evaluation to how to conduct an evaluation or develop
Terms of Reference. The material is designed to be used in conjunction with advice available from the
Evaluation Unit.
The department also offers targeted learning on programme logic and developing performance
measures.
15
Measures of success
Evaluation maturity
Developing and maintaining evaluation maturity is an ongoing process that must be balanced with
other organisational objectives. This Strategy establishes a framework to guide the department
through the stages of maturity which encompass good evaluation practices.
To establish a baseline from which we can identify strengths and weaknesses and priorities for
improvement, the department has assessed its current evaluation maturity. While it is following best
practice in some elements of evaluation maturity, overall it is at the developing stage of maturity.
Maturity level 
Integrated
Beginning
Developing
(Current rating)
Embedded
(June 2016)
Leading
(June 2018)

Awareness of the
benefits of
evaluation is low.




Evaluation is seen
as a compliance
activity and threat.
Appreciation of the
benefits of
evaluation
improving.

Evaluation is being
viewed as core
business for the
department, not
simply a compliance
activity.
Evaluations
motivate
improvements in
programme design
and policy
implementation.

Demonstrated
commitment to
continuous learning
and improvement
throughout the
agency.

Department is
recognised for its
evaluation and
performance
monitoring
expertise, and
innovative systems
and procedures.



Fear of negative
findings and
recommendations
leads to a
perception of
‘mandatory
optimism’ regarding
programme
performance.
Insufficient
resources allocated
to evaluation
activities.


Evaluation and
performance
measurement skills
and understanding
limited, despite
pockets of
expertise.

A culture of
evaluative thinking
and continual
improvement is
introduced and
communicated
across the
department.
Skills in
performance
measurement and
evaluation
developed through
targeted training
and guidance
materials.
Evaluation website
and guidance
materials
developed.

The role of the
Evaluation Unit is
widely
communicated. Unit
seen as the
authoritative source
for advice.

Developing further
expertise in the
Evaluation Unit.
A culture of
evaluative thinking
and continual
improvement is
embedded across
the department, with
lessons learnt being
acted upon.

Evaluation is seen
as an integral
component of sound
performance
management.

General evaluation
skills widespread.

Improved skills and
knowledge in
developing quality
performance
measures.

Evaluation Unit
team members
have high order
skills and
experience which is
leveraged by the
agency.

Evaluation Unit
team members hold
and are encouraged
to undertake formal
qualifications in
evaluation and
related subjects.
16
Fit for
Purpose
Evidencebased
Specific and
timely


Frequency and
quality of evaluation
is lacking.

Guidelines for
prioritising and
scaling evaluation
activity are used.

Priority programmes
are evaluated.

Evaluation effort is
scaled accordingly.

Evaluations use fit
for purpose
methodologies.

Specialist and
technical skills well
developed to apply
appropriate
methodologies.
Data holdings and
collection methods
are insufficient or of
poor quality.

Planning at
programme outset
improves data
holdings and
collection methods.

A range of
administrative and
other data is used in
the assessment of
performance.


Developing skills
and knowledge in
applying robust
research and
analytical methods
to assess impact
and outcomes.

Robust research
and analytical
methods are used
to assess impact
and outcomes.
The department
continually develops
and applies robust
research and
analytical methods
to assess impact
and outcomes.

Evaluation and
performance
measurement
conforms to
recognised
standards of quality.

The department’s
approach to
evaluation and
performance
planning is seen as
the exemplar.

All programmes
assessed as
‘evaluation ready’.

Quality of
evaluations is
improving.

Effort and resources
are allocated in an
ad hoc and reactive
manner with little
foresight.

Evaluation activity is
coordinated. An
Evaluation plan is in
place and regularly
monitored.

Developing
performance
information at the
inception of a
programme is ad
hoc and of variable
quality.

Strategically
significant and risky
programmes are
prioritised.


17
Planning for
evaluation and
performance
monitoring is being
integrated at the
programme design
stage.

Evaluations conform
to agency
standards.

The department
employs strategic
risk-based, wholeof-department
criteria to prioritise
evaluation effort.
Evaluation Plans
are updated
annually and
progress is
monitored on a
regular basis.

Planning for
evaluation and
performance
measurement is
considered a
fundamental part of
policy and
programme design.

All programmes
have programme
logic, performance
and evaluation
plans in place.
All programmes are
assessed for being
‘evaluation ready’.
Transparent
Independent
9

Findings and
recommendations
held in programme
and policy areas.

No follow up on the
implementation of
recommendations.

Independent
conduct and
governance of
evaluations is
lacking.

Evaluations are
conducted and
overseen by the
policy or
programme areas
responsible for
delivery of the
programme.



Findings and
recommendations
viewed as an
opportunity to
identify lessons
learned.
Evaluation findings
and
recommendations
available in the
research
management
system to improve
the dissemination of
lessons learned and
inform policy
development.
There is an
improved level of
independence in the
conduct and
governance of
evaluations.

Findings widely
disseminated and
drive better
performance.

Website and
guidance materials
valuable resource
for staff.

Evaluation findings
and reports are
published where
appropriate.

All evaluations
include a level of
independence.

Findings are
consistently used to
optimise delivery
and have influence
outside the
department.
Reviewing the Evaluation Strategy
This Strategy will be reviewed in 12 months (June 2016) to assess whether it is meeting the needs of
the department. The measures of success will include that it is:





Consistent with the PGPA Act
Efficiently allocating evaluation effort
Leading to more effective conduct of evaluations
Fostering a culture of evaluative thinking
Ultimately contributing to more effective programmes.
Results of the review will be communicated to the Executive Board. The review will include an
assessment of the department’s level of evaluation maturity, one year on.
The evaluation maturity table is adapted from:
ACT Government (2010), Evaluation Policy and Guidelines, Canberra, p.17.
18
APPENDIX A – Glossary
Term
Definition
Activity
A distinct effort of an entity undertaken to achieve specific results or aim.
An entity’s purpose may be achieved through a single activity or multiple
activities.
Baseline
Information collected before or at the start of a programme that provides
a basis for planning and/or assessing subsequent programme progress
and/or impact.
Component
A programme or initiative within the department.
Cost Benefit
Analysis/Cost
Effectiveness Analysis
Evaluation of the relationship between programme costs and outcomes.
Can be used to compare different programmes with the same outcome
to determine the most efficient intervention.
Counterfactual
A hypothetical statement of what would have happened (or not) had the
programme not been implemented.
Evaluation
A systematic assessment of the operation and/or the outcomes of a
programme or policy, compared to a set of explicit or implicit standards,
as a means of contributing to the improvement of the policy or
programme.
Expenditure Review
Principles
The principles used to review Australian Government activities and in
the assessment of proposals for new government activities. They
include: appropriateness, effectiveness, efficiency, integration,
performance assessment and strategic policy alignment.
Formative evaluation
Generally undertaken while a programme is forming (prior to
implementation), this type of evaluation is typically used to identify
aspects of a programme that can be improved to achieve better results.
Impact evaluation
Impact evaluations are usually larger and more complex evaluations
which allow for assessment of a programme’s performance. Where
possible they would test this against a ‘counterfactual’.
Key Performance
Indicator
Quantitative or qualitative variables that provide a reliable way to
measure intended changes. Performance indicators are used to observe
progress and to compare actual results with expected results.
Monitoring evaluation
Monitoring evaluation draws on performance information to monitor the
programme’s progress. Usually suited to programmes which have hit a
“business as usual” patch in the programme lifecycle, the findings
provide: an indication of performance, contributing to the measurement
of the department’s strategic objectives; a basis for future reviews; and
the opportunity to test the programme’s data sources to see whether
they are providing the performance information required.
Outcome
A result or effect that is caused by or attributable to the programme.
Output
The products, goods and services which are produced by the
programme.
19
Programme logic
A management tool that presents the logic of a programme in a diagram
(with related descriptions). It links longer-term objectives to a
programme’s intermediate and shorter-term objectives. The programme
logic is used to ensure the overall programme considers all the inputs,
activities and processes needed to achieve the intended programme
outcomes.
Post-commencement
evaluation
Post-commencement evaluations focus on initial implementation,
allowing decision-makers to identify early issues regarding programme
administration and delivery and take corrective action.
Prospective evaluation
Prospective evaluations ensure the programme is prepared for robust
evaluations from the outset, using a programme logic model to refine the
proposal and determine performance measures and data sources.
Purpose
Summative evaluation
In the context of the PGPA Act, the strategic objectives, functions or
roles of an entity, against which entities undertake activities.
Summative evaluations generally report when the programme has been
running long enough to produce results, although they should be
initiated during the programme design phase. They assess positive and
negative results, as well as intended and unintended outcomes. This
form of evaluation is used to determine whether the programme caused
demonstrable effects on specifically defined target outcomes.
20