Third Innovation Academics’ Workshop – 23 May 2013
‘Evaluation and Innovation’
Program
8.15-8.45
Registration, tea & coffee
8.45-9.00
Opening

9.00-10.40
Welcome (Mark Cully, Chief Economist, DIICCSRTE)
Session #1 of Short presentations (15 mins each), followed by round table discussion
Evaluation and Innovation – complexity of issue
 Categories of Innovation and Impact Assessment (Prof Shirley Gregor, ANU, and Prof Al
Hevner, University of South Florida)
 The Fundamentals of Evidence-Based Innovation Policy (Prof Paul Jensen, UniMelb)
 Outsourcing and Innovation: An empirical exploration of the dynamic relationship (Prof
Robert Breunig, ANU, and Dr Sasan Bakhtiari, UNSW)
 Outcome-oriented science policy: Improving prioritisation and evaluation (Dr Paul Harris,
ANU)
10.40-11.00
Morning tea
11.00-12.40
Session #2 of Short presentations (15 mins each), followed by round table discussion
Measurement and evaluation – impact of innovation
 Innovation: Impact Assessment Frameworks (Dr John Howard, UC)
 Sense-T – Evaluation of the economic impact (Dr Helmut Fryges, UTAS)
 Broadband Investment: Private Provision of Public Good and Information Asymmetry (Dr
Philip Thomas, UNE, and Dr Erkan Yalcin, Flinders University, and Prof Theodore Alter
and Dr Michael Fortunato, Pennsylvania State Uni, USA)
 VC/PE (Funds), Government Grants and Innovation in Newly Public Firms (Dr Jo-Ann
Suchard and Dr Mark Humphery-Jenner, UNSW)
12.40-13.10
Lunch
13.10-14.55
Session #3 of Short presentations (15 mins each), followed by round table discussions
Measurement and evaluation – impact of innovation (ctd.)
 The use of alternative evaluation methods for public sector innovations (Prof Anthony
Arundel, UTAS)
 A pilot study on the development of patent metrics (Vera Lipton, IP Australia)
 Using Leximancer for Innovation Evaluation Settings (Prof Kerry Brown, Prof Robyn
Keast and Dr Nateque Mahmood, SCU)
 The impact of small business advisory services on small business innovation (Dr
Sukanlaya Sawang and Prof Rachel Parker, QUT)
14.55-15.15
Afternoon tea
15.15-16.45
Group Discussion: Sharing / reporting on the outcome of group discussions
How to evaluate the impact of innovation? (Facilitated by Prof Brian Head, UQ)
16.45-17.00
Close: The way forward
1
Overview
The Department of Industry, Innovation, Climate Change, Science, Research and Tertiary
Education hosted the third Innovation Academics’ Workshop on 23 May 2013 held in
partnership with the HC Coombs Policy Forum at the Crawford School (ANU, Canberra).
The Workshop, ‘Evaluation and Innovation’, focused on developing a better understanding of
the evaluation of innovation – how its impact can be better measured, what indicators we
could move forward with, and how this could be used to influence new policy. It was also
hoped that the Workshop would facilitate dialogue between academics and public servants to
further promote mutual understanding and build ground for future collaboration.
The Workshop included representation from universities, publicly funded research
organisations, state government, the Institute of Public Administration of Australia, and the
Department.
The Workshop featured presentations from academics and government sharing their
experiences and knowledge of the issues surrounding evaluation of innovation programs and
instruments for measuring the impact of innovation. Each session comprised four short
presentations followed by discussion, and the last session focused on brainstorming around
the Enterprise Solutions Program (ESP) which is currently being designed by the
Department.
Presentations
Opening - Mark Cully (Department of Industry and Innovation)
The Department’s Chief Economist Mark Cully opened the Workshop with a talk on the
importance of the economic evidence-base for government support of innovation. The link
between economic growth and innovation is especially important now that living standards in
the United States and parts of Europe have fallen back to the levels of a decade ago.
In 2012-13 the Australian government invested an estimated $8.9 billion in science, research
and innovation, including an estimated $1.8 billion in company tax revenue foregone from
the R&D Tax Incentive, with further government contribution to such aspects of the
innovation system as, for example, the underpinning regime of intellectual property rights
administered by IP Australia.
Economists and policy analysts are working on evaluation of the effectiveness of our
investment in science, research and innovation. There is solid evidence that investment in this
intangible capital yields considerable returns for individual businesses and some evidence of
positive spill overs to other businesses. However policy makers need to be convinced that the
level of additionality and/or spill overs is such that the social benefits exceed the costs. One
way to proceed is to accumulate an evidence base built on carefully designed studies that
allow treatment effects to be isolated and measured.
To promote collaboration between researchers and government, the Department can offer the
following four things: encouragement, insight, data, and engagement. We could put more
money on the table, we could be more transparent, we could lessen restrictions on data
access, and we could listen more.
2
Categories of Innovation and Impact Assessment - Shirley Gregor (ANU)
Professor Shirley Gregor presented a new typology (known as the knowledge-innovation
typology) for categorizing innovations, and the types of value that can be achieved. Studies
undertaken in the field of Information Technology found that two thirds of top innovations
arise from industry - research collaboration. The work found that the definition of innovation
is pivotal, and that innovation occurs frequently when one adopts a new IT system.
The knowledge-innovation typology can be applied to collaborative ventures between
industry and research organizations. The typology results from a classification of innovations
and knowledge contributions on two dimensions:
(i)
(ii)
application domain maturity; and
solution (knowledge) maturity.
The resulting matrix has four quadrants:
(1) Invention: radical innovation/exploration (low solution maturity, low application
maturity);
(2) Exaptation: exapted exploration (high solution maturity, low application maturity);
(3) Improvement: incremental exploration (low solution maturity, high application maturity);
and
(4) Adoption: routine design, exploitation, (high solution maturity, high application maturity).
For example, the original invention of bicycles was followed by incremental improvements
and use of different types of bicycles in different setting (exaptation) and widespread
adoption.
The typology offers a means for researchers and industry to categorize innovations and
understand the range of outcomes and value to be expected with each type, noting that
movement would occur between the quadrants.
The Fundamentals of Evidence-Based Innovation Policy - Paul Jensen (University of
Melbourne)
Professor Paul Jensen is currently evaluating a series of programs from one state. He has
identified that innovation policy research in Australia lags behind other areas of policy
(including health, health economics and trade).
Professor Jensen addressed the issue of evaluating the impact of government support for
innovation by offering the following four solutions:
(1) Creating systematic data infrastructure;
(2) Promoting access to the data infrastructure;
(3) Building capacity in universities and government; and
3
(4) Systematically evaluating innovation programs.
The 2010 ‘Strengthening Evidence-based Policy’ conference organised by the Productivity
Commission provides some good background for evaluation.
For innovation program evaluation, the social science field offers good examples of methods,
each having advantages as well as trade-offs. Randomized controlled trials are acknowledged
as the “gold standard”, but they are also the most costly method. Other methods which could
be used to evaluate innovation programs include: “difference-in-differences” (e.g. to compare
two states – before and after one implements a new policy or program) is thought to be
simple to administer and powerful; “regression discontinuity” can only be used in certain
contexts; “quasi-natural experiments” are a possibility, although these are rare in Australia;
and, case studies which are claimed to have only limited use.
Professor Jensen’s research is using the ABS Business Longitudinal Database, and the
Confidential Unit Record File, along with firm’s ABN, to see the effects of innovation on
individual firm’s performance data such as employment and exports.
Outsourcing and Innovation: An Empirical Exploration of the Dynamic Relationship Robert (Bob) Breunig (ANU)
Professor Robert Breunig and Dr Sasan Bakhtiari have studied the implications of vertical
integration on innovation performance using firm-level data on Australian manufacturing.
They have used the data to distinguish between low-cost-oriented and innovation-oriented
outsourcing. It has been found that outsourcing without innovation lowers costs at the
expense of damaging the future chances of innovation, while innovation-oriented outsourcing
leads to higher costs but increases the likelihood of future innovation. For firms that innovate
and outsource, the probability of future innovation is 49 per cent compared to 8 per cent for
those who outsource without innovating.
Comparing across firms that innovate, simultaneously outsourcing increases the probability
of future innovation by 5 per cent. Innovation-oriented outsourcing is accompanied by firms
shifting expenditure to research and development. The results offer strong support that
outsourcing may be used not just as a cost-cutting strategy, but as part of comprehensive firm
strategy to innovate and improve.
Outcome-oriented science policy: Improving prioritisation and evaluation - Paul Harris
(ANU)
Paul Harris proposed a focus on outcome-oriented, rather than evidence-based, science
policy. The effects of government investment in science include not only an increase in
knowledge, but also contribute to economic productivity through innovation. It is also
expected that a range of social benefits will result from investment in science, including
enhanced health and wellbeing, and improved sustainability and security. However the ways
in which we measure the effectiveness and appropriateness of current investments rely on a
small set of scientific and economic metrics – we have little evidence of the connection to
broader societal outcomes. Paul Harris questioned the supremacy of R&D investment (and
other simplistic input/output metrics) as a measure of the performance of science policy.
4
The Department of Finance and Deregulation 2012 report ‘Sharpening the Focus’ highlights
that most indicators for public sector performance are financial. However, non-financial
indicators are needed to more accurately assess the outcomes of public expenditure. Paul
Harris noted that in total, currently there are over 3,500 different performance metrics which
public sector agencies are required to report against; and argued that instead of adding more
metrics, we need to be asking the right questions.
The government has established the Australian Research Committee (or ARCom) and
National Research Investment Plan to improve coordination – this provides a strong platform
for further work on a consistent outcomes framework for public investment in science. The
National Environmental Research Program was provided as a good practice example, as it
elicits high quality research to enable policy makers to make better decisions, and also
requires researchers and policy-makers to work together to articulate the expected outcomes
prior to being granted funds.
Science policy should be re-oriented around outcomes, with prioritisation and evaluation
following on from there. This will be challenging, as an outcome-oriented science policy
framework needs to allow for a diversity of investments, institutions and outcomes, but this is
precisely why one size, or one set of metrics, will not fit all.
Measurement and Evaluation of Impact of Innovation – Impact Assessment Frameworks John Howard (Howard Partners)
Dr John Howard spoke about three broad frameworks for evaluation of innovation:
(1) Economic frameworks which measure the economic impact of research in terms of
change in national economic output and productivity, as well as growth in industry output
(production) or firm level output (sales);
(2) Knowledge transfer approaches combined with case study approaches to indicate change
– as reflected, for example, in the work of the Go8 and the Australian Technology Network
of Universities in the Excellence in Innovation Trial. These approaches focus more on the
industry and enterprise level; and
(3) Program logic frameworks which are used widely in program design, evaluation and
performance monitoring; they have a strong process orientation.
All three types of frameworks are useful, although they all have their limitations.
No matter which model is followed, it is important to collect evidence, and so a range of
approaches to collecting evidence were presented, including economic modelling, surveys,
consultations and focus groups, stories and narratives, as well as expert judgment / peer
review. No single method is best; the choice would depend on the researchers’ resources,
time available, and priorities. Different methods would be used in evaluation of systems as
distinct from programs.
Sense-T – Evaluation of the Economic Impact - Helmut Fryges (University of Tasmania)
Dr Helmut Fryges reported on Sense-T, the world’s first economy-wide intelligent sensor
network, initially trialled in Tasmanian farming. Sense-T is a partnership between the
5
University of Tasmania, CSIRO, the Tasmanian Government and IBM. The first stage of the
project has been funded by the Australian Government, Department of Regional Australia,
Local Government, Arts and Sport.
Sense-T offers a combination of different sensors in a single, large-scale integrated
information repository. It combines historical data, existing sensory data (for example
weather data) and novel sensory data in a sensor cloud. The collected data are aggregated and
interpreted, with hypotheses formulated and simulation models developed in order to detect
relevant information from the sensor cloud. The information will then be provided to the
community via web applications. Thus far four projects using Sense-T have been
implemented, one of which is a dairy and beef project which can identify whether cows are
sick, and it is hoped that the project will result in a product innovation in designer milk. Webbased decision support tools for the four initial projects will be provided until the end of
2013. The evaluation of the Sense-T projects will focus on three aspects: 1) whether the
sensors work; 2) whether decision support tools provide useful information; and 3) whether
projects have an economic, social and environmental impact. Various evaluation techniques
are currently considered to evaluate Sense-T’s economic impact. These include quantitative
methods (randomised control trials, difference-in-difference estimations) as well as
qualitative methods (case studies, focus groups, outcome mapping, etc.). Research will be
conducted in order to identify sustainable business models.
Broadband Investment: Private Provision of Public Good and Information Asymmetry Philip Thomas and Erkan Yalcin (Flinders University) with colleagues (Prof Theodore
Alter and Dr Michael Fortunato, Pennsylvania State Uni, USA)
Dr Philip Thomas and Dr Erkan Yalcin used an example of broadband investment in
Australia (which has a public-private investment structure in the implementation of the
NBN), compared to the USA (which has a private investment structure for broadband), to
present an example of asymmetry of information in public-private contractual negotiation,
and the implications of public-funded investment in innovation.
The mechanism design can be explained as follows:
The contractor has the knowledge to make an accurate assessment of the actual cost of the
broadband investment. On the other hand, the government can only estimate what the cost
will be.
In order to deal with this situation, the government can design a procurement mechanism for
the infrastructure project to minimize information asymmetry. Two common methods for
writing such contracts are:


Cost-plus; and
Fixed-price.
In practice the optimal procurement mechanism is a balance these two methods.
The issue is that the strategic opportunities under asymmetric information typically lead to
inefficient market outcomes. It is not possible to eliminate Informational Asymmetry
completely, but it is possible to reduce it. This research aims to determine the impact of
6
Informational Asymmetry on the delivery of social benefit. The aim is to develop a method to
assess the effect of Informational Asymmetry on infrastructure investments, creating a
decision making tool to guide future investment in critical infrastructure.
VC/PE Funds, Government Grants and Innovation in Newly Public Firms - Jo-Ann Suchard
(University of New South Wales)
Dr Jo-Ann Suchard reported on the analysis of the impact of Venture Capital/Private Equity
(VC/PE) backing and grants on firm level innovation. The levels of innovation inputs (R&D
expenditure) and outputs (patents and citations) were analysed for a sample of 436 VC/PE
backed and non-VC/PE backed firms at the time they listed on the Australian Stock
Exchange.
It was also found that VC funds are better at increasing innovation inputs (R&D expenditure),
whereas PE funds are better at increasing innovation outputs, consistent with the idea that
they are helpful in commercializing latent technologies. Grants help to encourage innovation
inputs and outputs.
Grants, VC, and PE backing are complementary, with the combination increasing innovation
more than either one alone. Government grants can encourage VC investment as VC
investors see grants as an important indicator of the quality of the firm, and its potential,
which highlights the role of competitive grants as a signal to investors. Regarding fund-level
characteristics, it was found that large funds increase innovation outputs, and foreign funds
are less able to increase Australian innovation inputs or outputs. The results can assist policymakers in determining the structure of future VC/PE funding schemes in order to select the
types of funds that are best suited to increase innovation.
Alternative methods for improving public sector innovation outcomes - Anthony Arundel
(University of Tasmania)
Professor Anthony Arundel discussed the issue of the use of alternative evaluation methods for
public sector innovation. Whereas the gold standard for evaluating government programs is to
conduct an independent evaluation, using quantitative methods that can adjust for self-selection
and other biases, many public sector innovations are incremental or minor and may not warrant
the cost of a full independent evaluation.
Public sector managers could improve the quality of their innovations through alternative low-cost
methods, such as a formal process for evaluating ideas put forward by staff, client satisfaction
surveys, involving users in the design of an innovation, and trial and error testing of new ideas.
The effect of alternative evaluation methods has been examined using data from two large
surveys: the 2011 Australian Public Service’s State of the Service survey, and the 2010
Innobarometer survey in Europe. The Innobarometer study evaluates the innovation after its
completion, so it strongly correlates with successful innovation. The results of data analysis
from both surveys suggest that self-reported evaluation of ideas, and evaluation after
completion of innovations are positively correlated with successful outcomes. However,
given the cross-sectional nature of the data, it is not possible to assume a cause and effect
relationship between self-reporting evaluation and successful innovation. Although the
analyses controlled for many possible confounding variables, the existence of evaluation
7
systems and pilot testing could be linked to other hidden factors that influence the outcome,
such as a correlation between a positive attitude to innovation and a tendency to see
favourable outcomes.
Assessing research performance through patents - Vera Lipton (IP Australia)
Vera Lipton reported on the findings of an analytics study from IP Australia to develop patent
metrics to assess university research. The study established the scope and impact of patenting
activity originating from 12 Australian universities, including larger, smaller and regional
universities. The study found that the rate of patenting does not correlate with the impact of
university inventions on follow-on innovation. The impact was measured through various
patent citation analyses.
Some other key findings are:

Smaller and regional universities tend to concentrate their
patent filings in a few (4 – 5) technology fields, while the patents
filed by larger universities generally spread across several
technology fields (over 15).

Patents originating from research of two smaller universities
have received considerably more citations than the Australian
average. These patents relate to medical technology, electrical
machinery, civil engineering and pharmaceuticals.
Using Leximancer for Innovation Evaluation Settings - Kerry Brown (Southern Cross
University)
The study conducted by Professors Kerry Brown and Robyn Keast and Dr Nateque Mahmood
(SCU) has demonstrated the potential of context analysis software Leximancer for analysing
innovation policy documents. The novel use of the software involved comparative analysis of
reports on innovation, their overall discussions, and goals for public sector innovation, as
constructed in those texts.
Leximancer was used for data mining and visualisation mapping to identify what concepts
innovation documents and data sets identified as important, and what concepts may be
missing, for example in the relationship between education and innovation policy.
Leximancer operates on both thematic (conceptual) and relational (semantic) systems of text
analysis to determine the strength of association and semantic similarity between concepts.
Leximancer clusters together concepts occurring in similar semantic contexts.
This is a visualisation technique that has the potential for highlighting the important concepts
in the data set and the relationships between these concepts in a relatively easy to understand
way, and for analysis of new documents government has to respond to, as well as for
construction of new public documents effective in targeting their intended audiences.
The impact of small business advisory services on small business innovation - Sukanlaya
Sawang (QUT)
Dr Sukanlaya Sawang reported on a study of the impact of small business advisory services
on small manufacturing firms’ innovation. The research used surveys, matching their samples
8
with ABS data, which included all SMEs in the Australian economy, except for business
units in non-employing businesses and government enterprises. In the ABS sample, there
were 1,690 firms that did not receive business advice in the form of government assistance.
It was found that firms participating in business advisory programs innovated much more
than non-participants, indicating that small business advisory programs are an effective
mechanism for governments to stimulate firm innovation behaviours.
It was also found that particular types of program content are useful for firms seeking to
introduce new products/services or organisational processes. Different types of program
content were important for firms whose aim was to adopt new marketing techniques or to
access new markets. This suggests that there is a benefit in advisors undertaking a diagnosis
of firms’ key needs and tailoring program content to support specific types of innovation
behaviour.
The findings indicate that the delivery method of a small business advisory program affects
its level of success in impacting on firm behaviour. Programs which involve opportunities for
shared learning between firms and advisors, and incorporate practical content that addresses
the specific needs of the firm, have a greater impact on the firm’s acquisition of skills and
capabilities. Programs which require firms to analyse their business and implement changes
are associated with higher subsequent levels of organisational innovation. This requires a
deeper form of intervention that would involve repeated and ongoing interaction with the
firm over a period of time.
Whole Group Discussion around Enterprise Solutions Program
The Workshop participants considered a specific case study of the Enterprise Solutions
Program (ESP) being designed by the Department as a procurement program for
Commonwealth, State and Territory government agencies requiring government to identify
and articulate a problem for which they require a technological solution.
The ESP is a pilot program, funded for five years, which aims to: (a) stimulate innovative
activity in SMEs, by raising their R&D and innovative capacity; (b) increase SMEs access to
government procurement; (c) equip SMEs to respond to government agencies requirements;
and (d) improve the innovation procurement culture and identification of needs. Importantly,
the ESP aims to increase SMEs’ access to government procurement. The ESP is based on the
Victorian Market Validation Program (MVP), which is aimed at SMEs operating in Victoria,
whereas the ESP is for SMEs based anywhere in Australia.
In order to develop a framework for evaluation of the effectiveness of the program, it was
acknowledged that it is critical to start with defining the expected outcomes. There are
evaluation methodologies used by the Victorian government for the MVP, as well as
AusIndustry for their programs, which could possibly be adapted.
It was agreed participating SMEs should be required to report back after five years, and that
an impact study on all participants could be conducted, with case studies developed after the
five years. Specific indicators could include the participating SMEs’ expenditure on R&D,
sales after their involvement in the program, marketing of the solution, take up of the
innovative solution by other agencies, government jurisdictions (State, Territory or
Commonwealth), and countries, and that ATO data could be used where possible. Feedback
9
on the SMEs’ willingness to work with Government in procurement matters prior to, and
after completion of the ESP, could also be obtained.
It was suggested that to help government to identify their requirements for innovation
procurement, a problem management framework could be developed, with problem solving
round tables conducted in government agencies.
Behavioural and cultural change in procurement could be measured; however this goal may
be too ambitious. Intermediate evaluation of the Program could be publically available for
transparency, and to demonstrate the learning of participants. It was acknowledged that
funding for evaluation should be allocated prior to the ESP’s release, and allow for the
capturing and monitoring of data.
The way forward - Future Workshops
The Workshop was well received by the participants, with an anticipation of a continuing
dialogue between researchers and government.
For future workshops, there was a desire to see presentations by public servants, particularly
those involved in research around government programs, with a view of identifying areas
where academics could add value. It was felt that presentation on topics that came about
through collaboration between academics and public servants would be of interest. Overall, a
theme based approach was considered to be the best approach. Topics suggested included
linking productivity and innovation outcomes, procurement and innovation, absorption of
overseas originated innovation, cross-industry and cross-sector diffusion, and measuring
innovation systems.
The participants wanted to see more presentations and discussion about research meeting
industry, with a focus on strategies and options for linking SMEs with universities and
research organisations, e.g., the work currently being implemented through the establishment
of Industry Innovation Precincts.
As to the format of a future workshop, it has been suggested that more time be spent on
discussing issues around tables. Brainstorming in smaller groups was considered valuable,
with longer sessions dedicated to key issues on the program.
The Department looks forward to further dialogue with academics and researchers, with a
possible theme of emerging innovation opportunities.
10