"N^
MIT. LJeRARIES
-
DEWEY
^
.:^eS.
HD28
.M414
MIT LIBRARIES
UDUPL
II
'
International Center for Research on the
Management
3 9080 00932 771 6
of Technology
Metrics to Evaluate R&D Groups
Phase I: Qualitative Interviews
Florian Zettelmeyer
John R. Hauser
WP # 125-95
March 1995
Sloan
©
WP
# 3818
1995 Massachusetts Institute of Technology
Sloan School of Management
Massachusetts Institute of Technology
38 Memorial Drive, E56-390
Cambridge,
02139
MA
Acknowledgment
This research was funded by the International Center for Research on the
Management of Technology, Sloan School of Management, M.I.T. We
wish to thank the managers, scientists, and engineers who donated their
time to talk to us about this important topic.
;»/iASSACHUSE;ns institute
OF TECHNOLOGY
JUN
2 8 1995
UBRAR!F£
Florian Zettelmeyer (Ph.D. student):
John R. Hau.ser (Kirin Professor of
Marketing):
Massachusetts
Institute
of Technology
Sloan School of Management
38 Memorial Drive, E56-345C
Cambridge.
MA 02 139
tel(617) 253-3198
fax (617)
258-7597
tlorian@mit.edu
Massachusetts Institute of Technology
Sloan School of Management
38 Memorial Drive. E56-314
Cambridge.
tel
MA 02 139
(617) 253-2929
fax (617)
258-7597
Jhuuser @sloan. mii.edu
Abstract
We
describe the results of in-depth qualitative interviews with Chief Executive Officers,
Chief Technical Officers, and researchers
interviews
we explored how
at ten large research-intensive organizations.
these organizations measure success in the
they provide incentives to managers and researchers in
tiers
of the
R&D
R&D. We
R&D
In these
mission and
how
gained insight into the three
mission: exploring the tools of the future, creating the tools, and pioneering
the use of the tools.
We
learned
Metrics of success vary by
how
these tiers relate to the university and the business units.
in part,
tier,
because the role of corporate investment, the role of
managers, and the role of internal customers vary by
This report provides an
initial
practicing managers and researchers.
suggests research topics which
we
tier.
perspective on
R&D
This perspective, which
metrics from the viewpoint of
is
Phase
I
of our research project,
will explore in greater depth in subsequent Phase
summarize these research directions
at the
end of
this report.
II.
We
In 1990, in the U.S. alone, private corporations spent over $70 billion on research and
This was approximately 3.4% of
development (R&D).
Similar spending occurred in Japan (over $27 billion) and
to justify
R&D
(over $19 billion). In order
exceeds the firms' targets.
However, investments
clearly visible
in
R&D
may be
While the costs are
are inherently difficult to evaluate.
and are recorded as they incur,
years in the future and
R&D
Germany
total profits.
investments of this magnitude, private industry must believe that the return on
investment (ROI) for
if
and 46.8% of
total sales
the return
on
R&D
investment
hard to attribute to a specific project.
develops a strategic technical competence that
pursue must be
to
applied across
is
particularly true
many
projects and
Furthermore, decisions on which projects
pervades everything that the organization does.
fund or which strategic competence
is
This
may occur many
made under
to
considerable market
uncertainty and technology uncertainty.
Our long-term goal
to allocate investments
want
to
know how
and we want
to
to understand
we must
first
how
best to allocate
understand
how
R&D
investments, but in order
R&D
firms evaluate the
We
function.
firms decide whether a project, a program, or a strategic direction succeeds
know how
and managers based on
firms
is
now perform
top
management rewards and motivates
their past
these tasks
R&D
scientists, engineers,
performance and/or their potential. Once we understand how
we
will be better able to
develop a theory
to describe
and
to
improve these actions.
In this paper
and researchers
us.
We
we
describe phase
at a variety
of firms
I
of our research.
who
invest heavily in
have purposefully chosen not
published framework.
Instead
themselves to describe the world
we have
in
to
In this phase
R&D. We
we spoke
structure this description within any previously
attempted to allow the managers and researchers
which they operate.
We
chose
this strategy
This report provides a snapshot of the current beliefs;
For example, one organization, which was once run as a central laboratory,
units, another organization has
is
now
R&D
were
"laid
because many
R&D
or are in
later
phases of
aligned with the business
given the business units more power by allowing them to fund
organization reorganized so the manufacturing and
managers
report here what they told
of the organizations in our sample have recently changed the way they evaluate
the process of doing so'.
to
R&D directly,
together." another organization
from a center of excellence to the divisions, and another organization has restructured
them on developing projects that result in profitable products and services.
its
R&D
moved
another
its
R&D
division to focus
R&D
Metrics to Evaluate
Groups
Page 2
our research will evaluate these beliefs from the perspectives of history and theory.
The paper
methods.
We
is
structured as follows.
We provide a brief description of the
then describe a tiered perspective on the
R&D
mission and within that structure
describe the role of customers, managers, and corporate investment.
that firms use
and
relate those metrics to motivation
suggested research directions for the continuation of
sample and our
and incentives.
We
describe the metrics
We
close with a set of
research stream.
this
Sample and Methods
This report
GmbH,
Cable
&
is
based on qualitative interviews at
AT&T
Wireless, Chevron Petroleum Technology
(Direction des Etudes et Recherches), Hoechst Celanese
Company,
Research Lab), and Varian
sufficient to raise
many of the
Vacuum
issues of
We
Systems.
measuring
R&D
feel that this
who
reported to the
CTO, and
researchers within the
group of firms
is
Later phases of this research
success.
speak to the Chief Technology Officer (CTO), the person(s) to
person(s)
France
Army (Missile RD&E Center,
At each firm we attempted
will attempt to quantify these issues through a larger scale survey.
to
Electricite de
Advanced Technology Group, Polaroid
Corporation, Schlumberger (Measurement and Systems), the U.S.
Army
Bosch
Bell Laboratories, Robert
whom
R&D
the
CTO
reported, the
organization.
In total
we
interviewed 43 managers or researchers.
The interviews varied from approximately one hour
to a
fiill
day of interviews. In some
simations the interviewee showed us around the facility and introduced us to
organization.
In each interview
affected by the organization,
success, and
how
we
how
sought to understand
it
should measure
motivation and evaluation of employees and
were freewheeling.
We
people in the
the interviewee affected and
the interviewee believed that the organization measured
he or she believed
employee behavior and the firm's
how
many
how formal and
profitability.
R&D
success.
We
informal incentives
was
R&D
discussed the
(if
any) affected
Because the interviews were exploratory, they
allowed each interviewee to explore the topics in any way that he or
she found comfortable and
we encouraged each person
to
speak about related topics which they
found interesting.
Because of the seniority of the people
in
our sample and because of the sensitive nature
,
Metrics to Evaluate
of the interviews,
R&D
we
Groups
Page 3
documents
detailed
notes
that
were given
which where
interrelated topics.
We
later
by the interviewees.)
to us
R&D
mission.
the
each interview
word,
strucmre
at
6.2,
6.6 to describe their
...,
"tier,"
R&D
we
tie
any
Instead the interviewer kept
analysis of the
content suggested 37
from one of our interviewees, but we found
We
tiers.
1.
Army
feel that this tiered
Tiers of the
R&D
R&D
Laying the foundations
E.xploring the tools of
the basic
uses funding numbers such as 6.
strucmre
varies
is
important
from one
to
1
our story
tier to another.
Mission
Example
Performed by
Activity
1
not
begin by describing a tiered strucmre of the
For example, the U.S.
site.
Table
Tier
we would
Mission
because metrics and incentives vary as the mission of
TierO
that
attempt to cover each of these topics in the following sections.
metrics and incentives
We draw
An
transcribed.
Tiers of the
To understand
promised
(The only exceptions are quotes from widely
interviewee (or organization) directly to a quote.
circulated
We
did not use audio recording.
University or
Mathematics
Basic Research Lab
coding
R&D
for
Algorithm
development
the future
Tier 2
Creating the tools
R&D
Tier 3
Pioneering the use of
R&D
Pilot
Business Units
Technology
Algorithm
implementation
system
the tools
Tier 4
Using the tools
The
tiered strucmre
is
described
transfer
to business units
routinely
in
Table
1.
Tiers
1.
2.
and
3 describe the missions of
R&D
Metrics to Evaluate
R&D
the
laboratory
.
We
and 4 are included to describe the interface between
Tiers
(The boundaries vary by firm, e.g., in some cases
suppliers and customers.
university; in
some cases
Page 4
Groups
it
and
tier
its
the
is
a basic facility within the firm.)
is
illustrate the role
R&D
of each
tier
through an example from one of our interviewees.
Consider an organization that wants to communicate thousands of high-resolution detailed 3dimensional (3D) images to and from a remote field
could do
its
site.
If the
firm could do
this
then the firm
job more effectively and with significantly lower cost than having to analyze the
images on-site. However, the sheer volume of data means that today's technology does not have
the
bandwidth
Tier
images,
to
is
accomplish the goal.
the basic research that lays the foundations for later tiers.
might be the development of the
tier
be coded for transmission.
The
university or a central facility.
must have the
ability to identify
R&D
its
solution.
laboratory
3D
mathematics that allows these images
These mathematics may or may not have been developed
application, but they prove important to
it
fractal
In the case of
Most
may
for this
were developed
likely they
to
at a
not need to develop these tools, but
whether these mathematics
exist, to find out
where they
exist,
and access the knowledge.
Tier
images,
tier
the images.
1
1
uses basic foundations to explore the tools of the future.
might include the development of algorithms that use
Once developed,
the algoridtans
fractal
3D
mathematics
to
3D
code
might have other uses within the firm and, once
developed, the algorithms might give the firm a competitive advantage.
might have focused only on
In the case of
imaging or they
may have been trying
The
tier 1 researchers
to explore tools that could
solve a portfolio of problems.
Tier 2 creates the tools.
For example,
tier
2 researchers might write the software and
develop (or buy) the hardware to implement the algorithms.
Although the knowledge might be
generalizable, tier 2 researchers are usually problem driven.
in the business units
and focus on solving the customers' problems.
Tier 3 pioneers the use of the tools.
application to demonstrate the
The
tier 3 pilot
They often work with customers
system
may
In tier 3, the laboratory might develop a pilot
3D imaging system and
not pay for
developed will become cost effective
in
itself,
to solve the
but, if tier 3
is
subsequent applications.
problems of implementation.
successful, the system that
In
many
is
cases the experience
R&D
Metrics to Evaluate
Groups
Page 5
gained with the pilot system can be used by more than one business unit, hence there are
In such cases,
benefits that can be shared.
would reap
the other business units
the benefit without the risk or direct cost.
Tier 4 uses the tools routinely.
system to the business
By
units.
a single business unit paid for the tier 3 research,
if
In tier 4, the laboratory might hand over the
imaging
the costs and the benefits of the system can be
this time,
projected and each business unit can
3D
make an ROI-based
The
decision.
R&D
laboratory might
be involved in the technology transfer and might play an ongoing support role, but the bulk of
tier
4 investment
is
within the business unit rather than with the
R&D
laboratory.
Recursion
Tier
1
is
R&D
the
laboratory of the
R&D
the core technological competencies that allow
R&D
to
compete
If
R&D
Many
R&D
managers
in
Indeed, in
to the business units.
and
products or
out tier
1
its
processes.
some
research.
tier
cases, the business units can
1
research directly, but tier
tier 3 research, just like
By
the
The firms
same analogy,
that
we
tier 3
On
research.
tier
metrics can be used to evaluate
For example,
in
1
that
R&D
(Here
we assume
tier
R&D
R&D
services
R&D
gives the
laboratory
sell its
gives the laboratory the ability lo carry
tier
1
research and
in tier 0.
may
1
tier
2 or
tier 3
be more difficult to evaluate
if
research, then analogous tier 2 or
research.
in
this
project
one can use internal customers
new product and
that better
customers
outside the firm.
(Hauser,
then use those targets to evaluate the
will enable the
Simester,
and
to evaluate internal suppliers.
For example, we might ask a product-development group within a business unit
satisfaction targets for a
its
gives the firm the ability to
research
an earlier research paper
Wemerfelt 1995) we suggest
1
"sell"
provides the
the other hand, the recursive property suggests that
business unit metrics can be used to evaluate
tier 3
tier
buy
1
to serve
interviewed recognized the need for
This recursive property suggests that
2 and
R&D
knowledge generated
the need to facilitate access to the
tier
it
our sample are facing greater pressure to
Rarely do the business units "buy"
the ability to sell tier 2
provides the firm with
effectively, then tier
laboratory with the core technological competencies that allow
better.
than
it
laboratory.
to set sales-and-
R&D
product-development group
to
laboratory.
choose more
Metrics to Evaluate
R&D
aggressive targets,
hence the targets measure the success of
Groups
Page 6
R&D.)
In that paper
we
demonstrate that such an internal customer- internal supplier system will encourage both groups
to
choose the actions and technology that maximize the long-term profits of the firm. Thus,
we were
to
ask
the customers
tier
R&D's
R&D
customers to evaluate the
would be
laboratory with this linked system, then
really evaluating tier 3 or perhaps tier 2.
2 and tier 3 researchers to evaluate tier
1
and the
if
tier 1
We
mm,
could, in
use the
researchers to evaluate tier 0.
Balance the Short-term and the Long-term
Tier 3 researchers are short-sighted!
The customers of
interviewees.
pressure on
tier 3
tier 3
R&D
We
Thus, the long-term
among
Any
implies
a
from a number of
They prefer knowledge and technology
contrast this time horizon with that of tier
the tiers.
this
to deliver so that fruits are visible within that horizon.
highly uncertain.
projects,
heard comments like
research often have a 3-5 year time horizon and they put
tier 3
researchers will take fewer risks.
available.
We
1
organizational changes in footnote
is
currently
vs. short-term focus is implicit in the allocation of effort
long-term tradeoff.
1
that
which might be 8-15 years away and
organizational strucmre, such as funding
short-term vs.
This means that
as a shift
all
R&D
through business-unit
(Our interviewees perceived the
by top management
to a focus
on short-term
goals.)
Evaluation Criteria Vary by Tier
Each organization recognized
For example,
tier
1
research
judgments of the long-term
evaluated by
R&D's
is
by
tier.
often evaluated by peers or by committees based on informed
viability
and value of the research direction. Tier 3 research can be
customers directly on measured returns.
look for "milestones" in evaluating
Many
that the evaluation criteria (and the evaluator) vary
of the formal methods, like
tier 1,
PRTM^,
One
organization stated that you
but you look for "deliverables" in evaluating
tier 3.
are viewed as better for evaluating tier 3 than tier
"See e.g. McGrath, Anthony, and Shapiro (1992)
1.
R&D
Metrics to Evaluate
To explore
Groups
variation further,
this
described the role of each
We
tier.
responsive.
organizations
we
/
/
One
Business units are viewed as the customers of
CTO
felt
especially tier 3
R&D
which
Consider the following examples:
management now recognizes
that top
more customer
the importance of his function
is
Another
CTO
sees the business units as customers and feels that his organization will be
Another
viewed as
how
CTO
well
it
a partner of the business units.
delivers
new
business opportunities to those customers.
told us that business units can "buy"
R&D
from
his organization and.
if
needs are not met, they can buy outside the firm.
In one organization,
five years,
/
R&D,
function
R&D
their
/
R&D.
R&D
because
evaluated by
/
R&D's Customers
Role of
provides an internal market with which to value
our interviewees
tier 3.
our sample are making the
in
how
provide more detail on
begin with
- The
Tier 3
Many
Page 7
it
when
aligned
the corporation introduced a business unit strucmre in the last
R&D
with the business units.
compete with one another
In another organization different laboratories
for tier
3
funding.
This orientation toward customers appears to imply three trends.
as a metric, attempts by
R&D
Customer
satisfaction
laboratories to determine customer needs, and a profit-center-like
evaluation of tier 3 research.
Customer Satisfaction
In
/
many
organizations customer satisfaction
"Customer
satisfaction
is
the
number one
is
a
key metric for
tier 3 research.'
priority." (from a facility with mostly tier 3
funding)
/
"R&D
has to be developed in the marketplace." (from a
.Ml quotes in this
document
are paraphrases and not actual quotes.
facility that
does mostly systems
Metrics to Evaluate
R&D
Groups
Page 8
integration)
/
Technology assessment
"What does
is
it
do
for the customer?" (from a facility recently
reorganized)
/
"the
development organization wants
specifically for a
/
new technology ~
to
develop a new product and asks research
that process has a lot of
power"
"customers have direct input on the team performance and hence on the evaluation of
technical staff" (from a facility that depends
Also
in a non-profit
organization such as the U.S.
upon business
Army,
unit funding)
funding
tier 3
is
becoming much more
customer oriented and can depend upon customer satisfaction surveys.
However, customer
satisfaction feedback
is
One
not universal.
CTO
told us that he
wished that his people would be more comfortable stepping out of the lab and going
Another interviewee pointed out
market.
researchers
could
get
good performance
Furthermore, although there
it
is
usually based
on
is
even
the
if
customer feedback for research
qualitative
and
that there are other evaluation criteria
ratings
judgment of perceived value
customer was
in tier
1
to the
that
unsatisfied.
(and presumably
tier 2),
after a reasonable period of time.
Understanding or Predicting Customer Needs
A
consequence of a customer-satisfaction orientation
attempting to measure or predict customer needs.
in
is
R&D
that
laboratories are
For example, Hewlett-Packard (HP) was not
our sample, but researchers in our sample cited what they believed to be the
They believed
identify
more
that the
"only thing that successful companies have in
customer needs two years before the customer knew the need.
to technological solutions to
demonstrate that customer input
is
"
HP
common
While
principle.
that they
is
this
may
refer
underlying needs than the needs themselves, this belief does
key to the success of
Our interviewees suggested many ways
to
tier 3 research.
obtain
information on customer needs
including:
/
taking the customer's business metrics and figuring out what your research
is
worth
to
them
/
encouraging the business units
to
express needs that are independent of the current
Metrics to Evaluate
R&D
Groups
Page 9
programs
/
have
/
"most development effort
based on informal contacts with customers
scientists initiate proposals
is
initiated
by marketing" (from a
facility that deals
with a few
large customers)
We
did not probe on whether tier 3 research uses formal methods such as the voice of
Hauser 1993)
the customer (Griffin and
from previous research
that
on customer needs, but we know
to obtain information
such formal methods are growing
in popularity.
uncover the needs of leading-edge users and uncover those needs
articulating.)
As one
However,
"When
interviewee said,
These comments imply
ability
the desire
of the
R&D
by
R&D
starting
for
from
methods
to
that the
(Such methods also
customer has
difficulty
measure customer needs
scratch, the key
is
to
is
clear.
become customer focused."
that the set of metrics for tier 3 should include (1) a
measure of the
laboratory to identify customer needs and (2) a measure of whether the
laboratory fulfilled specific customer needs.
Profit-Center-Like Evaluations
Some
interviewees expressed the belief that one measure of customer satisfaction
whether customers come back for more funding.
Not
organizations have attempted to create internal markets.
sample) has
•
made
a pure profit center.
we found
that
However, no organization
some
(in
our
Consider the following examples:
At one organization the amount of money
R&D,
/
R&D
surprisingly,
is
is
a negotiation process
among marketing,
and manufacmring.
At another organization the "development
factories" are run as a hybrid
between a
profit
center and a cost center.
/
The development
get as
/
much
laboratories in one organization (as opposed to central
facilities)
funding as can be sold to clients.
At one firm, development puts forth
a binding contract
/
R&D
is
capabilities
and marketing puts forth
priorities
and
negotiated.
In another firm development proposes projects and the business units decide whether or
not to fund them.
R&D
Metrics to Evaluate
It is
clear that market measures can
organizations
seem
We
on explanations
speculate
make
Page 10
Groups
and are being used for
to recognize that there are forces that
for this
phenomenon
tier 3
prevent a pure profit-center approach.
satisfaction
measures can be used
profit (Hauser, Simester
satisfaction systems
Thus, market measures
later in the paper.
sense within a portfolio of measures, not as the only measure.
independent of customer satisfaction measures.
However,
research.
Market measures are not
We have shown in earlier research that customer
encourage a focus on long-term rather than on short-term
to
and Wemerfelt 1994) and
we have shown
that
some
internal
customer
have profit-center- like interpretations (Hauser, Simester and Wemerfelt
1995).
Tier 2
- The
Role of Management
Tier 2 provides the bridge from basic research to development.
ground
said,
that the role of
managers and managerial judgment
is
It
is
most important.
middle
in this
As one person
"The customer knows the direction but lacks the expertise; researchers have the expertise
but lack the direction.
The
capabilities
Tier 2 must match the expertise with the direction.
strategic technological
To make
projects.
"
this decision,
developed
in tier
competencies of the
the tier 2
fum
are implicit in the choice of tier 2
manager must understand
the potential of the
and must anticipate the needs of the customers of tier
1
3.
Because
of both technological and market uncertainty, these managers maintain a portfolio of projects.
Because economies result from focus, these managers maintain continuity and attempt
a
bank of knowledge.
In order to realize the synergies of focus,
potential of core technologies across business units.
We
R&D
to build
must communicate
the
address each of these issues in turn.
Core Technological Competency and the Choice of Tier 2 Projects
In
many
sometimes
cases the firm's strategic plan has a large influence on the choice of
tier 1) projects.
In
"Marketing/Strategy Input to
strategy input
some cases
R&D,"
comes from overseeing
the input
is
tier
2 (and
based on a formal document such as the
or facilitated by a formal office, while in other cases the
directors.
Most
sites
have some mechanism
to
ensure that
1
Metrics to Evaluate
R&D
Groups
Page
the projects that are selected are those that
1
the strategic plan and those that build (or continue)
fit
the technological core competence of the organization.
While a
technological
strategic plan helps a laboratory
can,
directions
For example,
competency.
if
implicitly,
choose technological directions, the choice of
determine
an
managers consistently
the tier 2
core
organization's
technological
depend upon
select projects that
small electrical motors, then the laboratory and the organization quickly develop specialized
knowledge
This knowledge
in that technology.
complete those projects more effectively,
to
then be exploited in other projects
is
faster,
and/or
at
An
lower cost.
implication of this role in setting strategic technological directions
is
order
interesting
R&D
that
in
acts as a
federator providing a link between the business units so that they coordinate their strategies
around core technological competencies.
This interrelationship
the
is
recognized in
(An organization's culmre
organization.
1984.) At another organization the
fits
2 metrics.
compensation depends upon the contribution
"at risk"
research
tier
tier
is
one firm
at
to the vision
and culture of the
researchers are evaluated on
1)
of
See Wemerfelt
often cited as a core competency.
2 (and tier
50%
For example,
how
well the
into the strategic plan.
Portfolio Issues
Tier 2 must anticipate the needs of
customers, but
tier 3's
Tier 2 must create capabilities based on the ideas of
tier I.
Thus, most firms maintain a portfolio of projects
in tier 2 in
respond
idea
customers
to
in tier 3
and
in
all tier 1
ideas prove feasible.
order to have the
flexibility to
fail.
technologies that ultimately
that failure
is
much
it
fail
They
in their estimates
(the estimates varied
part of the territory.
eliminate success.
than
can not do so perfectly.
order to select an alternative direction should a promising
While our interviewees varied
is
but not
it
They
that
if
how much money
20%
you
to
80%), they
to fail in tier 3.
when
all
when
allocated to
acknowledged
less
expectations are lower and less
The more expensive and
is
try to eliminate failure,
also indicated the value of taking risks early
better to fail in tier 2 (or tier 1)
is
felt
from
of
is
at risk.
money
time-critical tier 3 projects
you also
is
It
at risk
can then use
R&D
Metrics to Evaluate
Groups
Page 12
technologies that are highly likely to succeed.
Too
Tier 2 metrics must recognize portfolio issues.
a promising technology will force the researchers to take too
A
"safe" technologies.
portfolio.
points the
way
risks
may mean
and may encourage only
too
little
when
variance in the
a failed technology
technology that ultimately succeeds.
to a
organizations discussed the tradeoff of software vs. hardware as a portfolio issue.
organization stressed taking charge of
service to
few
Metrics might also consider the externalities that result
Two
One
focus on only "safe" technologies
strong a penalty for the failure of
its
tier 3
software because
it
in
that,
software in order to gain flexibility in providing
Another organization decided
customers.
felt
its
its
to shift
its
portfolio toward
advances in hardware were more rapid than
industry,
advances in software, hence, a software portfolio had greater continuity.
Continuity
and a Knowledge Bank
In order to
manage
manage
the portfolio,
continuity.
and
the
development of core technological competencies,
in order to learn
This continuity, and
organization to react quickly
technological expertise and
it
its
when
it
To be an
needs to do so.
It
fill
missing expertise. In some
basis for choosing
between
a customer orientation.
capabilities
and needs,
Such managers are
so that the business units can be intelligent customers of
evaluate technical suppliers.
One of
allows the organization to monitor
must be purchased outside the organization.
becomes the
R&D
what technology
In these
to buy.
Expertise
effective bridge
R&D expertise and
our interviewees stressed the need for
helps the organization identify and
cases hard- won instimtional knowledge
R&D
failures,
corollary of institutional knowledge, enables the
cases, tier 3 might need technology that
Managers with
from
in order to
the surprises
Such people are
also
R&D and
2 needs managers with
needed in the business
units
so that the business units can
rare.
from our interviews
managers. One firm explicitly acknowledged
tier
R&D
is
the role of
R&D
as a source of such
as a source of technical managers.
Another
R&D
Metrics to Evaluate
Groups
Page 13
organization stressed the need for continual renewal of the "helper ranks" because the
(managers and researchers) come from the "helper ranks.
stagnate
the helper ranks
if
were
They
"
with non-growing
filled
R&D
One
staff.
recognizes this role by rewarding the development of people.
are shifted between the business units and
felt that
In the
stars
the organization would
organization explicitly
same organization people
develop both an
in order to
new
R&D
and a
customer expertise.
Of course
management
is
not
R&D
Mechanisms
Not
selection.
in
all
all
organizations succeed.
and
illiterate
to
interviewee complained to us that top
bad decisions.
that this accounts for
develop managers vary, but one mechanism seems to be a Darwinian
self-
researchers have an interest in customers and not everyone with an interest
customers has technical expertise, but those
the managerial ranks are
more
One
more
likely to
that are
have both
promoted from the researcher ranks
The greater
skills.
the
combined
to
skills, the
likely they are to succeed.
Tier
Tier
1
provides
the
organization's strategic plan.
~A
1
Corporate Investment
raw material with which
Tier
1
tier
to
research develops the platform or the architecmre that
used across the products of many business
units.
It is
more
also
tools to solve a basic problem.
For example, the (corporate)
likely to consider a variety of
weapon
Development and Engineering Center
matches technology
2
is
Army
likely to solve the
is
likely to consider a variety of
Research Laboratory
delivery systems, whereas the
more
an
Army
is
more
Missile Research
problem by designing a
missile
system.
Benefits that result from
they are recognized, they
to the tier
1
facility, the
may
people
tier
1
research
may
not be recognized for 8-15 years and,
not be attributed to tier
in that facility
researchers have a time horizon shorter than 8-15 years,
risk
research.
may have changed
units have a time horizon shorter than 8-15 years they
market outcomes.
1
it
if
they are attributed
in the interim.
may undervalue
tier
1
If the business
research.
will be difficult to motivate
This measurement and reward challenge
and free-ridine.
Even
is
when
compounded with
If the
them by
the issues of
R&D
Metrics to Evaluate
The
benefits
Groups
Page 14
from any one idea exploration are highly uncertain
technological feasibility, (2) market demand, and (3)
If
we were
she
is
to
fit
in
terms of (1)
with the organization's strategic needs.
measure and reward a researcher only on the outcomes from the idea
we would expose
exploring,
the
researcher to
more
risk than
if
that he or
he or she were
compensated on the basis of the portfolio (equalizing both on expected compensation). That
mean outcome of the
balanced appropriately, the
if
the research portfolio
a
measure of uncertainty should be much higher than for the
to
is
reward the researcher based on the portfolio, he or she might recognize
based on the entire
tier 1
is
if
we
attempt
that the portfolio
is
organization while his or her personal costs are tied to one project.
The researcher might be tempted
There
portfolio divided by
However,
project.
is,
to free-ride
another free-riding problem
on
his or her colleagues.
we must
face
when
evaluating tier
the extent that the core technological capabilities developed in tier
1
benefit
1
research.
To
more than one
business unit, any single business unit can free-ride on investments by the other business units.
Taken
together, the issues of difficult-to-observe linkages, time horizon, project vs.
portfolio risk, free-riding by researchers, and free-riding by business units
to
measure the success of
We
researchers.
found
tier
1
and even more
that organizations
and reward structures for
tier
1
difficult to
make
it
very difficult
reward the efforts of
tier
1
recognize these challenges by using funding systems
research that differ from those used in other
tiers.
Corporate Funding
It is
common
for organizations to fiand tier
of funding (as a percent of
total
R&D
1
research from central coffers.
funding) varies from
firms in our sample allocated less than one-third of their
10%
R&D
to
The amount
70%, however, most of
funding to
tier 1.
the
The reasons
given for corporate funding vary but they seem to be related to the issues described above.
For example, many organizations provide a percentage of corporate funding
expect the business units to pick up the
business units of
tier
the net present value
for
money
is
less
I
research.
(NPV)
rest.
In this
If the benefits
calculations.
way
to tier
1
and
they lower the expected cost to the
remain the same, the corporate funding
For example, suppose
alters
that the corporate discount rate
than the perceived internal rate for the business units, then a project
may have
Metrics to Evaluate R«S:D Groups
NPV
a positive
Page 15
NPV
with the corporate rate but a negative
would be especially
true
if
cost reduction could then
with the business unit
the costs are front loaded as they are in tier
make
NPV
the perceived business unit
1
research.)
positive,
(This
rate.
Corporate
thus effectively
lowering the discount rate and making the business unit more long-term oriented.
We
One
CTO
found many comments
in
our interviews that are consistent with
had cut their R«&:D budgets
stated that business units
this interpretation.
to increase short-term profits but
A CEO
are willing to use his laboratory because corporate funding picked up part of the cost.
told us that he
He
managers.
The business
must constantly
units
would look more
business units are better judges of
units.
expenditure to his business unit
stated that in the absence of corporate funding they
profitable, but the firm
interviewee told us that they need a haven for
Of
R&D
justify the corporate
tier
research
1
tier
if
1
would
would under-invest
in
R&D.
One
lack corporate renewal.
research and another told us that the
they don't have to pay.
course, corporate funding can also address the issue of free-riding by the business
In essence corporate funding
is
a
way
to enforce cooperation
by the business units on the
development of a technological core.
Managerial Judgment
An
judgment
implication of the measurement and reward challenges cited above
is
more important
organizations allow the
the
CTO
leaves
in evaluating tier
CTO
20-25% of
areas.
that
managerial
research than in evaluating tier 3 research.
discretion in allocating part of the corporate funding.
Most
In one case
the corporate funding unallocated to fund serendipitous discoveries.
manager
is
given a component of discretionary funding to explore
In one organization tier 3
researchers have to follow the results of a formal
In another case, each research
new
1
is
prioritization process while tier
1
researchers need only treat
evaluations of researchers also play a greater role in tier
1
it
as information.
than they do in other
This greater use of managerial judgment reinforces the need for managers
technical expertise and customer expenise.
Managerial
tiers.
who have
both
R&D
Metrics to Evaluate
Groups
Page 16
Research Tourism
We
tourism."
found an interesting phenomenon in
In
tier 1 projects.
some cases research tourism was
informal; in other
was formalized with an advisory board or a peer-review board with members from other
it
firms,
one interviewee called "research
Research tourism recognizes the value of visitors to the laboratory which give
feedback on the choice of
cases
that
tier 1
government organizations, and academia.
research tourism assures an influx of
the peer-review board
new
ideas
The goals seemed
some of
First,
and broadens the portfolio of projects. Second,
complements managerial judgment
and, hopefully, overcomes
be two-fold.
to
in the evaluation
the difficulties with tier
1
of people and projects
evaluation.
Best People, Creative People, Experienced People
An
alternative
method
people rather than to evaluate
risk
and
to
to
overcome
tier 1
patents, or other
measures of
its
To
is
to bet
on good
mitigate short-term
inherent risk), these "best" people are usually given
are evaluated
scientific success.
on longer-term measures such
They may be given
as publications*,
special titles in order to free
These mechanisms enable the organization
them from short-term pressures.
experienced people
evaluation
1
output on a project-by-project basis.
encourage creativity (with
more protected space" and
"a bit
the difficulties of tier
who have demonstrated
past success.
to
(Note that university tenure
bet
on
fits this
model.)
However, some of
cases, usually "a
research area.
will
the best people are hard to evaluate
few percent of the
They may be asked
on formal measures.
staff," they are left pretty
to coordinate
much
alone, but bound to a
with the business units in the hope that they
be influenced to work on topics that ultimately serve the business units.
^One problem with
secret.
it
a
measure such
a strategic advantage.
This
is
One
interviewee
and only
5%
from salary and bonuses.
as publications is that the organization
may
not want to share an idea that
said that, for these people, motivation
gives
In these
is
95%
self-created
panicularly acute in the
US Army where many
projects are classified as
Metrics to Evaluate R«feD Groups
Page 17
Metrics of Success Depend
To
of
R&D.
select the right metrics^ for
how
This means recognizing
are interrelated.
For example,
R&D,
tier
1
succeeds
it
roles vary
succeeds
among
if it
It
must also provide
must be a source of basic platforms and
and developing a portfolio of projects
provide
tier 3
the tiers and recognizing
its
to internal
customers.
and
value of the business units.
rest
tier
2 must play
Tier 2 must
its
become
a
Tier 2
architecture.
customer needs by selecting
Tier 2 (and other
produce managers with both technological and customer-oriented
the rest of the organization,
the tiers
a conduit to research
that define the firm's core technological
with the capabilities to serve
how
provides the raw materials (basic ideas) that
matches the firm's technological capability
if it
the Tier
an organization must recognize the tiered structure
are later transformed into competitive advantages.
outside the firm (tier 0) and
Upon
skills,
competency and
tiers as well)
managers
must
that will serve
by enhancing the combined
role of federator
knowledge bank which can be used by
the
of the corporation for intelligent buying, for trouble-shooting, and for rapid response to
competitive actions.
customers
~
Tier 3, the largest in terms of research volume, succeeds
on the
missions.
relative size of the tier
In addition,
However,
next.
many of
serves
its
Tier 3 can be evaluated on internal customer satisfaction, the
the business units.
fulfillment of customers' needs, and with profit-center-like incentives.
differ
if it
1,
2,
Namrally organizations
and 3 efforts and on the detailed definition of their
the roles are blurred and vary continuously
from one
tier to the
the tiered strucmre provides a first-cut at matching metrics to mission.
All of our interviewees recognized the interdependencies of the tiers and
formalized the interdependencies with a "stage gate" process.
serious of formal stages through
which a project must
pass.
A
many
of them
stage gate process provides a
The
earlier stages require
fewer
resources per project (or technology), hence allowing a broader portfolio in which each project
can be more risky.
As
the project(s) proceed through the gates greater
the projects in terms of meeting the organization's objectives.
made
in the later stages.
By
"right metrics"
demands
are placed on
Formal ROI calculations are
For more on stage gate see Cooper (1990) and Griffin and Hauser
we mean
those metrics which,
long-term le.Tpected) profit for the firm.
if
maximized by
the
R&D
organization, lead to the greatest
Table
2.
R&D
Category
Qualitative
Strategic Goals
Judgment
Metrics Reported by Interviewees
Metric
Match
to organization's strategic objectives
Scope of the technology
Effectiveness of a
Quality/Value
new system
Quality of the research
Peer review of research
Benchmarking comparable research
Value of top 5 deliverables
People
activities
Quality of the people
Managerial involvement
Process
Productivity
Timely response
Quantitative
Customer
Relevance
Strategic Goals
Counts of innovations
Measures
Patents
Refereed papers
Competitive response
Quality/Value
Gate success of concepts
Percent of goal fulfillment
Yield
=
[(quality*opportunity*relevance*
leverage)/overhead]*consistency of focus
Process
Internal process
measures
Deliverables delivered
Fulfillment of technical specifications
Time
for completion
Speed of getting technology into new products
Time to market
Time of response
Customer
Customer
to
customer problems
satisfaction
Service quality (customer measure)
Number of customers who found
Revenues/Costs
Revenue of new product
in 3
faults
years/R&D
cost
Percent of revenues derived from 3-5 year old products
Gross margin on new products
Economic value added
Break even after release
Cost of conunitting further
Overhead cost of research
"
Metrics to Evaluate
and use
We
...
R&D
Page 19
Groups
that as the basis for
compensation decisions and development
usually adjust the job until the person does
it
activities.
reasonably well.
This can
lead to disconnects between compensation and contribution.
Target Value Concepts
A common approach was
a tendency to set targets.
given incentives to meet those targets.
which had
to
In
some
Researchers and managers were then
organizations, projects
would
set milestones
In another organization, researchers were evaluated by comparing the
be kept.
However,
goals at the beginning of the year with the delivery at the end of the year.
this
organization cautioned us that such evaluations could not be done quarterly because of the
uncertainty inherent in
R&D.
While some bonuses were paid
if
the goals
organizations actually gave the highest reward
than too high or too low.
We
if
were achieved or overachieved, other
the research project
was
right
on
target rather
found one very interesting combined target- value metric. In
organization four targets were established based on the product that was developed from an
The
effort.
targets
were production
to
weighted deviations on
target.
In this
way,
a project could be over
were a corresponding savings on labor
there
R&D
and production investment.
each measure and the project was asked
Weights were applied
if
cost, labor cost, quality cost,
that
to
keep the sum of the
on implied production
cost
cost*.
Challenges to Implementation
Ratings inflation
Our interviewees expressed
.
difficult to use for motivation
manager or
*Let
P
investment.
-
concern
If the
that qualitative metrics are
person providing the rating (the
the customer) does not incur a perceived cost for providing a high rating, then the
= actual production
Let A. B, C, and
the following equation
w,(C
because they are inflated.
a
Q) + w,(D
were w^,
-0=0
cost,
D
L = actual
labor cost.
Q = actual
quality cost,
and
I
= actual
production
Then the project is asked to maintain
P) + w,(B
L) +
weights established by management: Wp(A
be the targets established for these costs.
w,, w^.
and w, are
-
-
R&D
Metrics to Evaluate
Groups
Page 20
ratings tend to be clustered toward the top of the scale.
This
is
particularly true for jobs that
are not well-defined such as the role of scientists and engineers in tier
ratings carry
Wemerfelt 1995)
between the
rater
The
result
information about job performance and are less effective
less
We
incentives.
1.
show
that
an earlier paper in
in
such ratings inflation
and the
ratee, but that
is
this
at
is
that the
providing
research project (Hauser, Simester and
a natural result of the potential for gainsharing
one can design incentive systems to take ratings
inflation
into account.
Team
issues
incentive systems
we
.
Much
research
combine rewards
is
done
to individual
in teams.
Indeed
we have
performance with rewards
also found concerns that incentives not be based
on teams
found
to teams.
that are too large.
that
some
However,
For example,
laboratory-wide measures and incentives were believed to affect behavior less than measures
based on individual projects.
Research culmre
would
Two
.
violate the mindsets of the researchers.
scientists first
Other motivations
tier 1,
.
Scientists
needs.
but
it
at
odds
clearly conflicts with the goals of tier 3.
Our interviewees suggested
personal motivations including:
research.
Presimiably researchers see themselves as
and profit-maximizing employees second. This cultural issue may not be
with some of the goals of
fields.
organizations expressed the view that a business orientation
that researchers
have a variety of
Researchers tend to want to work in appreciated and/or popular
and engineers value highly the
ability
and the opportunity
to
conduct hands-on
Researchers become enamored with the project rather than with their customers'
Researchers are more interested in technological solutions than in commercialization.
While some of these
issues
may be
addressed in any incentive system.
facility specific, they are
deeply held beliefs that must be
Metrics to Evaluate
R&D
Groups
Page 21
Examples
Table 3
fairly
common,
lists
examples of the incentive schemes
that
we
observed.
but they varied in their perceived effectiveness.
We
Bonus systems were
observed some examples
of royalties and stock options, but our interviewees did not believe that royalties and stock
options were particularly effective.
It
an interesting comment on the
is
R&D
culture that
keeping one's funding or obtaining discretionary funding was perceived as a strong motivator.
Table
Bonuses
3.
Based on
Incentives
Other
how corporation did
how customer did
Royalties on patents
Stock options
customer satisfaction
License and support to
Based on
start
company
operational performance
Discretionary funds
contribution to vision
Based on
managers' judgment
Get
to
keep funding
Promotion up the
Based on
technical ladder
if
a
management review
company has a dual
peer review
ladder system
visiting
committee
small group level review
Based on
level in the hierarchy
(assumes good people
rise)
Metrics to Evaluate
R&D
Groups
Page 22
Suggested Research
Our
initial
interviews suggest a
number of important research
topics that are relevant to
both managerial and academic audiences.
Corporate Funding
vs. Profit- Center-Like
Mechanisms
Every firm allocated some percent of R&D funding from corporate sources, although
percent varied widely.
Earlier
we proposed two
to coordinate business units that
to
overcome
would otherwise
free-ride
and
(1) corporate funding as a
means
(2) corporate funding as a
means
the short-term orientation by the business units.
corporate funding should be greater in
We
explanations:
tier 1
than in
tiers
2 and
Both explanations suggest
that
3.
plan to explore this issue further with formal mathematical models in order to
which measurable conditions determine the percent of corporate funding
identify
this
We
allocated to each of the tiers.
also
hope
projects should be funded corporately and
to identify the conditions that indicate
which should be funded with
mechanisms. Once we identify these conditions, we will be able
of the proposed explanations,
if
any,
is
more
that should be
likely to
to use
them
which
profit-center-like
to determine
which
be the correct explanation.
Managers as an Output of R&D
Our interviewees suggest a Darwinian mechanism by which
as a filtering device to identify
and
are identified effectively and consistently, they
This phenomenon suggests a value for
We
plan to explore this
our sample and whether
outputs.
R&D
R&D
we can
learn
how
become a renewable
above and beyond
metrics and in
phenomenon
managing the business.
R&D
its
interest in both
When
such managers
asset of the organization.
technical output; this value
funding.
further to determine
to
R&D
and
train those rare individuals with expertise
the technical and the customer-oriented aspects of
should be reflected both in
the corporation uses
manage
if it is
generalizable beyond
the process better
and
to
measure
its
Metrics to Evaluate
R«&D Groups
Page 23
Incentives to Select and/or Build Core Technological Competency
clear
It is
directed by
from our interviewees
the organization's
strategic core technological
that are difficult to tie
back
made by R&D,
that the decisions
plan and,
strategic
determine the organization's
implicitly,
competency. However, such decisions have far ranging implications
to the person or
We
created.
such
plan to explore
hope
to
how
to
own
to define carefully
Research Directed
vs.
The evidence
likely to succeed.
managers,
from
later date.
that
is
and engineers
scientists,
the technology decisions that are in the
In order to
make
progress on this issue
we
will
what we mean by "core technological competency."
Customer Directed
is
R&D
make
best interests, they
best long-term interests of the organization.
have
much
benefits
measure the success of the technological competency
develop incentive systems for
that, acting in their
Any
group making those decisions.
these decisions usually accrue to other parts of the organization at a
We
especially tier 2, are
R&D
persuasive that those products which meet customer needs are more
(See review in Griffin and Hauser 1994, table
1.)
However,
this
does not
their needs,
R&D should be directed by customers. Customers are very good at expressing
but R&D has the technological expertise to determine how best to fulfill thoj.e needs
profitably.
In addition there are often synergies across projects that
mean
that all
R&D
must be considered.
For
might want to adopt a
common
architecmre across projects rather than re-invent
an architecture for each project. This
common
architecture enables the firm to gain experience
example,
and, in the long run,
more heavily
invest
a single project.
flexible
and
that
become more
in front-end
R&D
that priority
be given
can be used across a variety of projects.
that an organization
to those
ROI
calculation for
architecmres that are
This concept might extend to basic
For example, a firm might explore a multimedia application, even
were no immediate customer demand,
in this
Such goals might suggest
design than can be justified with an
These goals also suggest
technological expertise.
there
profitable.
in
order to develop the "know-how"
to
if
be a player
fast-moving market.
We
plan to explore the tradeoffs between a research-directed and a customer-directed
Metrics to Evaluate
R&D center.
R&D
Groups
Page 24
Our exploration of this
the three tiers of the
R&D
issue will provide insight
mission and
how
to allocate effort
on how
among
to allocate effort
projects.
Our
among
research
should also suggest metrics and incentives with which to measure and reward success.
The Best Set of R&D Metrics
Table 2 provides a
list
of metrics.
which metrics should be used when.
more formal model.
If
we
We
The
hope
tiered structure of the
to
improve on these
R&D
mission suggests
qualitative insights with a
succeed in the above research challenges
we
will be better able to
suggest which metrics should be used under which conditions and in which tiers of the
R&D
mission.
If
we succeed
in our research goals
lead to the best decisions
We
will then
theories
and
and
efforts
be in a position
to
ways
to
to suggest
by
we
will understand (1)
which metrics and incentives
R&D and (2) how one can measure the success of R&D.
undertake a large-scale survey of
R&D organizations
to test
our
improve the theories.
Summary
This report has summarized
with ten research organizations.
will include (1) a
many
We
are
of the issues identified by in-depth exploratory research
now beginning
which
formal theory based on these issues, (2) the development of large-scale data
collection to test the theory, and (3) the design
on the theory.
the next phase of our research
and implementation of prescriptive models based
Metrics to Evaluate
R&D
Page 25
Groups
References
(* indicates availability as a project
Cooper, Robert G. (1990), "Stage-Gate Systems:
A New
working paper)
New
Tool for Managing
Products,"
Business Horizons, (May-June), 44-54.
Griffin,
Abbie and John R. Hauser (1993), "The Voice of the Customer," Marketing Science,
*
12, 1, (Winter), 1-27.
(1994), "Integrating
_
Center
International
Cambridge,
(1995),
Center
Development:
M.I.T.,
Simester, and Birger Wemerfelt (1994), "Customer Satisfaction
327-350.
(Fall),
Research on
for
MA 02139.
Michael E.,
Paper #112-94,
Management of Technology,
the
*
Customers and Internal Suppliers,"
"Internal
Cambridge,
R&D," Working
*
Marketing Science. 13, 4,
International
McGrath,
I.
for Marketing and
Research on
for
MA 02139.
Hauser, John R., Duncan
Incentives,"
Mechanisms
Working Paper #118-95,
Management of Technology, M.I.T.,
the
*
Michael
T.
Anthony,
and
Amram
R.
Shapiro
(1992),
Product
Success Through Product and Cycle-Time Excellence, Butterworth-
Heinemann: Boston,
MA.
Wemerfelt, Birger (1989), "From Critical Resources
Management, Vol.
14, 3, Spring, 4-12.
to
Corporate Strategy," Journal of General
MIT LIBRARIES
l|l|l|l|lllll|l|l|
nuiiiini
3 9080 00932 7716
!97
ii8
Date Due
F£S.
J
:
]a^