"N^ MIT. LJeRARIES - DEWEY ^ .:^eS. HD28 .M414 MIT LIBRARIES UDUPL II ' International Center for Research on the Management 3 9080 00932 771 6 of Technology Metrics to Evaluate R&D Groups Phase I: Qualitative Interviews Florian Zettelmeyer John R. Hauser WP # 125-95 March 1995 Sloan © WP # 3818 1995 Massachusetts Institute of Technology Sloan School of Management Massachusetts Institute of Technology 38 Memorial Drive, E56-390 Cambridge, 02139 MA Acknowledgment This research was funded by the International Center for Research on the Management of Technology, Sloan School of Management, M.I.T. We wish to thank the managers, scientists, and engineers who donated their time to talk to us about this important topic. ;»/iASSACHUSE;ns institute OF TECHNOLOGY JUN 2 8 1995 UBRAR!F£ Florian Zettelmeyer (Ph.D. student): John R. Hau.ser (Kirin Professor of Marketing): Massachusetts Institute of Technology Sloan School of Management 38 Memorial Drive, E56-345C Cambridge. MA 02 139 tel(617) 253-3198 fax (617) 258-7597 tlorian@mit.edu Massachusetts Institute of Technology Sloan School of Management 38 Memorial Drive. E56-314 Cambridge. tel MA 02 139 (617) 253-2929 fax (617) 258-7597 Jhuuser @sloan. mii.edu Abstract We describe the results of in-depth qualitative interviews with Chief Executive Officers, Chief Technical Officers, and researchers interviews we explored how at ten large research-intensive organizations. these organizations measure success in the they provide incentives to managers and researchers in tiers of the R&D R&D. We R&D In these mission and how gained insight into the three mission: exploring the tools of the future, creating the tools, and pioneering the use of the tools. We learned Metrics of success vary by how these tiers relate to the university and the business units. in part, tier, because the role of corporate investment, the role of managers, and the role of internal customers vary by This report provides an initial practicing managers and researchers. suggests research topics which we tier. perspective on R&D This perspective, which metrics from the viewpoint of is Phase I of our research project, will explore in greater depth in subsequent Phase summarize these research directions at the end of this report. II. We In 1990, in the U.S. alone, private corporations spent over $70 billion on research and This was approximately 3.4% of development (R&D). Similar spending occurred in Japan (over $27 billion) and to justify R&D (over $19 billion). In order exceeds the firms' targets. However, investments clearly visible in R&D may be While the costs are are inherently difficult to evaluate. and are recorded as they incur, years in the future and R&D Germany total profits. investments of this magnitude, private industry must believe that the return on investment (ROI) for if and 46.8% of total sales the return on R&D investment hard to attribute to a specific project. develops a strategic technical competence that pursue must be to applied across is particularly true many projects and Furthermore, decisions on which projects pervades everything that the organization does. fund or which strategic competence is This may occur many made under to considerable market uncertainty and technology uncertainty. Our long-term goal to allocate investments want to know how and we want to to understand we must first how best to allocate understand how R&D investments, but in order R&D firms evaluate the We function. firms decide whether a project, a program, or a strategic direction succeeds know how and managers based on firms is now perform top management rewards and motivates their past these tasks R&D scientists, engineers, performance and/or their potential. Once we understand how we will be better able to develop a theory to describe and to improve these actions. In this paper and researchers us. We we describe phase at a variety of firms I of our research. who invest heavily in have purposefully chosen not published framework. Instead themselves to describe the world we have in to In this phase R&D. We we spoke structure this description within any previously attempted to allow the managers and researchers which they operate. We chose this strategy This report provides a snapshot of the current beliefs; For example, one organization, which was once run as a central laboratory, units, another organization has is now R&D were "laid because many R&D or are in later phases of aligned with the business given the business units more power by allowing them to fund organization reorganized so the manufacturing and managers report here what they told of the organizations in our sample have recently changed the way they evaluate the process of doing so'. to R&D directly, together." another organization from a center of excellence to the divisions, and another organization has restructured them on developing projects that result in profitable products and services. its R&D moved another its R&D division to focus R&D Metrics to Evaluate Groups Page 2 our research will evaluate these beliefs from the perspectives of history and theory. The paper methods. We is structured as follows. We provide a brief description of the then describe a tiered perspective on the R&D mission and within that structure describe the role of customers, managers, and corporate investment. that firms use and relate those metrics to motivation suggested research directions for the continuation of sample and our and incentives. We describe the metrics We close with a set of research stream. this Sample and Methods This report GmbH, Cable & is based on qualitative interviews at AT&T Wireless, Chevron Petroleum Technology (Direction des Etudes et Recherches), Hoechst Celanese Company, Research Lab), and Varian sufficient to raise many of the Vacuum issues of We Systems. measuring R&D feel that this who reported to the CTO, and researchers within the group of firms is Later phases of this research success. speak to the Chief Technology Officer (CTO), the person(s) to person(s) France Army (Missile RD&E Center, At each firm we attempted will attempt to quantify these issues through a larger scale survey. to Electricite de Advanced Technology Group, Polaroid Corporation, Schlumberger (Measurement and Systems), the U.S. Army Bosch Bell Laboratories, Robert whom R&D the CTO reported, the organization. In total we interviewed 43 managers or researchers. The interviews varied from approximately one hour to a fiill day of interviews. In some simations the interviewee showed us around the facility and introduced us to organization. In each interview affected by the organization, success, and how we how sought to understand it should measure motivation and evaluation of employees and were freewheeling. We people in the the interviewee affected and the interviewee believed that the organization measured he or she believed employee behavior and the firm's how many how formal and profitability. R&D success. We informal incentives was R&D discussed the (if any) affected Because the interviews were exploratory, they allowed each interviewee to explore the topics in any way that he or she found comfortable and we encouraged each person to speak about related topics which they found interesting. Because of the seniority of the people in our sample and because of the sensitive nature , Metrics to Evaluate of the interviews, R&D we Groups Page 3 documents detailed notes that were given which where interrelated topics. We later by the interviewees.) to us R&D mission. the each interview word, strucmre at 6.2, 6.6 to describe their ..., "tier," R&D we tie any Instead the interviewer kept analysis of the content suggested 37 from one of our interviewees, but we found We tiers. 1. Army feel that this tiered Tiers of the R&D R&D Laying the foundations E.xploring the tools of the basic uses funding numbers such as 6. strucmre varies is important from one to 1 our story tier to another. Mission Example Performed by Activity 1 not begin by describing a tiered strucmre of the For example, the U.S. site. Table Tier we would Mission because metrics and incentives vary as the mission of TierO that attempt to cover each of these topics in the following sections. metrics and incentives We draw An transcribed. Tiers of the To understand promised (The only exceptions are quotes from widely interviewee (or organization) directly to a quote. circulated We did not use audio recording. University or Mathematics Basic Research Lab coding R&D for Algorithm development the future Tier 2 Creating the tools R&D Tier 3 Pioneering the use of R&D Pilot Business Units Technology Algorithm implementation system the tools Tier 4 Using the tools The tiered strucmre is described transfer to business units routinely in Table 1. Tiers 1. 2. and 3 describe the missions of R&D Metrics to Evaluate R&D the laboratory . We and 4 are included to describe the interface between Tiers (The boundaries vary by firm, e.g., in some cases suppliers and customers. university; in some cases Page 4 Groups it and tier its the is a basic facility within the firm.) is illustrate the role R&D of each tier through an example from one of our interviewees. Consider an organization that wants to communicate thousands of high-resolution detailed 3dimensional (3D) images to and from a remote field could do its site. If the firm could do this then the firm job more effectively and with significantly lower cost than having to analyze the images on-site. However, the sheer volume of data means that today's technology does not have the bandwidth Tier images, to is accomplish the goal. the basic research that lays the foundations for later tiers. might be the development of the tier be coded for transmission. The university or a central facility. must have the ability to identify R&D its solution. laboratory 3D mathematics that allows these images These mathematics may or may not have been developed application, but they prove important to it fractal In the case of Most may for this were developed likely they to at a not need to develop these tools, but whether these mathematics exist, to find out where they exist, and access the knowledge. Tier images, tier the images. 1 1 uses basic foundations to explore the tools of the future. might include the development of algorithms that use Once developed, the algoridtans fractal 3D mathematics to 3D code might have other uses within the firm and, once developed, the algorithms might give the firm a competitive advantage. might have focused only on In the case of imaging or they may have been trying The tier 1 researchers to explore tools that could solve a portfolio of problems. Tier 2 creates the tools. For example, tier 2 researchers might write the software and develop (or buy) the hardware to implement the algorithms. Although the knowledge might be generalizable, tier 2 researchers are usually problem driven. in the business units and focus on solving the customers' problems. Tier 3 pioneers the use of the tools. application to demonstrate the The tier 3 pilot They often work with customers system may In tier 3, the laboratory might develop a pilot 3D imaging system and not pay for developed will become cost effective in itself, to solve the but, if tier 3 is subsequent applications. problems of implementation. successful, the system that In many is cases the experience R&D Metrics to Evaluate Groups Page 5 gained with the pilot system can be used by more than one business unit, hence there are In such cases, benefits that can be shared. would reap the other business units the benefit without the risk or direct cost. Tier 4 uses the tools routinely. system to the business By units. a single business unit paid for the tier 3 research, if In tier 4, the laboratory might hand over the imaging the costs and the benefits of the system can be this time, projected and each business unit can 3D make an ROI-based The decision. R&D laboratory might be involved in the technology transfer and might play an ongoing support role, but the bulk of tier 4 investment is within the business unit rather than with the R&D laboratory. Recursion Tier 1 is R&D the laboratory of the R&D the core technological competencies that allow R&D to compete If R&D Many R&D managers in Indeed, in to the business units. and products or out tier 1 its processes. some research. tier cases, the business units can 1 research directly, but tier tier 3 research, just like By the The firms same analogy, that we tier 3 On research. tier metrics can be used to evaluate For example, in 1 that R&D (Here we assume tier R&D R&D services R&D gives the laboratory sell its gives the laboratory the ability lo carry tier 1 research and in tier 0. may 1 tier 2 or tier 3 be more difficult to evaluate if research, then analogous tier 2 or research. in this project one can use internal customers new product and that better customers outside the firm. (Hauser, then use those targets to evaluate the will enable the Simester, and to evaluate internal suppliers. For example, we might ask a product-development group within a business unit satisfaction targets for a its gives the firm the ability to research an earlier research paper Wemerfelt 1995) we suggest 1 "sell" provides the the other hand, the recursive property suggests that business unit metrics can be used to evaluate tier 3 tier buy 1 to serve interviewed recognized the need for This recursive property suggests that 2 and R&D knowledge generated the need to facilitate access to the tier it our sample are facing greater pressure to Rarely do the business units "buy" the ability to sell tier 2 provides the firm with effectively, then tier laboratory with the core technological competencies that allow better. than it laboratory. to set sales-and- R&D product-development group to laboratory. choose more Metrics to Evaluate R&D aggressive targets, hence the targets measure the success of Groups Page 6 R&D.) In that paper we demonstrate that such an internal customer- internal supplier system will encourage both groups to choose the actions and technology that maximize the long-term profits of the firm. Thus, we were to ask the customers tier R&D's R&D customers to evaluate the would be laboratory with this linked system, then really evaluating tier 3 or perhaps tier 2. 2 and tier 3 researchers to evaluate tier 1 and the if tier 1 We mm, could, in use the researchers to evaluate tier 0. Balance the Short-term and the Long-term Tier 3 researchers are short-sighted! The customers of interviewees. pressure on tier 3 tier 3 R&D We Thus, the long-term among Any implies a from a number of They prefer knowledge and technology contrast this time horizon with that of tier the tiers. this to deliver so that fruits are visible within that horizon. highly uncertain. projects, heard comments like research often have a 3-5 year time horizon and they put tier 3 researchers will take fewer risks. available. We 1 organizational changes in footnote is currently vs. short-term focus is implicit in the allocation of effort long-term tradeoff. 1 that which might be 8-15 years away and organizational strucmre, such as funding short-term vs. This means that as a shift all R&D through business-unit (Our interviewees perceived the by top management to a focus on short-term goals.) Evaluation Criteria Vary by Tier Each organization recognized For example, tier 1 research judgments of the long-term evaluated by R&D's is by tier. often evaluated by peers or by committees based on informed viability and value of the research direction. Tier 3 research can be customers directly on measured returns. look for "milestones" in evaluating Many that the evaluation criteria (and the evaluator) vary of the formal methods, like tier 1, PRTM^, One organization stated that you but you look for "deliverables" in evaluating tier 3. are viewed as better for evaluating tier 3 than tier "See e.g. McGrath, Anthony, and Shapiro (1992) 1. R&D Metrics to Evaluate To explore Groups variation further, this described the role of each We tier. responsive. organizations we / / One Business units are viewed as the customers of CTO felt especially tier 3 R&D which Consider the following examples: management now recognizes that top more customer the importance of his function is Another CTO sees the business units as customers and feels that his organization will be Another viewed as how CTO well it a partner of the business units. delivers new business opportunities to those customers. told us that business units can "buy" R&D from his organization and. if needs are not met, they can buy outside the firm. In one organization, five years, / R&D, function R&D their / R&D. R&D because evaluated by / R&D's Customers Role of provides an internal market with which to value our interviewees tier 3. our sample are making the in how provide more detail on begin with - The Tier 3 Many Page 7 it when aligned the corporation introduced a business unit strucmre in the last R&D with the business units. compete with one another In another organization different laboratories for tier 3 funding. This orientation toward customers appears to imply three trends. as a metric, attempts by R&D Customer satisfaction laboratories to determine customer needs, and a profit-center-like evaluation of tier 3 research. Customer Satisfaction In / many organizations customer satisfaction "Customer satisfaction is the number one is a key metric for tier 3 research.' priority." (from a facility with mostly tier 3 funding) / "R&D has to be developed in the marketplace." (from a .Ml quotes in this document are paraphrases and not actual quotes. facility that does mostly systems Metrics to Evaluate R&D Groups Page 8 integration) / Technology assessment "What does is it do for the customer?" (from a facility recently reorganized) / "the development organization wants specifically for a / new technology ~ to develop a new product and asks research that process has a lot of power" "customers have direct input on the team performance and hence on the evaluation of technical staff" (from a facility that depends Also in a non-profit organization such as the U.S. upon business Army, unit funding) funding tier 3 is becoming much more customer oriented and can depend upon customer satisfaction surveys. However, customer satisfaction feedback is One not universal. CTO told us that he wished that his people would be more comfortable stepping out of the lab and going Another interviewee pointed out market. researchers could get good performance Furthermore, although there it is usually based on is even the if customer feedback for research qualitative and that there are other evaluation criteria ratings judgment of perceived value customer was in tier 1 to the that unsatisfied. (and presumably tier 2), after a reasonable period of time. Understanding or Predicting Customer Needs A consequence of a customer-satisfaction orientation attempting to measure or predict customer needs. in is R&D that laboratories are For example, Hewlett-Packard (HP) was not our sample, but researchers in our sample cited what they believed to be the They believed identify more that the "only thing that successful companies have in customer needs two years before the customer knew the need. to technological solutions to demonstrate that customer input is " HP common While principle. that they is this may refer underlying needs than the needs themselves, this belief does key to the success of Our interviewees suggested many ways to tier 3 research. obtain information on customer needs including: / taking the customer's business metrics and figuring out what your research is worth to them / encouraging the business units to express needs that are independent of the current Metrics to Evaluate R&D Groups Page 9 programs / have / "most development effort based on informal contacts with customers scientists initiate proposals is initiated by marketing" (from a facility that deals with a few large customers) We did not probe on whether tier 3 research uses formal methods such as the voice of Hauser 1993) the customer (Griffin and from previous research that on customer needs, but we know to obtain information such formal methods are growing in popularity. uncover the needs of leading-edge users and uncover those needs articulating.) As one However, "When interviewee said, These comments imply ability the desire of the R&D by R&D starting for from methods to that the (Such methods also customer has difficulty measure customer needs scratch, the key is to is clear. become customer focused." that the set of metrics for tier 3 should include (1) a measure of the laboratory to identify customer needs and (2) a measure of whether the laboratory fulfilled specific customer needs. Profit-Center-Like Evaluations Some interviewees expressed the belief that one measure of customer satisfaction whether customers come back for more funding. Not organizations have attempted to create internal markets. sample) has • made a pure profit center. we found that However, no organization some (in our Consider the following examples: At one organization the amount of money R&D, / R&D surprisingly, is is a negotiation process among marketing, and manufacmring. At another organization the "development factories" are run as a hybrid between a profit center and a cost center. / The development get as / much laboratories in one organization (as opposed to central facilities) funding as can be sold to clients. At one firm, development puts forth a binding contract / R&D is capabilities and marketing puts forth priorities and negotiated. In another firm development proposes projects and the business units decide whether or not to fund them. R&D Metrics to Evaluate It is clear that market measures can organizations seem We on explanations speculate make Page 10 Groups and are being used for to recognize that there are forces that for this phenomenon tier 3 prevent a pure profit-center approach. satisfaction measures can be used profit (Hauser, Simester satisfaction systems Thus, market measures later in the paper. sense within a portfolio of measures, not as the only measure. independent of customer satisfaction measures. However, research. Market measures are not We have shown in earlier research that customer encourage a focus on long-term rather than on short-term to and Wemerfelt 1994) and we have shown that some internal customer have profit-center- like interpretations (Hauser, Simester and Wemerfelt 1995). Tier 2 - The Role of Management Tier 2 provides the bridge from basic research to development. ground said, that the role of managers and managerial judgment is It is most important. middle in this As one person "The customer knows the direction but lacks the expertise; researchers have the expertise but lack the direction. The capabilities Tier 2 must match the expertise with the direction. strategic technological To make projects. " this decision, developed in tier competencies of the the tier 2 fum are implicit in the choice of tier 2 manager must understand the potential of the and must anticipate the needs of the customers of tier 1 3. Because of both technological and market uncertainty, these managers maintain a portfolio of projects. Because economies result from focus, these managers maintain continuity and attempt a bank of knowledge. In order to realize the synergies of focus, potential of core technologies across business units. We R&D to build must communicate the address each of these issues in turn. Core Technological Competency and the Choice of Tier 2 Projects In many sometimes cases the firm's strategic plan has a large influence on the choice of tier 1) projects. In "Marketing/Strategy Input to strategy input some cases R&D," comes from overseeing the input is tier 2 (and based on a formal document such as the or facilitated by a formal office, while in other cases the directors. Most sites have some mechanism to ensure that 1 Metrics to Evaluate R&D Groups Page the projects that are selected are those that 1 the strategic plan and those that build (or continue) fit the technological core competence of the organization. While a technological strategic plan helps a laboratory can, directions For example, competency. if implicitly, choose technological directions, the choice of determine an managers consistently the tier 2 core organization's technological depend upon select projects that small electrical motors, then the laboratory and the organization quickly develop specialized knowledge This knowledge in that technology. complete those projects more effectively, to then be exploited in other projects is faster, and/or at An lower cost. implication of this role in setting strategic technological directions is order interesting R&D that in acts as a federator providing a link between the business units so that they coordinate their strategies around core technological competencies. This interrelationship the is recognized in (An organization's culmre organization. 1984.) At another organization the fits 2 metrics. compensation depends upon the contribution "at risk" research tier tier is one firm at to the vision and culture of the researchers are evaluated on 1) of See Wemerfelt often cited as a core competency. 2 (and tier 50% For example, how well the into the strategic plan. Portfolio Issues Tier 2 must anticipate the needs of customers, but tier 3's Tier 2 must create capabilities based on the ideas of tier I. Thus, most firms maintain a portfolio of projects in tier 2 in respond idea customers to in tier 3 and in all tier 1 ideas prove feasible. order to have the flexibility to fail. technologies that ultimately that failure is much it fail They in their estimates (the estimates varied part of the territory. eliminate success. than can not do so perfectly. order to select an alternative direction should a promising While our interviewees varied is but not it They that if how much money 20% you to 80%), they to fail in tier 3. when all when allocated to acknowledged less expectations are lower and less The more expensive and is try to eliminate failure, also indicated the value of taking risks early better to fail in tier 2 (or tier 1) is felt from of is at risk. money time-critical tier 3 projects you also is It at risk can then use R&D Metrics to Evaluate Groups Page 12 technologies that are highly likely to succeed. Too Tier 2 metrics must recognize portfolio issues. a promising technology will force the researchers to take too A "safe" technologies. portfolio. points the way risks may mean and may encourage only too little when variance in the a failed technology technology that ultimately succeeds. to a organizations discussed the tradeoff of software vs. hardware as a portfolio issue. organization stressed taking charge of service to few Metrics might also consider the externalities that result Two One focus on only "safe" technologies strong a penalty for the failure of its tier 3 software because it in that, software in order to gain flexibility in providing Another organization decided customers. felt its its to shift its portfolio toward advances in hardware were more rapid than industry, advances in software, hence, a software portfolio had greater continuity. Continuity and a Knowledge Bank In order to manage manage the portfolio, continuity. and the development of core technological competencies, in order to learn This continuity, and organization to react quickly technological expertise and it its when it To be an needs to do so. It fill missing expertise. In some basis for choosing between a customer orientation. capabilities and needs, Such managers are so that the business units can be intelligent customers of evaluate technical suppliers. One of allows the organization to monitor must be purchased outside the organization. becomes the R&D what technology In these to buy. Expertise effective bridge R&D expertise and our interviewees stressed the need for helps the organization identify and cases hard- won instimtional knowledge R&D failures, corollary of institutional knowledge, enables the cases, tier 3 might need technology that Managers with from in order to the surprises Such people are also R&D and 2 needs managers with needed in the business units so that the business units can rare. from our interviews managers. One firm explicitly acknowledged tier R&D is the role of R&D as a source of such as a source of technical managers. Another R&D Metrics to Evaluate Groups Page 13 organization stressed the need for continual renewal of the "helper ranks" because the (managers and researchers) come from the "helper ranks. stagnate the helper ranks if were They " with non-growing filled R&D One staff. recognizes this role by rewarding the development of people. are shifted between the business units and felt that In the stars the organization would organization explicitly same organization people develop both an in order to new R&D and a customer expertise. Of course management is not R&D Mechanisms Not selection. in all all organizations succeed. and illiterate to interviewee complained to us that top bad decisions. that this accounts for develop managers vary, but one mechanism seems to be a Darwinian self- researchers have an interest in customers and not everyone with an interest customers has technical expertise, but those the managerial ranks are more One more likely to that are have both promoted from the researcher ranks The greater skills. the combined to skills, the likely they are to succeed. Tier Tier 1 provides the organization's strategic plan. ~A 1 Corporate Investment raw material with which Tier 1 tier to research develops the platform or the architecmre that used across the products of many business units. It is more also tools to solve a basic problem. For example, the (corporate) likely to consider a variety of weapon Development and Engineering Center matches technology 2 is Army likely to solve the is likely to consider a variety of Research Laboratory delivery systems, whereas the more an Army is more Missile Research problem by designing a missile system. Benefits that result from they are recognized, they to the tier 1 facility, the may people tier 1 research may not be recognized for 8-15 years and, not be attributed to tier in that facility researchers have a time horizon shorter than 8-15 years, risk research. may have changed units have a time horizon shorter than 8-15 years they market outcomes. 1 it if they are attributed in the interim. may undervalue tier 1 If the business research. will be difficult to motivate This measurement and reward challenge and free-ridine. Even is when compounded with If the them by the issues of R&D Metrics to Evaluate The benefits Groups Page 14 from any one idea exploration are highly uncertain technological feasibility, (2) market demand, and (3) If we were she is to fit in terms of (1) with the organization's strategic needs. measure and reward a researcher only on the outcomes from the idea we would expose exploring, the researcher to more risk than if that he or he or she were compensated on the basis of the portfolio (equalizing both on expected compensation). That mean outcome of the balanced appropriately, the if the research portfolio a measure of uncertainty should be much higher than for the to is reward the researcher based on the portfolio, he or she might recognize based on the entire tier 1 is if we attempt that the portfolio is organization while his or her personal costs are tied to one project. The researcher might be tempted There portfolio divided by However, project. is, to free-ride another free-riding problem on his or her colleagues. we must face when evaluating tier the extent that the core technological capabilities developed in tier 1 benefit 1 research. To more than one business unit, any single business unit can free-ride on investments by the other business units. Taken together, the issues of difficult-to-observe linkages, time horizon, project vs. portfolio risk, free-riding by researchers, and free-riding by business units to measure the success of We researchers. found tier 1 and even more that organizations and reward structures for tier 1 difficult to make it very difficult reward the efforts of tier 1 recognize these challenges by using funding systems research that differ from those used in other tiers. Corporate Funding It is common for organizations to fiand tier of funding (as a percent of total R&D 1 research from central coffers. funding) varies from firms in our sample allocated less than one-third of their 10% R&D to The amount 70%, however, most of funding to tier 1. the The reasons given for corporate funding vary but they seem to be related to the issues described above. For example, many organizations provide a percentage of corporate funding expect the business units to pick up the business units of tier the net present value for money is less I research. (NPV) rest. In this If the benefits calculations. way to tier 1 and they lower the expected cost to the remain the same, the corporate funding For example, suppose alters that the corporate discount rate than the perceived internal rate for the business units, then a project may have Metrics to Evaluate R«S:D Groups NPV a positive Page 15 NPV with the corporate rate but a negative would be especially true if cost reduction could then with the business unit the costs are front loaded as they are in tier make NPV the perceived business unit 1 research.) positive, (This rate. Corporate thus effectively lowering the discount rate and making the business unit more long-term oriented. We One CTO found many comments in our interviews that are consistent with had cut their R«&:D budgets stated that business units this interpretation. to increase short-term profits but A CEO are willing to use his laboratory because corporate funding picked up part of the cost. told us that he He managers. The business must constantly units would look more business units are better judges of units. expenditure to his business unit stated that in the absence of corporate funding they profitable, but the firm interviewee told us that they need a haven for Of R&D justify the corporate tier research 1 tier if 1 would would under-invest in R&D. One lack corporate renewal. research and another told us that the they don't have to pay. course, corporate funding can also address the issue of free-riding by the business In essence corporate funding is a way to enforce cooperation by the business units on the development of a technological core. Managerial Judgment An judgment implication of the measurement and reward challenges cited above is more important organizations allow the the CTO leaves in evaluating tier CTO 20-25% of areas. that managerial research than in evaluating tier 3 research. discretion in allocating part of the corporate funding. Most In one case the corporate funding unallocated to fund serendipitous discoveries. manager is given a component of discretionary funding to explore In one organization tier 3 researchers have to follow the results of a formal In another case, each research new 1 is prioritization process while tier 1 researchers need only treat evaluations of researchers also play a greater role in tier 1 it as information. than they do in other This greater use of managerial judgment reinforces the need for managers technical expertise and customer expenise. Managerial tiers. who have both R&D Metrics to Evaluate Groups Page 16 Research Tourism We tourism." found an interesting phenomenon in In tier 1 projects. some cases research tourism was informal; in other was formalized with an advisory board or a peer-review board with members from other it firms, one interviewee called "research Research tourism recognizes the value of visitors to the laboratory which give feedback on the choice of cases that tier 1 government organizations, and academia. research tourism assures an influx of the peer-review board new ideas The goals seemed some of First, and broadens the portfolio of projects. Second, complements managerial judgment and, hopefully, overcomes be two-fold. to in the evaluation the difficulties with tier 1 of people and projects evaluation. Best People, Creative People, Experienced People An alternative method people rather than to evaluate risk and to to overcome tier 1 patents, or other measures of its To is to bet on good mitigate short-term inherent risk), these "best" people are usually given are evaluated scientific success. on longer-term measures such They may be given as publications*, special titles in order to free These mechanisms enable the organization them from short-term pressures. experienced people evaluation 1 output on a project-by-project basis. encourage creativity (with more protected space" and "a bit the difficulties of tier who have demonstrated past success. to (Note that university tenure bet on fits this model.) However, some of cases, usually "a research area. will the best people are hard to evaluate few percent of the They may be asked on formal measures. staff," they are left pretty to coordinate much alone, but bound to a with the business units in the hope that they be influenced to work on topics that ultimately serve the business units. ^One problem with secret. it a measure such a strategic advantage. This is One interviewee and only 5% from salary and bonuses. as publications is that the organization may not want to share an idea that said that, for these people, motivation gives In these is 95% self-created panicularly acute in the US Army where many projects are classified as Metrics to Evaluate R«feD Groups Page 17 Metrics of Success Depend To of R&D. select the right metrics^ for how This means recognizing are interrelated. For example, R&D, tier 1 succeeds it roles vary succeeds among if it It must also provide must be a source of basic platforms and and developing a portfolio of projects provide tier 3 the tiers and recognizing its to internal customers. and value of the business units. rest tier 2 must play Tier 2 must its become a Tier 2 architecture. customer needs by selecting Tier 2 (and other produce managers with both technological and customer-oriented the rest of the organization, the tiers a conduit to research that define the firm's core technological with the capabilities to serve how provides the raw materials (basic ideas) that matches the firm's technological capability if it the Tier an organization must recognize the tiered structure are later transformed into competitive advantages. outside the firm (tier 0) and Upon skills, competency and tiers as well) managers must that will serve by enhancing the combined role of federator knowledge bank which can be used by the of the corporation for intelligent buying, for trouble-shooting, and for rapid response to competitive actions. customers ~ Tier 3, the largest in terms of research volume, succeeds on the missions. relative size of the tier In addition, However, next. many of serves its Tier 3 can be evaluated on internal customer satisfaction, the the business units. fulfillment of customers' needs, and with profit-center-like incentives. differ if it 1, 2, Namrally organizations and 3 efforts and on the detailed definition of their the roles are blurred and vary continuously from one tier to the the tiered strucmre provides a first-cut at matching metrics to mission. All of our interviewees recognized the interdependencies of the tiers and formalized the interdependencies with a "stage gate" process. serious of formal stages through which a project must pass. A many of them stage gate process provides a The earlier stages require fewer resources per project (or technology), hence allowing a broader portfolio in which each project can be more risky. As the project(s) proceed through the gates greater the projects in terms of meeting the organization's objectives. made in the later stages. By "right metrics" demands are placed on Formal ROI calculations are For more on stage gate see Cooper (1990) and Griffin and Hauser we mean those metrics which, long-term le.Tpected) profit for the firm. if maximized by the R&D organization, lead to the greatest Table 2. R&D Category Qualitative Strategic Goals Judgment Metrics Reported by Interviewees Metric Match to organization's strategic objectives Scope of the technology Effectiveness of a Quality/Value new system Quality of the research Peer review of research Benchmarking comparable research Value of top 5 deliverables People activities Quality of the people Managerial involvement Process Productivity Timely response Quantitative Customer Relevance Strategic Goals Counts of innovations Measures Patents Refereed papers Competitive response Quality/Value Gate success of concepts Percent of goal fulfillment Yield = [(quality*opportunity*relevance* leverage)/overhead]*consistency of focus Process Internal process measures Deliverables delivered Fulfillment of technical specifications Time for completion Speed of getting technology into new products Time to market Time of response Customer Customer to customer problems satisfaction Service quality (customer measure) Number of customers who found Revenues/Costs Revenue of new product in 3 faults years/R&D cost Percent of revenues derived from 3-5 year old products Gross margin on new products Economic value added Break even after release Cost of conunitting further Overhead cost of research " Metrics to Evaluate and use We ... R&D Page 19 Groups that as the basis for compensation decisions and development usually adjust the job until the person does it activities. reasonably well. This can lead to disconnects between compensation and contribution. Target Value Concepts A common approach was a tendency to set targets. given incentives to meet those targets. which had to In some Researchers and managers were then organizations, projects would set milestones In another organization, researchers were evaluated by comparing the be kept. However, goals at the beginning of the year with the delivery at the end of the year. this organization cautioned us that such evaluations could not be done quarterly because of the uncertainty inherent in R&D. While some bonuses were paid if the goals organizations actually gave the highest reward than too high or too low. We if were achieved or overachieved, other the research project was right on target rather found one very interesting combined target- value metric. In organization four targets were established based on the product that was developed from an The effort. targets were production to weighted deviations on target. In this way, a project could be over were a corresponding savings on labor there R&D and production investment. each measure and the project was asked Weights were applied if cost, labor cost, quality cost, that to keep the sum of the on implied production cost cost*. Challenges to Implementation Ratings inflation Our interviewees expressed . difficult to use for motivation manager or *Let P investment. - concern If the that qualitative metrics are person providing the rating (the the customer) does not incur a perceived cost for providing a high rating, then the = actual production Let A. B, C, and the following equation w,(C because they are inflated. a Q) + w,(D were w^, -0=0 cost, D L = actual labor cost. Q = actual quality cost, and I = actual production Then the project is asked to maintain P) + w,(B L) + weights established by management: Wp(A be the targets established for these costs. w,, w^. and w, are - - R&D Metrics to Evaluate Groups Page 20 ratings tend to be clustered toward the top of the scale. This is particularly true for jobs that are not well-defined such as the role of scientists and engineers in tier ratings carry Wemerfelt 1995) between the rater The result information about job performance and are less effective less We incentives. 1. show that an earlier paper in in such ratings inflation and the ratee, but that is this at is that the providing research project (Hauser, Simester and a natural result of the potential for gainsharing one can design incentive systems to take ratings inflation into account. Team issues incentive systems we . Much research combine rewards is done to individual in teams. Indeed we have performance with rewards also found concerns that incentives not be based on teams found to teams. that are too large. that some However, For example, laboratory-wide measures and incentives were believed to affect behavior less than measures based on individual projects. Research culmre would Two . violate the mindsets of the researchers. scientists first Other motivations tier 1, . Scientists needs. but it at odds clearly conflicts with the goals of tier 3. Our interviewees suggested personal motivations including: research. Presimiably researchers see themselves as and profit-maximizing employees second. This cultural issue may not be with some of the goals of fields. organizations expressed the view that a business orientation that researchers have a variety of Researchers tend to want to work in appreciated and/or popular and engineers value highly the ability and the opportunity to conduct hands-on Researchers become enamored with the project rather than with their customers' Researchers are more interested in technological solutions than in commercialization. While some of these issues may be addressed in any incentive system. facility specific, they are deeply held beliefs that must be Metrics to Evaluate R&D Groups Page 21 Examples Table 3 fairly common, lists examples of the incentive schemes that we observed. but they varied in their perceived effectiveness. We Bonus systems were observed some examples of royalties and stock options, but our interviewees did not believe that royalties and stock options were particularly effective. It an interesting comment on the is R&D culture that keeping one's funding or obtaining discretionary funding was perceived as a strong motivator. Table Bonuses 3. Based on Incentives Other how corporation did how customer did Royalties on patents Stock options customer satisfaction License and support to Based on start company operational performance Discretionary funds contribution to vision Based on managers' judgment Get to keep funding Promotion up the Based on technical ladder if a management review company has a dual peer review ladder system visiting committee small group level review Based on level in the hierarchy (assumes good people rise) Metrics to Evaluate R&D Groups Page 22 Suggested Research Our initial interviews suggest a number of important research topics that are relevant to both managerial and academic audiences. Corporate Funding vs. Profit- Center-Like Mechanisms Every firm allocated some percent of R&D funding from corporate sources, although percent varied widely. Earlier we proposed two to coordinate business units that to overcome would otherwise free-ride and (1) corporate funding as a means (2) corporate funding as a means the short-term orientation by the business units. corporate funding should be greater in We explanations: tier 1 than in tiers 2 and Both explanations suggest that 3. plan to explore this issue further with formal mathematical models in order to which measurable conditions determine the percent of corporate funding identify this We allocated to each of the tiers. also hope projects should be funded corporately and to identify the conditions that indicate which should be funded with mechanisms. Once we identify these conditions, we will be able of the proposed explanations, if any, is more that should be likely to to use them which profit-center-like to determine which be the correct explanation. Managers as an Output of R&D Our interviewees suggest a Darwinian mechanism by which as a filtering device to identify and are identified effectively and consistently, they This phenomenon suggests a value for We plan to explore this our sample and whether outputs. R&D R&D we can learn how become a renewable above and beyond metrics and in phenomenon managing the business. R&D its interest in both When such managers asset of the organization. technical output; this value funding. further to determine to R&D and train those rare individuals with expertise the technical and the customer-oriented aspects of should be reflected both in the corporation uses manage if it is generalizable beyond the process better and to measure its Metrics to Evaluate R«&D Groups Page 23 Incentives to Select and/or Build Core Technological Competency clear It is directed by from our interviewees the organization's strategic core technological that are difficult to tie back made by R&D, that the decisions plan and, strategic determine the organization's implicitly, competency. However, such decisions have far ranging implications to the person or We created. such plan to explore hope to how to own to define carefully Research Directed vs. The evidence likely to succeed. managers, from later date. that is and engineers scientists, the technology decisions that are in the In order to make progress on this issue we will what we mean by "core technological competency." Customer Directed is R&D make best interests, they best long-term interests of the organization. have much benefits measure the success of the technological competency develop incentive systems for that, acting in their Any group making those decisions. these decisions usually accrue to other parts of the organization at a We especially tier 2, are R&D persuasive that those products which meet customer needs are more (See review in Griffin and Hauser 1994, table 1.) However, this does not their needs, R&D should be directed by customers. Customers are very good at expressing but R&D has the technological expertise to determine how best to fulfill thoj.e needs profitably. In addition there are often synergies across projects that mean that all R&D must be considered. For might want to adopt a common architecmre across projects rather than re-invent an architecture for each project. This common architecture enables the firm to gain experience example, and, in the long run, more heavily invest a single project. flexible and that become more in front-end R&D that priority be given can be used across a variety of projects. that an organization to those ROI calculation for architecmres that are This concept might extend to basic For example, a firm might explore a multimedia application, even were no immediate customer demand, in this Such goals might suggest design than can be justified with an These goals also suggest technological expertise. there profitable. in order to develop the "know-how" to if be a player fast-moving market. We plan to explore the tradeoffs between a research-directed and a customer-directed Metrics to Evaluate R&D center. R&D Groups Page 24 Our exploration of this the three tiers of the R&D issue will provide insight mission and how to allocate effort on how among to allocate effort projects. Our among research should also suggest metrics and incentives with which to measure and reward success. The Best Set of R&D Metrics Table 2 provides a list of metrics. which metrics should be used when. more formal model. If we We The hope tiered structure of the to improve on these R&D mission suggests qualitative insights with a succeed in the above research challenges we will be better able to suggest which metrics should be used under which conditions and in which tiers of the R&D mission. If we succeed in our research goals lead to the best decisions We will then theories and and efforts be in a position to ways to to suggest by we will understand (1) which metrics and incentives R&D and (2) how one can measure the success of R&D. undertake a large-scale survey of R&D organizations to test our improve the theories. Summary This report has summarized with ten research organizations. will include (1) a many We are of the issues identified by in-depth exploratory research now beginning which formal theory based on these issues, (2) the development of large-scale data collection to test the theory, and (3) the design on the theory. the next phase of our research and implementation of prescriptive models based Metrics to Evaluate R&D Page 25 Groups References (* indicates availability as a project Cooper, Robert G. (1990), "Stage-Gate Systems: A New working paper) New Tool for Managing Products," Business Horizons, (May-June), 44-54. Griffin, Abbie and John R. Hauser (1993), "The Voice of the Customer," Marketing Science, * 12, 1, (Winter), 1-27. (1994), "Integrating _ Center International Cambridge, (1995), Center Development: M.I.T., Simester, and Birger Wemerfelt (1994), "Customer Satisfaction 327-350. (Fall), Research on for MA 02139. Michael E., Paper #112-94, Management of Technology, the * Customers and Internal Suppliers," "Internal Cambridge, R&D," Working * Marketing Science. 13, 4, International McGrath, I. for Marketing and Research on for MA 02139. Hauser, John R., Duncan Incentives," Mechanisms Working Paper #118-95, Management of Technology, M.I.T., the * Michael T. Anthony, and Amram R. Shapiro (1992), Product Success Through Product and Cycle-Time Excellence, Butterworth- Heinemann: Boston, MA. Wemerfelt, Birger (1989), "From Critical Resources Management, Vol. 14, 3, Spring, 4-12. to Corporate Strategy," Journal of General MIT LIBRARIES l|l|l|l|lllll|l|l| nuiiiini 3 9080 00932 7716 !97 ii8 Date Due F£S. J : ]a^