EBSCOhost Full Display
wysiwyg://bodyframe.78/http://ehostvgw5...ilter=&dth=96&Forward.x=13&Forward.y=12
Result 80 of 96
[Go To Full Text]
[Tips]
Title: A comparative analysis of two concept evaluation methods for new product
development projects.
Subject(s): INDUSTRIAL project management
Source: Project Management Journal, Dec97, Vol. 28 Issue 4, p46, 6p, 3bw
Author(s): Pascale, Steven; Carland, James W.; et al
Abstract: Examines two different concept evaluation techniques in pursuit of new product
development projects. Discussion of the strengths and weaknesses of the evaluation
methodologies; Influence of differences in perception on innovative concept evaluation
and selection.
AN: 135898
ISSN: 8756-9728
Database: Business Source Elite
Print:
Click here to mark for print.
[Go To Citation]
Best Part
PROJECT MANAGEMENT JOURNAL; DECEMBER 1997
A COMPARATIVE ANALYSIS OF TWO CONCEPT
EVALUATION METHODS FOR NEW PRODUCT
DEVELOPMENT PROJECTS
Abstract
This paper examines two different concept evaluation techniques in pursuit of new product
development and includes discussion of the strengths and weaknesses of these evaluation
methodologies and human factors.
Keywords: change; decision-making organizational change; innovation
The Tucker Torpedo automobile, the space shuttle, and the computer all have one thing in common:
all are headliner innovations from the twentieth century that produced a path-breaking shift. For
centuries, modernization, like these products, has been spawned from inventive, divergent concepts.
Plain and simple.
Just how important are these innovations to technology and progression, and how significant is
fostering creative new ideas to our companies? That is a question whose answer is straightforward to
all of us and whose implications are far-reaching. Oddly, what isn't so apparent is how to sustain
product performance advancement and cultivation of originality.
Clever ideas and product modifications are vital to growth and are the very essence to change. Hence,
innovative concept designs leading to corporate prosperity work best when an evaluation process is
utilized; optimizing both convergent and divergent thinking.
The Role of Innovation
Innovation might well be considered the product or the result of creativity. However, creativity, as a
1 of 8
27/07/2000 16:26
EBSCOhost Full Display
wysiwyg://bodyframe.78/http://ehostvgw5...ilter=&dth=96&Forward.x=13&Forward.y=12
process, continues to have its critics in the world of project management despite its long-held and
wall-established history in product development breakthroughs. In project management circles, one
often hears that an individual is either born with creative tendencies or is not; therefore, one may not
teach a project manager to be creative. Such a perspective demonstrates a limited understanding of
the concepts of creativity. Research in project management clearly indicates that existing and often
resisting paradigms can be changed. The precepts are not complex and are amendable to
management and/or engineering professionals as well as other endeavors.
Aspects of the creative process are not mystical. The problem has simply been that product managers
in the past have tended to focus on the right answer, the tools required to obtain that answer, the
theory in vogue at the moment, or the measurement of tangible outputs. Further, the project
management environment has increasingly attracted specialists in various disciplines; specialists who
are frequently unable to synthesize, much less to mentor creative skills.
The process of innovation usually can begin by using warm-up techniques where words, sounds, or
images are used to flex one's mind by producing something and then building on that concept to
produce something new. These principles include making divergent thinking acceptable, allowing
freedom from threat of evaluation, inviting regression (such as to younger times or fantasy), and
freeing the team from inhibiting mindsets that allow them to think of the creative rather than the
traditional. But more than that, learning to ask and to allow appropriate questions is an essential
vehicle in the augmentation of the creative product development process.
Innovation Techniques
Since evaluation techniques are sometimes unique to product development companies, perhaps a
small beginning is necessary. First, it's helpful if companies formalize a process for consistency
implications in the beginning stages of all concepts being reviewed for feasibility design. As a general
rule, favorable and unfavorable evaluations should take place in a responsive, nonthreatening
environment. Once a comfortable location is determined, the next step includes effective exercises.
Examples of an exercise might be to provide project teams with incomplete drawings of concepts and
current market research reports and ask them to list or draw imaginary solutions that contain form and
functionality to satisfy market demand. Likewise, this approach coupled with a formalized process
best supports an open management style where pushing the envelope of alternatives and discussing
future implications are encouraged. What's more, records, product calendars, and ROI information
should be compiled periodically for validating and changing innovative techniques.
The relative importance of involvement in exercises enable the project team to break down their
existing paradigms about particular company cultures and beliefs and to view the new possibilities
differently. Too often, the tendency is to enhance and improve, not invent or create. Importantly,
product life cycles mandate an everincreasing flow of ideas in order for companies to remain
competitive; therefore, it behooves mentoring project teams to be open and creative.
Product Managers and Designers
Rouse (1991) distinguishes between the talents and visions of product managers and designers,
indicating that their goals, approaches and criteria as well as evaluation process and perspectives are
very different. Such differences mandate a different focus. Individuals who become project/product
managers have a longer-term perspective, prefer individual evaluation and are productdriven,
preferring to do it right regardless of constraints. This is the arena in which ideas for new products
reside. The new, unusual, and original are the focus of product development projects while the
implementation of those inventions is the arena for the product designers. Not surprisingly, more
often than not the focus of designers is more short term in nature. For this reason, a teamwork
approach to getting incremental product improvements is desired for keeping the product
development effort going in the right direction in light of constraints.
2 of 8
27/07/2000 16:26
EBSCOhost Full Display
wysiwyg://bodyframe.78/http://ehostvgw5...ilter=&dth=96&Forward.x=13&Forward.y=12
Two Concept Selection and Evaluation Processes
The decision process for the innovation comes down to the evaluation and determination of
implementation. Ulrich and Eppinger (1995) present a two-stage, structured concept selection
methodology to evaluate and refine design concepts. The first stage is called concept screening and
the second is called concept scoring. Each is supported by a decision matrix that is used to rate, rank
and select the best concepts. Screening is a quick method of evaluation aimed at producing a small
number of viable alternatives. Scoring is a more detailed analysis of the few concepts that result from
screening. The focus of scoring is to choose the single concept most likely to lead to product success.
Both stages follow a six-step process: prepare the selection matrix; rate the concepts; rank the
concepts; combine and improve the concepts; select one (or more in the case of screening) concept(s);
and, reflect on the results and the process.
The process assumes that a project design team will be involved in the selection process. The first
step requires the team to construct a selection matrix that contrasts selection criteria for each of the
product concepts that are under consideration. This matrix allows the team to rate each concept as
"better than," "same as," or "worse than" the other concepts in each of the selection criteria areas.
Assigning these ratings is facilitated by the establishment of a reference concept that becomes a
benchmark for comparison purposes. Summing the number of each rating that each concept was
awarded allows the team to rank order the concepts. At this point, the team must verify that the results
make sense and consider whether there might be ways to combine and/or improve the concepts. When
the team is satisfied with their understanding of each concept and its worth relative to the other
concepts, they can decide which concepts are to be selected for further refinement. In essence, the
process provides a vehicle through which the team can develop a clear sense of which concepts are
the most promising. The team assigns a weighting factor to each of the selection criteria (totaling
100%). At this point, Ulrich and Eppinger recommend using a fivepoint Likert scale that rates each
concept on each criterion as "much worse" than the reference criterion (rating of 1) to "much better"
than the reference criterion (rating of 5). These ratings are multiplied by the weighting factor for each
criterion and then summed to produce a total score for each concept. The totals are then compared to
produce a ranking of the concepts. Again, the team searches for opportunities to combine or improve
the concepts. Additional iterations of the process are completed until the team is ready to make a
selection of one or more concepts for product development. Finally, the team reflects upon the
selected concept(s) and on the selection process. This is the point of "no return" for the development
process. When the team is satisfied and solidly committed to the concepts selected, the development
process begins.
Pugh (1991) approaches the process of concept selection through "controlled convergence." Like
Ulrich and Eppinger (1995), this process employs a selection matrix. The matrix is an evaluation
process, much like Ulrich and Eppinger, that compares each concept on a set of established selection
criteria. Pugh describes the strengths of the method as support for convergent thinking as a reduction
in the number of concepts occurs, while also allowing divergent thinking as the reasoning process
supports the generation of new concepts. The essential feature of the process is the comparison of
each concept against a "peer" concept in such a manner as to avoid a fixed viewpoint on any one
concept. This "peer" concept is strongly reminiscent of the Ulrich and Eppinger "reference concept."
A peer concept is one that is considered the best; i.e., the one most likely to meet the constraints of
the product development system in the most effective manner.
Pugh's (1991) process has 15 steps in Phase 1:
1. Ensure that all ideas and embryonic solutions are generated against the background to the product
development systems.
2. Depict the solutions (concepts) in sketch form.
3 of 8
27/07/2000 16:26
EBSCOhost Full Display
wysiwyg://bodyframe.78/http://ehostvgw5...ilter=&dth=96&Forward.x=13&Forward.y=12
3. Establish a concept comparison and evaluation matrix.
4. Incorporate all the visual sketches of the concepts into the matrix.
5. Ensure that the comparison of the concepts is valid.
6. Choose criteria against which to evaluate the concepts.
7. Choose a datum with which all other concepts are to be compared.
8. Use a legend of "+", "-", or "S" to compare each concept to the datum.
9. Compare each concept on each criterion.
10. Assess the individual concept scores.
11. Look at the negatives of the strong concepts to see if they can be improved or eliminated;
introduce modified concepts into the matrix while leaving the original concept in the matrix.
12. Look at weak concepts and attack their negatives to see whether they can be improved upon;
introduce revised concepts into the matrix while leaving the original in place.
13. Eliminate truly weak concepts from the matrix.
14. If strong concepts do not emerge, investigate whether the criteria are ambiguous or whether
concepts are subsets of one another.
15. Rerun the matrix using the strongest concept from the last run as the datum.
Phase II of the process involves modification of the strongest concepts that emerged from Phase I in
preparation for their introduction into development.
In the first run, the concepts are evaluated in exactly the same fashion as described by Ulrich and
Eppinger (1995), i.e., ratings of "better than," "worse than," or the "same as" are awarded to each
concept for each criterion by the evaluation team. These ratings occur in the form of +, - or S. At this
point, the team examines each concept to determine whether the negative ratings can be improved or
eliminated. If changes can be made to accomplish this end, the new, revised concepts are entered into
the matrix for an additional iteration. Once this process has exhausted any improvement possibilities,
then the truly weak concepts are eliminated from the matrix. The end result of the process is the
emergence of one or more concepts that display a uniformity of strengths on the evaluation criteria.
This result is similar to the selection process described by Ulrich and Eppinger except that the
structured assignment of mathematical weights and scores is not part of the selection process. Pugh
(1991) relies on the iterative nature and constant examination of ratings to produce team consensus
with regard to the concepts to be selected for development. Ulrich and Eppinger describe the need for
"reflection" that will result in team consensus, while Pugh concentrates on team assessment and
reassessment of strengths and weaknesses to produce the same end.
The two methods are alike in that both employ a matrix of evaluation criteria versus design concepts.
Both employ a rating system with a reference or peer concept as the basis for determining whether a
given concept is strong, weak or indifferent with respect to each of the criteria. Both emphasize team
member convergence with respect to overall judgment of a concept as being one that should be
forwarded to development. Pugh (1991) emphasizes that new concepts can be introduced to the
evaluation process through changes that will correct or eliminate weaknesses. Ulrich and Eppinger
(1995) achieve the same end by stressing that concepts should be modified, when possible, to correct
or eliminate weaknesses.
4 of 8
27/07/2000 16:26
EBSCOhost Full Display
wysiwyg://bodyframe.78/http://ehostvgw5...ilter=&dth=96&Forward.x=13&Forward.y=12
The two methods differ in that Ulrich and Eppinger (1995) employ a structured, weighting process to
produce scores that can be ranked for the emerging concepts. Pugh (1991), on the other hand,
emphasizes a process that modifies or eliminates concepts from consideration in an iterative fashion.
Scoring, ranking or mathematical weights are not employed, as the focal point of the process is team
satisfaction with the elimination or further consideration of a concept in each iteration of the
evaluation. Further, under Pugh's approach, the standard for comparison, the "reference" or "peer"
concept changes to become the strongest concept that emerges from an iteration of the evaluation.
Strengths and Weaknesses of the Two Evaluation Methodologies
The Ulrich and Eppinger (1995) methodology provides users with a quantitative scoring and ranking
system whereas the Pugh (1991) approach does not. Pugh depends upon a subjective decision by the
team to eliminate a concept from consideration. There is little need for a scoring system when
differences between concepts are clear cut; however, this is not always the case. The ranking system
makes comparing concepts that have relatively minor differences more direct and allows team
members to reach consensus with regard to concepts that merit further consideration in a less
disruptive fashion. One of the primary objectives of the evaluation process is the commitment of the
team members to the development of the emerging concepts. This commitment is more attainable and
sustainable if the process of evaluating and choosing concepts is less dependent upon a champion or
an opponent for individual concepts.
Further, the Ulrich and Eppinger (1995) approach has a clear-cut line of demarcation between
screening and selection. This structure allows team members to spend less time on concepts that are
obviously less desirable or for which there is a clear-cut failure to satisfy criteria. At the same time,
the structure allows the team to spend more time on the emerging concepts; those that may well be
selected for development. In contrast, the Pugh (1991) approach requires a great deal of time to be
expended early in the process in attempting to improve weak concepts.
Essentially, the Pugh (1991) system involves changing the standard for comparison in the various
iterations of the evaluation process. The Ulrich and Eppinger (1995) approach maintains a constant
reference concept throughout the process. This permits the team to assess whether any of the concepts
under review meet the organization's criteria for development. By moving the reference point to the
strongest concept under review for subsequent iterations, Pugh implicitly assumes that at least one of
the concepts will be selected for development.
On the other hand, the Pugh (1991) system does provide greater support for generating alternative
concepts during the evaluation process. The evaluation emphasizes to a greater degree the necessity to
exhaust improvement or correction strategies for a concept before it is eliminated. This process could
result in the team recognizing viable alternatives for development that it might otherwise overlook.
Further, concepts that appear to be weak are quickly eliminated under the Ulrich and Eppinger (1995)
approach. The Pugh (1991) system supports a full examination of every concept before it is
eliminated from consideration. Consequently, there is a possibility for the Ulrich and Eppinger
process to eliminate a potentially viable concept.
Finally, the Pugh (1991) system explicitly recognizes that criteria might be ambiguous or that
concepts might be subsets of one another when the process fails to produce a clear decision point.
The Ulrich and Eppinger (1995) process has an opportunity for such a discovery during the
"reflection" phase; however, there is no explicit evaluation of these potential problems in evaluation.
The Human Factor
A viable project selection process must recognize the human factor. Since any evaluation system will
depend upon people to perform analyses and make decisions about concepts, it is important to
5 of 8
27/07/2000 16:26
EBSCOhost Full Display
wysiwyg://bodyframe.78/http://ehostvgw5...ilter=&dth=96&Forward.x=13&Forward.y=12
recognize how people process information. Proctor and Zandt (1994) describe human information
processing as consisting of three stages: the perceptual stage, the cognitive stage, and the action stage.
In the perceptual stage an individual receives information through the senses, then employs pattern
organization and recognition to determine what has been perceived. Decision-making occurs during
the cognitive stage and it proceeds from the thought and reflection concerning the patterns and what
one remembers about such patterns. Finally, in the action stage, an individual selects, prepares and
activates a response to the decision that has been reached. Clearly, any system of concept selection
should recognize how the information about various concepts will be perceived, how users of the
system will make decisions about the concepts, and how those users will act upon their decisions.
Woodson, Tillman and Tillman (1992) have a great deal to say about how a system should be
designed. They emphasize that the first step in any system design should be a determination of who
the users will be, what functions the users are to perform, individually and in conjunction with others.
They go on to describe the need to determine how the users will perform the functions in the system.
This insight suggests that a concept evaluation system should be designed from the perspective of the
people who will actually be performing the evaluation and should explicitly consider how those
people will use the system.
Human-centered design stresses a top-down approach (Rouse, 1991; Salvendy, 1992). One must
establish goals for a concept, define the form and functions, describe how it should perform, and
determine objectives for the concept that define the activities that must be accomplished. Rouse
stresses the need for compatibility between the physical nature of the presentations to a user and the
responses expected from the user; understandability or meaningful communication between a user
and a system; and, effectiveness, or the extent to which a system leads to improved performance.
The human factors make it clear that the manner in which concepts are presented to an evaluation
team will have a major impact upon the outcome of the evaluation. This is a direct result of the
perceptual stage of information processing that consists of pattern matching. Users of an evaluation
system are likely to see each concept in terms of concepts that have been previously approved.
Concepts that are totally new are likely to suffer during the perception stage of human information
processing and yet these are the very concepts that have the potential to change the industry. At the
same time, a totally new concept has a higher level of project risk in that we cannot predict the
market's reaction.
Product development project managers think of "what might be" and try to "optimize design,
schedules, risk, and even manufacturing processes," while designers focus on "what is" and attempt to
"satisfy" in terms of incremental improvement. Rouse (1991) goes on to say that concept design tends
to be constraints-driven and their concerns will be dominated by budgets, credibility, inertia in terms
of previous solutions and vested interests, and, above all, project schedules. Project managers tend to
view a project evaluation as an opportunity to assess the validity of a concept, while designers tend to
look at evaluation as a requirement to assure the acceptability of a concept.
Conclusion
Differences in perception have a tremendous influence on innovative concept evaluation and
selection. Since perception is largely based on pattern matching (Proctor & Zandt, 1994), users of an
evaluation system will be moved toward decision and choice in a positive manner when a concept
matches the established patterns in an evaluator's mind. There is nothing wrong with this approach
when design concepts under review do represent incremental improvements over established
products. In fact, such a paradigm will lead to a rapid and effective evaluation of such concepts
because the team will understand the concepts well. For that reason, we propose a two-pronged
evaluation process: concepts should be divided into two groups prior to evaluation, each group to be
handled differently.
First, the team will examine each concept to determine whether it represents an incremental advance
6 of 8
27/07/2000 16:26
EBSCOhost Full Display
wysiwyg://bodyframe.78/http://ehostvgw5...ilter=&dth=96&Forward.x=13&Forward.y=12
or an entirely new idea. Those concepts that represent incremental advances will proceed through
screening and selection in accordance with the Ulrich and Eppinger (1995) methodology. In fact, the
only real criticism of Ulrich and Eppinger that is valid is that the process represents convergent
thinking as opposed to divergent thinking. The former works well with incremental change, but the
latter is required for truly new concepts and products.
Traditional thinking and observation leads one to conclude that the vast majority of concepts will fall
into the incremental change category. Separating the concepts initially will then lead to a much
smaller pool of ideas that need to be evaluated under a different approach. The question is how to
evaluate those few concepts. Because of the pattern matching that takes place in human information
processing (Proctor & Zandt, 1994), the criteria established for evaluation are likely to foreclose
totally new ideas from concepts without establishing a matrix of evaluation criteria. In fact, the sole
criteria will be exploring the changes in the organization and its structure that would be required to
make the concept feasible. After that process, the team will address whether those changes can be
accomplished within the organization. That is, does the organization have or can it acquire the people,
tools, capital, etc., that are necessary? Then, the team must examine whether the organization is
willing to make those changes. Will the changes move the company away from its established goals
and objectives? Are these new directions desirable? Do we have the top management commitment
and support that will make the change effective?
Decisions of this type require input from the highest levels of the organization. Only top management
support can make organizational change effective. Consequently, the new evaluation team must
include top management members and have their support in the evaluation process. The entire
organization will be involved in change, which mandates interest and involvement. The two-pronged
evaluation system would make the best use of both convergent and divergent thinking while
supporting the organization in a real attempt to become innovative, creative, entrepreneurial, and
forward-looking.
Indeed, the role of innovation begins with a belief in the value of creativity and original ideas. Armed
with a comprehensive grasp of fundamentals from the project management body of knowledge,
typically PMPs and other management professionals can design processes that embody innovation
geared for new product development success.
PHOTO (BLACK & WHITE): Steven Pascale
PHOTO (BLACK & WHITE): James Carland
PHOTO (BLACK & WHITE): JoAnn Carland
References
Proctor, R., & Zandt, T. (1994). Human factors in simple and complex systems. Boston: Allyn and
Bacon.
Pugh, S. (1991). Total design. Wokinham, England: Addison-Wesley Publishing Company.
Rouse, W. (1991). Design for success. New York: John Wiley & Sons, Inc.
Salvendy, G. (1992). Handbook of industrial engineering, 2nd edition. New York: John Wiley &
Sons, Inc.
Ulrich, K., & Eppinger, S. (1995). Product design and development. New York: McGraw-Hill, Inc.
Woodson, W., Tillman, B., & Tillman, P. (1992). Human factors design handbook, 2nd edition. New
York: McGraw-Hill, Inc.
7 of 8
27/07/2000 16:26
EBSCOhost Full Display
wysiwyg://bodyframe.78/http://ehostvgw5...ilter=&dth=96&Forward.x=13&Forward.y=12
~~~~~~~~
By Steven Pascale, 1770 Wedgewood Commons, Concord, Massachusetts 01742, James W. Carland,
Western Carolina University, 105 Foresyth, Cullowhee, North Carolina 28723 and JoAnn Co
Carland, Western Carolina University, 105 Foresyth, Cullowhee, North Carolina 28723
Steven Pascale, PMP, is a business operations manager at the Digital Equipment Corporation
corporate headquarters. He has 10 years of experience in new product development of projects.
Previously he was a program manager on a variety of projects at AT&T (GIS). He received his B.S. in
aeronautics from California State University in San Jose. He also holds an M.B.A. and M.S. in
industrial engineering from Clemson University, and an M.S. in project management from Western
Carolina University
James Carland, Ph.D., received a B.S. in mathematics from the University of North Carolina at
Asheville, a Ph.D. in management policy from the University of Georgia and an M.S. in business
from Western Carolina University. He is a certified management accountant and a certified public
accountant. He has 10 years of business experience, and has been involved in the approval process for
loans to small businesses as well as the preparation and approval of SBA loans
JoAnn Carland holds a Ph.D. in education psychology from the University of Georgia, a B.A. in
French and Latin from Meredith College, and an M.S. in education from Western Carolina
University. She is a certified data processing professional and has five years of business experience.
Copyright of Project Management Journal is the property of Project Management Institute and its content may not be
copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission.
However, users may print, download, or email articles for individual use.
Source: Project Management Journal, Dec97, Vol. 28 Issue 4, p46, 6p, 3bw.
Item Number: 135898
Result 80 of 96
8 of 8
[Go To Full Text]
[Tips]
27/07/2000 16:26