Agile Software Development: Evidence from the Field
AGILE SOFTWARE DEVELOPMENT:
EVIDENCE FROM THE FIELD
Alan MacCormack
(amaccormack@hbs.edu)
Harvard Business School
Agile Development Conference
June 2003
©Alan MacCormack 2003
1
Agile Software Development: Evidence from the Field
Introduction
• Research Background: Product Development Flexibility
– Multi-year, multi-industry research project
– Common Theme: Highly uncertain and dynamic environments (tech + mkt)
– Problem: Significant changes occur in the needs a product must address and the
technologies it employs to meet these needs during a development cycle
• Multiple Studies in the Software Industry
– 30 Internet Software Projects in 96-98 timeframe (extreme uncertainty)
– 30 HP Projects in 98-00 timeframe – broaden context of study
– 100+ Projects in 2002 timeframe – analysis of data ongoing
• This Presentation
– Detail the major themes that emerge in the Software Studies
– Explode some myths about more flexible practices
– Communicate why it is HARD to do….PROPERLY!
©Alan MacCormack 2003
2
Agile Software Development: Evidence from the Field
Problems with Traditional Processes
E.g. A Stage-Gate Model of Development
Concept design
Product and feature specifications
Coding
Testing
Response Time
START
DESIGN
FROZEN
END
Problem: When the market/technology advances at a faster pace than you can respond
©Alan MacCormack 2003
3
Agile Software Development: Evidence from the Field
Achieving Flexibility
Stable Environments:
A Stage-Gate Process
Uncertain Environments:
A More Flexible Process
Note: “Speed” is a subtle concept in a more flexible process
©Alan MacCormack 2003
4
Agile Software Development: Evidence from the Field
A More Flexible “Model” of Development
First
Integration
Concept
Frozen
New Information
CONCEPT DEVELOPMENT
DETAILED DESIGN
SYSTEM-LEVEL TEST
Feedback on Performance
•
Characteristics of such a process
– Overlapping stages (Krishnan, Eppinger & Whitney, 1997)
– Iterative process (Synch and Stabilize”, Cusumano, 1995; “Experiential”, Eisenhardt and
Tabrizi, 1995; “Sense, Test and Integrate”, Iansiti and MacCormack, 1997)
– Learning and adaptation (Tushman and O’Reilly, 1997) versus planning and execution
©Alan MacCormack 2003
5
Agile Software Development: Evidence from the Field
An Early Case Study
Netscape’s Navigator 3.0 Project
The point: >50% of the new code was developed after the first beta release!
©Alan MacCormack 2003
6
Agile Software Development: Evidence from the Field
The Aim: Integrated Technology and
Market Streams
EVOLUTION OF TECHNICAL POS S IBILITIES
Initial Input
BETA 1
BETA 2
BETA 3
Product
Release
EVOLUTION OF CUS TOMER PREFERENCES
Note: A flexible process results in “better’ performing products as perceived by the
customer (i.e., there may still be trade-offs with other performance dimensions)
©Alan MacCormack 2003
7
Agile Software Development: Evidence from the Field
Research in Internet Software
• Internet software industry
– Industry “created” in 1993, with development of GUI to the internet
– Tremendous uncertainty during period of study (96-98)
– Thousands of new firms (e.g. Netscape, Yahoo!), hundreds of new applications (e.g.
Internet telephony), variety of alternative technologies (Java, ActiveX)
• Survey data on 31 projects from 17 different firms
– Projects include Microsoft Explorer 3.0 and 4.0, Netscape Navigator 3.0 and 4.0, My
Yahoo!, Intuit QuickenMortgage, 411 Rocketmail, Planetall, Altavista Intranet
– Average size of project: 375,000 lines of code (largest project = 1.5mn lines)
• Data on product performance
– Product Quality: Panel of experts rated features, technical performance, and reliability
– Productivity: Resources consumed by project adjusted for size and complexity
©Alan MacCormack 2003
8
Agile Software Development: Evidence from the Field
Success Factors - Early Release to Customers
Quality vs % Product Functionality at First Beta
6
5.5
Quality
5
4.5
4
3.5
3
30
40
50
60
70
80
90
100
% Product Functionality at First Beta
©Alan MacCormack 2003
9
Agile Software Development: Evidence from the Field
Not Just Multiple Beta Versions…
Quality vs Number of Betas
6
Quality
5.5
5
4.5
4
3.5
0
1
2
3
4
5
6
Number of Betas
©Alan MacCormack 2003
10
Agile Software Development: Evidence from the Field
A Note on Results of the Beta Analysis
•
Functionality at First Beta is a Better Predictor than Functionality at First
System Integration
– Naturally, these two measures are highly correlated
– Suggests projects which integrate early but do not release this version of the product to
the market fail to capture all the benefits from a more flexible approach
– Interpretation: The dominant source of uncertainty in this emerging industry is related
to the market, not to the underlying technologies (software code)
•
Field Work Suggests the Number of Discrete Betas is not as Important as the
Nature of the Interactions with Users
– Wide variation in beta distribution policy - wide versus narrow
– Some firms are much more careful about which users are selected for early versions,
and how they work with these users once beta testing begins
– Having excessive numbers of betas creates problems in version control
©Alan MacCormack 2003
11
Agile Software Development: Evidence from the Field
Deciding on Beta Strategy
HI
My Yahoo!
NetDynamics
LO
Microsoft
Netscape
RISK OF
EXTERNAL
TESTING
(Competitive imitation,
impact of buggy product,
support requirements, etc.)
LO
HI
NEED FOR EXTERNAL TESTING
(Availability of internal test resources, are employees = users,
sophistication of application, etc.)
©Alan MacCormack 2003
12
Agile Software Development: Evidence from the Field
Success Factors - Rapid Feedback
Quality vs Feedback Time (hours)
6
Quality
5.5
5
4.5
4
3.5
0
10
20
30
40
50
60
70
80
Feedback Time (hours)
Note: How you “configure” experimentation capacity is vitally important
©Alan MacCormack 2003
13
Agile Software Development: Evidence from the Field
The Value of Experience
• In dynamic environments, the value of
Experience is questioned (at least in academia!)
– Knowledge atrophies quickly
– Inertia leads to applying old or inappropriate skills
• What type of Experience is valuable?
– Not Experience measured by “years of tenure”
– Experience measured by number of project “generations”
• So how does it work
– Setting direction for experimentation “strategy”
– Learning at the “System” level
©Alan MacCormack 2003
14
Agile Software Development: Evidence from the Field
Success Factors - Architectural Design
• Program Manager for Microsoft Internet Explorer 3.0
– ...the most important aspect of the project was that we developed the product
architecture in a way that separate component teams could feed into the project. The
idea was to build a good core infrastructure, and have the rest of the team add
components on top of it. In fact, at the first integration, all we had was the core
infrastructure. Most other features were missing. We didn’t have any support for Java
at this point because the license for Java had only been signed a few days before...
• Chief of Development for Altavista
– ...our architectural design efforts are structured to give priority not to performance, but
to independence. We create interfaces to buffer the impact of uncertainty – when one
module changes, the others are therefore insulated. If we were trying to optimize the
size and efficiency [of a design] we would not do this, but optimizing a design typically
makes it more complex and subsequently very difficult to change...
©Alan MacCormack 2003
15
Agile Software Development: Evidence from the Field
A Modular and Scaleable System: Linux
Evolution of the Linux Kernel
Year
Version
1991
Lines of Code
Users
0.01
10,000
1
1992
0.96
40,000
1,000
1993
0.99
100,000
20,000
1994
Linux 1.0
170,000
100,000
1995
Linux 1.2
250,000
500,000
1996
Linux 2.0
400,000
1,500,000
1997
Linux 2.1
800,000
3,500,000
1998
Linux 2.1.110
1,500,000
7,500,000
Source: Forbes Magazine, August 10th 1998
©Alan MacCormack 2003
16
Agile Software Development: Evidence from the Field
Flexible Processes in Action
Microsoft’s Milestone Build Process
Vision statement
Overall product specification
Detailed feature specification
Coding
Testing
feature set 1
feature set 2
feature set 3
Stabilization
Source: Microsoft Office 2000, HBS Case, MacCormack 1999
©Alan MacCormack 2003
17
Agile Software Development: Evidence from the Field
Extending this Process
The Evolutionary Delivery Model: First Stage = Process Design
Vision
Architecture design and overall specification
Micro-project
Micro-project
Micro-project
Feature design
Coding
Integration/Test
User Feedback
User Feedback
User Feedback
Stabilization
Questions: How many Micro-projects? On what parameters should this decision depend?
(Current research, with Mike Cusumano, Chris Kemerer)
©Alan MacCormack 2003
18
Agile Software Development: Evidence from the Field
Exploding some Myths about Flexibility…
©Alan MacCormack 2003
19
Agile Software Development: Evidence from the Field
Myth 1: It’s a new “Best” practice…
Low Mkt Uncertainty (MAD<0.49)
1.50
1.50
1.00
1.00
Product Quality
Product Quality
High Mkt Uncertainty (MAD>0.49)
0.50
0.00
0.00
-0.50
-0.50
-1.00
-40
0.50
-20
0
20
% Functionality at Beta
40
-1.00
-40
-20
0
20
40
%Functionality at Beta
Note: Very divergent responses when projects faced with high uncertainty
©Alan MacCormack 2003
20
Agile Software Development: Evidence from the Field
Matching Process and Context
Market
Uncertainty
Technical
Uncertainty
PROCESS
CHOICES
Platform
Complexity
Platform
Uncertainty
Other Relevant
Parameters
©Alan MacCormack 2003
21
Agile Software Development: Evidence from the Field
Myth 2: It’s all about making late changes…
Market Uncertainty and Late Changes to a Module
Last Change to Module
1
0.9
0.8
0.7
0.6
0.5
0.4
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Market Uncertainty (MAD)
Relationship is significant with p<5%
©Alan MacCormack 2003
22
Agile Software Development: Evidence from the Field
Myth 3: You trade-off quality, productivity…
STUDY OF 30 HP PROJECTS
Defect Rate
Productivity
34.9****
(4.20)
INITIAL RESULTS
• More complete functional specification is
correlated with higher productivity
• More complete design specification is
correlated with lower defect rate
• Lends support to a “Waterfall-style”
model of development
Constant
16.36
(1.28)
Systems
14.87**
(2.75)
%FuncProto
0.48***
(3.71)
RegressionTest
-12.64 **
(-2.18)
FINAL RESULTS
• Trade-offs disappear when you account
for other development practices that
provide an alternative mechanism for
providing information that a spec
provides (e.g., prototype vs func spec)
DesignReview
-19.65 **
(-2.33)
R-squared (adj)
74.6%
52.8%
F-ratio
15
11.1
Df
15
16
©Alan MacCormack 2003
DailyBuilds
-0.42***
(-2.91)
16.89**
(2.35)
23
Agile Software Development: Evidence from the Field
Myth 4: We’re way ahead of the world…
India
Japan
USA
Europe&Other
Total
24
27
31
22
104
Projects
Arch. specs
%Yes
83.3%
70.4%
54.8%
72.7%
69.2%
Functional specs
%Yes
95.8%
92.6%
74.2%
81.8%
85.6%
Detailed designs
%Yes
100%
85.2%
32.3%
68.2%
69.2%
Subcycles
%Yes
79.2%
44.4%
54.8%
86.4%
64.4%
More than 1 Beta
Release
% Yes
66.7%
66.7%
77.4%
81.8%
73.1%
Pair Tester
%Yes
54.2%
44.4%
35.5%
31.8%
41.3%
Pair Programming
%Yes
58.3%
22.2%
35.5%
27.2%
35.3%
Daily builds
%Beginning
%Middle
%End
16.7%
12.5%
29.2%
22.2%
25.9%
37%
35.5%
29.0%
35.5%
9.1%
27.3%
40.9%
22.1%
24%
35.6%
%Yes
91.7%
96.3%
71.0%
77.3%
83.7%
Regression test on
each build
Source: Survey of 104 completed projects (see Cusumano et al, forthcoming, IEEE Software)
©Alan MacCormack 2003
24
Agile Software Development: Evidence from the Field
The BIG Managerial Challenge
• A flexible process requires a change in mindset
– Much perceived wisdom about “good” practice runs counter to a flexible process, e.g. a
large number of design changes late in a process is bad - DEFINITIVELY NOT TRUE
– Key is a process which allows you to pro-actively generate new information and
respond to this by evolving the design in a controlled fashion - otherwise, it’s chaos…
• An example from the field:
– One firm volunteered data on a “good” and a “bad” project
– The “bad” project had higher quality and took less resources to complete
– “Bad” project involved a process of continual change, responding to market,
competition, etc. - final result didn’t look anything like the original specification
– “Good” project ran in a structured fashion, highly optimized design, difficult to change
- final result closely resembled the specification (design had been “frozen in time”)
The point: Which looks like the better process to a senior manager?
©Alan MacCormack 2003
25
Agile Software Development: Evidence from the Field
Want to Learn More?
•
CONCEPTS: Developing Products on Internet Time, with Marco Iansiti,
Harvard Business Review, Sep-Oct 1997.
•
EVIDENCE: Product Development Practices that Work: How Internet
Companies Develop Software, Sloan Management Review, Jan 2001.
•
CAVEATS: Exploring Trade-Offs between Productivity and Quality in the
Selection of Software Development Practices, Forthcoming, IEEE Software.
•
HOW TO: Harvard Business School Case Studies
– Microsoft Office 2000
– Living on Internet Time
•
As well as all the usual books and articles on agile, adaptive, XP etc.
©Alan MacCormack 2003
26