Issues and challenges of Agile
Development: Some perspectives
from Academic Research
Dr. Sridhar Nerur
University of Texas at Arlington
1
Presentation Outline
Are Two Heads Better Than One – Pair
Programming Study
 Pairs versus Individuals on Design Tasks: A
Distributed Cognition Perspective
 Case study of XP (Extreme Programming)
adoption
 Other research projects

2
Are Two Heads Better Than One?
A Pair Programming Study (Balijepally,
Mahapatra, Nerur & Price, 2009)





Methodology development driven by
practitioners
Effectiveness reports were anecdotal or
experiential
No systematic effort to provide theoretical
explanation of practices
Inconsistent findings on the effectiveness of pair
programming
Lack of theoretical and methodological rigor
(e.g., Nosek 1998, Williams 2000)
3
Effectiveness of Pair Programming

Research Objectives
◦ To examine the efficacy of pairs vis-à-vis
individuals in a programming task
◦ To study the effect of pairing on a
programmer’s task satisfaction and confidence
in performance
◦ To understand how the above are affected by
programming task complexity
4
Theoretical Background

Task Typology
◦ Based on Steiner’s typology (1972)
programming tasks are unitary (not divisible
among group members), optimizing (quality is
more important than speed), and disjunctive
(group must arrive at a single solution)
◦ Tasks may be intellective or judgmental (
Laughlin and Ellis 1986)
◦ Programming tasks are relatively intellective
with high solution demonstrability
5
Theoretical Background

Nominal Pair (Laughlin et al. 2002)
◦ A nominal pair is created by randomly pairing
individuals who worked alone
◦ Comparing the performance of a
collaborating pair with the best and the second
best members of a nominal pair provides
better insight into group performance
6
Theoretical Background

Group Performance
◦ Group members have access to a larger pool of
resources (Flor and Hutchins 1991; Hinsz et al.
1997)
◦ Group processes may suffer from process losses
due to social loafing (Karau and Williams 1982)
◦ Performance is contingent on task type
◦ With intellective tasks, groups outperform an
average individual but rarely perform better than
the best individual (Hill 1982, Laughlin et al. 2003)
◦ Quest for assembly bonus effect (when the group
outperforms the best individual) still continues
(Collins and Guetzkow 1964)
7
Research Model
Task
Complexity
H4
Programming
Setting
H1
Software quality
Individuals or Pair*
Perceptual Outcomes
Satisfaction
H2
H3
*1 – Best in a nominal pair
Confidence in
Performance
2 – Second-best in a nominal pair
3 – Collaborating pair
8
Research Method and Data Analysis







Lab Experiment
2 (Individual vs. pair) X 2(Simple Vs. Complex
task) factorial design
120 student subjects (30 pairs and 60 individuals)
Pilot done to check experimental protocol and to
determine time needed for the experimental task
15 minutes for the warm up task and 2 hours for
the experimental task
MANCOVA followed by ANCOVA for each
dependent measure
Programming ability used as a covariate
9
Results
1
2
Hypothesis
Software Quality
Programming performance of a
collaborating pair measured in
terms of software quality will be
higher than the performance of
the second-best member of a
nominal pair
Satisfaction
A. The mean satisfaction with
the programming task of a
collaborating pair will exceed
the level reported by the best
member of a nominal pair.
B. The mean satisfaction with
the programming task of a
collaborating pair will exceed
the level reported by the secondbest member of a nominal pair.
Result
Explanation
Supported
(p < 0.01)
A collaborating pair
outperforms the second-best
member of an independently
working nominal pair.
Supported
(p=0.04)
Collaborating pairs are more
satisfied than each member of
an independently working
nominal pair.
Supported
(p = 0.02)
10
Results
Hypothesis
Result
Explanation
Confidence in Performance
A. The mean confidence in
performance of a collaborating
programming pair will exceed the
level reported by the best member
of a nominal pair.
B. The mean confidence in
performance of a collaborating
programming pair will exceed the
level reported by the second-best
member of a nominal pair.
Moderating Effect of Task
Complexity
Task complexity affects the
performance, in terms of software
quality, of the best member of a
nominal pair more adversely than
that of a collaborating pair
Not
Supported
(p = 0.15)
Supported
(p < 0.01)
Not
Supported
(p = 0.18)
Collaborating pairs are only as
confident of their performance
as the best member of an
independently working nominal
pair, but more confident than its
second-best member.
Task complexity affects the
performance of collaborating
pairs and nominal pairs in
similar ways.
11
Contributions




One of the earliest theoretically grounded
empirical research studies on pair
programming.
Offers insight into performance of
programming pairs
Examines the contingent effect of task
complexity on programmer’s performance
Creates a new benchmark to evaluate pair
programming performance through the use
of nominal pairs
12
Pairs vs Individuals on Design Tasks
(Mangalaraj, Nerur, Mahapatra, &
Price – under review)

Motivation
◦ Research on pair designing is sparse
◦ Design tasks are cognitively more challenging
◦ Very limited empirical research on design
patterns in tasks that are design-centric
13
Theoretical Foundation
According to the theory of distributed
cognition, cognition is distributed across
problem-solvers (i.e., social interactions),
artifacts (i.e., embodied cognition), and
time
 Two aspects of distributed cognition that
are relevant to our research: social and
structural (or embodied)
 We use pairs (social) and design patterns
(cognitive artifacts) in our study

14
Research Objectives
To study the role of design patterns in
facilitating the solution of a software
design problem
 To assess the effectiveness of pairs vs.
individuals working on a software design
task

15
Theoretical Background
Design problems are ill structured, ambiguous,
and have no pre specified solution path (Simon
1973)
 Key activities in solving a design problem are

◦ Problem structuring
◦ Searching through the solution space
Problem structuring is facilitated by availability of
partial solutions and external representation
media (Guindon 1990, Zhang and Norman 1994)
 Expert developers use their design experience
(stored as design schemas) to constrain the
search space (Guindon et al. 1987)

16
Research Model
Individual/Pair
H4, H5a, H5b,
H6
Outcome
Variables
Solution quality
Task completion
time
With design
patterns/without
design patterns
Task satisfaction
H1, H2, H3
17
Research Method
Individual
Patterns
No patterns
Condition I
Condition II
n = 24
n = 23
12 nominal pairs
11 nominal pairs
Condition III
Condition IV
n = 24
n = 23
12 collaborating
11 collaborating
pairs
pairs
Pair
Total sample size (N) = 92
Two-by-Two Factorial Design
18
Research Method & Data Analysis







Lab Experiment
2 (Individual vs. pair) X 2 (Design Pattern Vs. No
Design Pattern) factorial design
92 professionals with no prior pattern experience
Pilot done to check experimental protocol and to
determine time needed for the experimental task
Participants were provided a free seminar on
patterns
ANOVA/ANCOVA used for analysis
Object-oriented programming experience was
used as a covariate
19
Results
Hypothesis
H1: Quality of the design solution will be higher when design patterns are
used during software design.
H2: Time taken to complete a design task will be shorter when design
patterns are used.
Result
Supported
Supported
H3: Task satisfaction will be higher when design patterns are used in a
software design task.
Supported
H4: Solution quality of collaborating pairs will be higher than that of the
second best member of a nominal pair in a software design task.
Supported
H5a: Time taken by collaborating pairs to complete a software design task will
Not Supported
be longer than that of the best member of a nominal pair.
H5b: Time taken by collaborating pairs to complete a software design task will Supported
be longer than that of the second-best member of a nominal pair.
H6: Task satisfaction will be higher among collaborating pairs when
compared with the average level of task satisfaction of nominal pairs.
Supported
20
Contributions
One of the earliest theoretically grounded
empirical research on the impact of design
patterns on design performance
 Demonstrates the benefits of using design
patterns

◦ Improves design quality
◦ Reduces task completion time
◦ Enhances developer satisfaction

Consistent with group literature, pairs were
found to be better than second-best individuals of
nominal pairs
21
Synthesis
Theoretically grounded research provides a
deeper understanding of the performance of pairs
in software development
 Studies demonstrate that pairs outperform
second-best individuals of nominal pairs
regardless of task type and experience level.
 This confirms the long-standing research in group
theory, but is counter to the dominant belief in
the agile community
 The use of nominal groups in this stream of
research is a novelty. It permits a more finegrained analysis that is more meaningful to
practitioners

22
Implications

Pairing should be used judiciously
◦ Performance gains are realized when lowperforming rather than best developers are
paired
The use of design patterns is beneficial
 Our studies have established a benchmark
for theoretically grounded research in
software engineering

23
Case Study – Problems with Adoption of
XP practices at ABC Corp.
Table 1: Core XP practices adopted by ABC Corp.
XP Practice Definition*
Release
Planning
Part of the planning game (original XP practice) in
which the customer specifies the requirements for the
project and creates the release plan based on the
developer’s cost/time estimates.
Customer Customer is part of the development team. He/she
Team
determines requirements and performs acceptance tests.
Collective Members of the project team collectively own the code
Ownership and any one can change any of the code at any time.
Pair
Pair programming involves two programmers working
Programmi side by side at one computer, collaborating on the
ng
design, coding and testing on a continuous basis.
Iteration
A part of the planning game that involves planning for
Planning
the two-week iterations.
Sustainable Teams work at a sustainable pace, usually 40 hour per
Pace
week. Team members work overtime only when needed.
Relative
Importance
0.139
0.126
0.122
0.109
0.100
0.098
Refactoring
Ongoing redesign of software to improve its responsiveness
to change.
Simple Design Simplest design is done for the functionality that has been
defined and not in anticipation of future requirements.
Unit Tests
0.049
0.046
Small
Releases
Involves the development of the test module prior to writing 0.040
the code.
Software is released in small increments that address the most 0.040
valuable business requirements.
Acceptance
Tests
Continuous
Integration
Customer writes the acceptance tests. Testing is largely
automated.
Builds are done every few hours to lessen code integration
problems.
0.037
Metaphor
It is the common vision of how the system works. The
metaphor is intended to provide a broad view of the project’s
goal.
Developers work in an open lab-like environment with no
cubicles.
Rules that everyone in the team must follow in writing code.
0.023
0.028
Open
0.022
Workspace
Coding
0.020
Standards
* These definitions are based on the following sources: (Highsmith, 2002; Beck, 2000;
Jeffries, 2001)
Table 2: Comparison of the two projects
Characteristics
Project X
Application type
New project
Language/Technolog Java
ies
Project size
Approximately 30,000
lines of code.
Tools used
IntelliJ and
CruiseControl
Team Size
6 members
XP Score (Maximum 2.64
4.0)
Project Y
Maintenance and enhancement
C, C++, Java, Motif
Several hundred thousand
lines, with more than 10 year
old code base.
C++ editor, SlickEdit, and
ClearCase
7 members
1.58
Examples of differences between the two projects
Table 3: Comparison of core XP practices across the two projects.
XP Practice Project X
Project Y
Release and
Release planning is done
Release planning is done
iteration
periodically.
periodically.
planning
Iteration planning involves all
Though iteration planning involves
Customer
team
Collective
ownership
members of the team.
Developers have the freedom to
choose stories they wish to
work on.
Proxy customer writes user
stories and collaborates in
acceptance testing.
Team members are collectively
responsible for code
development and maintenance.
It is done to a greater extent.
Pairprogramming In general, team members
seemed to enjoy pairprogramming and they hold
positive attitude towards it.
all members of the team, stories are
assigned based on skill
specialization.
No dedicated customer/proxy
customer is involved.
Limited collective code ownership
due to skill specialization.
Pair programming was seldom done.
Individual Factors
•Attitude towards XP
•Knowledge of XP
Team Factors
•Task management
•Team leadership
Technology Factors
•Compatibility
•Tools support
Acceptance of Extreme
Programming
Task Factors
•Type of application
•Project size
Environmental Factors
•Budget constraints
•Time constraints
Figure 3. Extreme Programming
Acceptance Framework
Other research studies
Efficacy of pairs vs individuals with and
without Test-Driven Development
 Impact of problem representation on
solution quality
 Understanding cognitive processes –
mental models, cognitive ability
 Agile Project Management challenges

29
Questions??
30