Information Systems
Overview
Organizing Complex Projects
Around Risks Arising From
System Dynamic Behavior
Neil Siegel
Sector Vice-President & Chief Engineer
Northrop Grumman
9 March 2011
Agenda
• Scope
• Hypothesis
• The design-based technique
• Research results
• Summary
2
Scope of proposed study
This study aims to assess the efficacy of one
particular method of improving the outcomes on
complicated, large-scale, software-intensive
system development projects
3
“Bottom-line up front”
Systems are
important to
society . . .
Hypothesis:
centralizing
control of
system dynamic
behavior while
applying critical
skills will lead to
superior results
. . . but system
development
efforts often
fail
Assess in 1
domain
Assess in 4
additional
domains
4
Select scope /
characteristics
of systems of
interest
Formulate
conclusions
Systems are important to society,
but many system development efforts fail
• What do we mean by “fail”:
– Do not produce a product that meets the original specifications
– Produce such a product only after taking significantly more time
and/or money than originally expected.
– In the extreme, many such projects are cancelled before
completion
• Failure is apparently common:
– (Glass 2001) cites data indicating that only about 16% of the
system development projects that he surveyed were listed as
successful by their own developers
– (Johnson 2006) cites data from the Standish Group, which
describes results from a 2004 survey: just 29% of development
projects succeeded
5
Scope of the systems of interest
• Complex emergent behavior, as described by (Rechtin
1991)
• Interactions with physical devices (physically-moving
mechanisms, other time-sensitive mechanical devices,
etc.)
• Stressing asynchronous stimuli (such as extraordinary
high data-ingest rates, or highly-stressed
communications structures)
• Extraordinarily high availability / reliability requirements
• Development efforts of large size
• Systems that need to display much early progress
through prototyping and re-use
6
Hypotheses
During the development phase of a largescale, complex computer-based system, the
use of a specific design-based technique that
centralizes the control of the dynamic
behavior of a system will lower the density of
those defects that are attributable to
unplanned adverse dynamic system behavior
7
Partitioning the work
• I propose using the design process to
partition the work into different “skill bins”,
so as to provide better ways to assign
people to tasks.
– This allows a particular difficult task (control and
management of the system’s dynamic behavior) to be
partitioned away from most of the development team.
• Under current methods, the “hard” parts of
the work can be disseminated into a large
portion of the tasks
8
How I accomplish that partitioning
The System Architecture Skeleton
Describe system’s desired dynamic behavior, to the level of every independently-schedulable entity
Take a design decision to centralize control of system dynamic behavior into a small portion of the
design and implementation (“control kernel”)
Recognize every mechanism available to a developer that could create concurrency and other forms
of dynamic behavior
Designate a small team with suitable expertise as the implementers of the control kernel
Train the developers outside of those implementing the control kernel that they are not to use these
mechanisms, and create ways to enforce that proscription
Provide a mechanism for specification & implementation of the threads and allowable
interconnects / interactions amongst the components
Provide a mechanism that allows for a small portion of the implementation to specify and control the
dynamic behavior of the system, e.g., implement the control kernel
Provide for a mechanism that supports the instrumented exercising and analysis of the threads under
realistic stimuli, and do so in a way that allows the dynamic behavior to be observed and adjusted
separately from the specific functionality of the system, and can do so even prior to the availability of
actual system components
Allow for the entities within the system that are designed and implemented without reference to
dynamic behavior in some fashion to “inherit” a set of behavioral controls in accordance with the
parameters and algorithms implemented within the control kernel
Automatically translate the specification of the threads and allowable interconnects / interactions
amongst the components (whether hardware or software or people) into an executable mechanism
9
Results
10
All 6 periods / all 4 cases
Did not use the
design-based
technique
30
25
20
Project YYYY period II
Project AAAA
15
Project BBBB
Project YYYY period I
Project YYYY period IIII
10
Project ZZZZ
5
0
1
2
3
4
5
6
7
8
9
11
10
11
Did use the
design-based
technique
Defect density
“Use of the design-based technique will reduce the density
of defects attributable to errors in managing system dynamic behavior”
Density – opened this month / 1M SLOC
YYYY,
I-IIIplot
contractor
Periods I-IIIProject
contractor
testperiods
on a single
– FBCB2 –tests,
problem reports
attributable toattributable
unplanned dynamic
system
behavior
– opened per month
problem
reports
by month.
(adjusted for peer-review and unit-test procedure changes)
(time is discontinuous between periods)
25
20
Monthly results
UNPL
15
LNPL
2-s upper
10
2-s lower
1-s upper
5
1-s lower
average
0
1
2
3
-5
4
5
6
7
8
1998
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
2007
Initial version 31 December 2010
This version 31 December 2010
2009
7
6
5
4
MR
3
MR limit
2
1
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Period II behavior is materially different – o(4x) more problem reports attributable to this cause
12
Cost performance also tracks
the dependent variable
Actual cost per month / budget per month
Periods I-III contractor test on a single plot – FBCB2 –
actual cost per month divided by budgeted cost per month
(time is discontinuous between periods)
1.3
1.25
1.2
Monthly results
1.15
UNPL
LNPL
1.1
2-s upper
1.05
2-s lower
1-s upper
1
1-s lower
0.95
average
Initial version 2 February 2011
This version 2 February 2011
0.9
0.85
1
2
1998
3
4
5
6
7
8
2007
2009
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
13
Summary & interpretations
14
Summary & interpretations
• The data indicate that the proposed designbased technique may in fact lead to better
outcomes.
• Indicated by a materially-lower density of
defects (on the order of four times lower) that
were attributable to errors in controlling
system dynamic behavior
15
Summary & interpretations
• The design-based technique considered
herein only applies to projects where such
centralization is possible
– The study did demonstrate that this set of projects is not
vanishingly small
• This specific technique, however, is only one
possible technique for creating a partitioning
of a project into easy and difficult portions
– Future studies could propose and assess other such
techniques.
16
Implications for practice
• Control structures may be more important
than generally recognized
• New goal for the design process
17
Questions?
18
NGIS clearance approval number: ISHQ-2011-0010
19