Systems of Systems Acquisition and
Management Critical Success Factors
October 2006
Rick Selby
Head of Software Products
Northrop Grumman Space Technology
Rick.Selby@NGC.com, 310-813-5570
Adjunct Professor of Computer Science
University of Southern California
Rick.Selby@USC.edu
0
© Copyright 2006. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.
Systems and Software Engineering Organizations
Have Different Customers and Strategies
Commercial Software
Products
Examples:
Large-Scale
Organization Microsoft
Amadeus
Products,
Services,
Systems
Project
management
& analysis
products
Desktop/server
platform and
productivity
products
Information Technology &
Internet Financial Services
Examples:
1
Small-Scale
Large-Scale
Small-Scale
Government Aerospace
Systems
Examples:
Large-Scale
Small-Scale
Organization PIMCO
US Business
Exchange
Organization Northrop
Grumman
prime
contractor
Northrop
Grumman subcontractors
and suppliers
Products,
Services,
Systems
Mergers and
acquisitions
workflow
services
Products,
Services,
Systems
System
components
Asset
management
and
operations
services
Spacecraft,
satellite communication,
and laser
systems
© Copyright 2006. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.
Different Systems and Software Engineering
Organizations Have Different Goals and Needs
2
Characteristics
Commercial
Software Products
Information Technology & Government Aerospace
Internet Financial Services
Systems
Market
Commercial
Information technology &
internet
Government
Industry
Software
Financial
Aerospace
Packaging
Products
Services
Systems
Primary Output
Software
Integrated system engr &
HW & SW & network
Integrated system engr &
HW & SW & network
Purpose
User empowerment:
effectiveness,
efficiency, creativity
Organization/business
operations
Mission/science
capabilities
Project Duration
1-36 months
1-18 months
6 months - 10 years
Team Size
1-1000's
1-1000's
10's-1000's
Ratio of Custom
to COTS/Reuse
Software: Low-high
Business logic: High
Others: Low
All: High
Agreement
License
Service level agreement
Contract
Customer
External
Internal and external
External
# Customers
100's-1,000,000's
1-1,000,000's
1
Focus
Features, Time-tomarket, Ship it
User experience, Workflow
cycletime, Uptime
Reliability, Milestones,
Interdependencies
© Copyright 2006. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.
Different Best Practices Apply to Different Systems
and Software Engineering Organizations (1 of 2)
Commercial
Software Products
Information Technology &
Internet Financial Services
Government Aerospace
Systems
Requirements &
design reviews
X
X
X
Incremental or
spiral lifecycles
X
X
X
Risk management
X
X
X
Automated
regression tests
X
X
X
Best Practices
System modeling
X
Product lines
X
Code generators
X
Daily builds
X
Built-in macro
languages
X
X
X
X
Independent
testing teams
Beta testing
3
X
X
Parallel new &
legacy systems
X
Redundant HW
X
X
© Copyright 2006. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.
Different Best Practices Apply to Different Systems
and Software Engineering Organizations (2 of 2)
Commercial
Software Products
Information Technology &
Internet Financial Services
Government Aerospace
Systems
Parallel small
teams
X
X
X
Design reuse
X
X
X
Opportunity mgnt.
X
X
X
Tradeoff studies
X
Usability lab
X
Best Practices
X
Deterministic,
simple designs
X
COTS-based
designs
X
X
Portability layer
X
X
StateData rollback
Resource margins
X
X
X
Earned value
Remote admin.
X
X
Database
replication
Separate offices
4
X
X
X
X
X
© Copyright 2006. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.
Researchers Define Example Roadmap for
Empirical Software Engineering
COVERAGE:
Problem Domain
COVERAGE:
Process/Technique/Phase
COVERAGE
Comprehensive domain coverage
Comprehensive processes,
techniques, and phases
Multiple domains
MATURITY:
Research Methodology
Multiple techniques
and phases
COVERAGE:
Artifact Scale
Single domain
Objective framework, standards
Systems-of-systems
Single project
Multiple techniques
or phases
Repeatable methods
Large-scale systems
Integrated techniques
Single technique/
phase
Understand technique tradeoffs
Single artifact
Small-scale systems
Subsystems
Units/components
MATURITY
Isolated techniques
Subjective views
MATURITY:
Cohesiveness of Field
Individual research plan
Common terminology
Guidelines
Standards
Common research plan
Handbook
Junior expertise
Intermediate expertise
Advanced expertise
COVERAGE:
Subject Expertise Level
Isolated examples
Subset of Empirical
Software Engineering
Research conjectures
Organizational adoption
Empirical Software
Engineering
Supportive observations
Common patterns/similarities
Multi-organizational adoption
Software
Engineering
Industry-wide shift
Validated theories
Computer
Science
IMPACT:
Industry
Systems Engineering and/or
Information Technology
IMPACT
5
Replicated results
IMPACT:
Science/Engineering
UNDERSTANDING:
Evidence
UNDERSTANDING
© Copyright 2006. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.
Enterprise Maturity Models Assess Progress and
Improvements Toward Long-Term Goals
Goal
Process Maturity
Optimization
Improvement
Process
Measures
Process Cost
Characterization
Cycle Time
Internal
(Stovepipe)
CrossProcess
External
Quality
Value Stream Focus
Three Dimensions to Enterprise Maturity
6
© Copyright 2006. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.
Synergistic Strategies Help Enable Large-Scale
Software System Development and Management
Analysis
Infrastructure and
techniques for system
modeling, analysis, and
simulation
Modeling
Infrastructure &
Techniques
Evaluations &
Feedback
System modeling,
evaluation, tradeoff, and
prediction using
simulations and empirical
studies
Analysis
Capabilities
Processes &
Architectures
7
Requirements &
Opportunities
Synthesis
Flexible lifecycle process
models, extensible
system architectures, and
pro-active development
guidance mechanisms
Models,
Relationships &
Feedback
 Research focus: Large-scale, mission-critical embedded software systems
 Research themes: Early lifecycle, system perspective, frequent design cycles,
multi-artifact integration, scalable modelware
© Copyright 2006. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.
Systems and Software Engineering Research
Tackles Challenges for the Future
 Research and technology roadmap for Systems and
Software Engineering
 Validated experiences and data for best practices and the
circumstances when they apply
 Independent objective interoperability mechanisms and
benchmarks for facilitating information exchange
 Unified constructive framework for modeling tradeoff
analyses throughout the lifecycle
 Strategies, architectures, and techniques for development
and management of systems and software
 Multiple customers and markets
 Spectrum of small-scale through large-scale
8
© Copyright 2006. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.