Ultra-Large-Scale (ULS) Systems
& Their Impact on Technology & Society
Thursday, April 09, 2015
Dr. Douglas C. Schmidt
d.schmidt@vanderbilt.edu
www.dre.vanderbilt.edu/~schmidt
Institute for Software
Integrated Systems
Vanderbilt University
Nashville, Tennessee
The Road Ahead
CPU & network performance has increased by orders of
magnitude in past decades
10 Megahertz to
4+ Gigahertz
Extrapolating these trends another decade or so yields
• ~10 Gigahertz desktops
• ~100 Gigabits/sec LANs
• ~100 Megabits/sec wireless
• ~10 Terabits/sec Internet backbone
Unfortunately, software quality &
productivity hasn’t improved as
rapidly or predictably as hardware –
especially for mission- & safetycritical systems
1,200 bits/sec to
10+ Gigabits/sec
Example Mission- & Safety-Critical Systems
Why Hardware Improves So Consistently
Advances in hardware & networks stem largely from
R&D on standardized & reusable APIs & protocols
x86 & Power PC chipsets
TCP/IP
Why Software Fails to Improve as Consistently
In general, software has not been as standardized or
reusable as hardware, especially for DRE systems
Proprietary & Stovepiped Application & Infrastructure Software
Application
Software
Application
Software
Application
Software
Standard/COTS Hardware & Networks
Application
Software
What Makes Software Hard?
“Human Nature”
Technical Complexities
Accidental Complexities
•Low-level APIs & debug tools
• Organizational impediments
•Algorithmic decomposition
• Economic impediments
Inherent Complexities
• Administrative impediments
•Function & quality property
assurance
• Political impediments
•Scheduling & synchronization
• Psychological impediments
•Deadlock
• etc. …
•etc. …
www.dre.vanderbilt.edu/~schmidt/reuse-lessons.html
Key Challenges for Software Developers
Developers & users of software face
challenges in multiple dimensions
Logical
View
Process
View
Physical
View
Development
View
Key Challenge for Software Developers
Determining units of abstraction
for system (de)composition,
reuse, & validation
Logical
View
• Popular technologies & tools provide
inadequate support for
• Checking pre-/post-conditions & invariants
• Specifying, analyzing, & optimizing
dependencies
• Expressing design intent more clearly
using domain concepts
Our Solution Approach
Devising composable
abstractions whose interfaces &
QoS properties can be
specified/analyzed/optimized
via metadata
Logical
View
18%
reduction
Deployment-Time Fusion
Total
Footprint
45%
reduction
Required Interface
Event Source
Component
Collocation Group
Provided Interface
Event Sink
Physical Assembly
Deployment Plan
www.dre.vanderbilt.edu/~schmidt/IEEE-RTAS-08.pdf
Application Assembly
Key Challenge for Software Developers
• Popular technologies & tools
provide inadequate support for
• Configuring & customizing
components for application
requirements & run-time
environments
• Automated optimized mapping
of components onto nodes in
target environments
Physical
View
Integrating/deploying diverse new &
reusable application components in a
networked environment to ensure endto-end QoS requirements
Our Solution Approach
Physical
View
Model-driven development & analysis
techniques for optimizing, verifying, &
automating the deployment &
configuration process
Gigabit Ethernet
www.dre.vanderbilt.edu/~schmidt/JSS-06.pdf
Key Challenge for Software Developers
• Popular technologies & tools
provide inadequate support for
• Identifying & reducing
performance & robustness
risks earlier in system lifecycle
• Satisfying multiple (often
conflicting) QoS demands
• e.g., secure, real-time,
reliable
• Satisfying QoS demands in
face of fluctuating/insufficient
resources
• e.g., mobile ad hoc
networks (MANETs)
Devising execution architectures, concurrency
models, & communication styles that ensure
multi-dimensional QoS & correctness of
new/reusable components
Process
View
Our Solution Approach
Skoll Server(s)
Distributed continuous quality assurance
techniques run on virtual computing grid
subtask res.
subtask
register
req.
Skoll
clients(s)
subtask
client kit
Process
View
Intelligent Steering Agent
Configuration
Model
Automatic
Adaptation
Subtask
Visualization
(ISA)
characterization
strategies
Code
Adaptation
Configuration
e.g.,
e.g.,
e.g.,
classification
nearest
scoreboard
neighbor
trees
www.dre.vanderbilt.edu/scoreboard
<sub-task>
Strategies
Option
Model
Settings
Interpretation
<download>
{gcc2.96, SUNCC5_1}
Compiler
AMI
0 = NO}
Enable Feature
<cvs>cvs.doc.wustl.edu</cvs>
… {1{1 == Yes,
CORBA_MSG
Yes, 0 = NO}
Enable Feature
CALLBACK
{1
=
Yes,
0
=
NO}
Enable Feature
<module>ACE+TAO</module>
Output: Subtask
Intelligent
Steering
POLLER
{1 = Yes, 0 = NO}
Enable Feature
<version>v5.2.3</version>
Agent {1 = Yes, 0 = NO}
run(T)
Test T runnable
</download>Constraints
AMI = 1 → CORBA_MSG = 1
… = 1 → (Compiler = SUNCC5_1)
run(Multiple/run_test)
Subtask
</subtask>
Client
Results
Compiler
Characteristic
www.dre.vanderbilt.edu/~schmidt/IEEE-TSE-07{a,b}.pdf
Key Challenge for Software Developers
• Popular technologies & tools
provide inadequate support for
avoiding
• Cyclic dependencies, which
make unit testing & reuse
hard
• Excessive link-time
dependencies, which bloat
the size of executables
• Excessive compile-time
dependencies, where small
changes trigger massive
recompiles
(De)composing systems into reusable
modules (e.g., packages, subsystems,
libraries) that achieve/preserve required
QoS properties
Development
View
Our Solution Approach
Standards-based, General-purpose,
Layered Middleware Architecture
Protocol
Interface
OBJ
REF
Services
Interface
Component
(Servant)
in args
operation()
out args +
return
IDL
SKEL
DII
Services
Client
Component
Interface
Customized Middleware Stack
Feature-Oriented
CUStomizer (FOCUS)
DSI
Container
IDL
STUBS
ORB
INTERFACE
Object Adapter
ORB CORE
GIOP/IIOP/ESIOPS
Average end-to-end
measures improved
by ~43%
Development
View
Feature-oriented specialization
optimization tool, e.g., layer folding,
deployment platform, memoization,
constant propagation, etc.
www.dre.vanderbilt.edu/~schmidt/ACM-EuroSys-06.pdf
New Challenges: Ultra-Large-Scale (ULS) Systems
A ULS System has unprecedented scale in some of these
dimensions:
• Lines of code
• Amount of data stored, accessed, manipulated, & refined
• Number of connections & interdependencies
• Number of hardware elements
• Number of computational elements
• Number of system purposes & user perception of these
purposes
• Number of routine processes, interactions, & “emergent
behaviors”
• Number of (overlapping) policy domains & enforceable
mechanisms
• Number of people involved in some way
•…
ULS systems are socio-economic ecosystems comprised of
software-intensive systems, people, policies, cultures, & economics
ULS systems report is available from www.sei.cmu.edu/uls
Example Emerging ULS Systems
ULS systems are systems of systems at Internet scale
ULS system examples from Linda Northrop’s OOPSLA 2006 keynote talk
Scale Changes Everything
Characteristics of ULS systems that arise
because of their scale include
• Decentralization
• Inherently conflicting, unknowable, &
diverse requirements
• Continuous evolution & deployment
• Heterogeneous, inconsistent, & changing
elements
• Erosion of the people/system boundary
• “Normal” failures
• New paradigms for acquisition & policy
• These characteristics appear in some of
today’s systems, but in ULS systems they
dominate
• These characteristics undermine the
assumptions that underlie today’s
technology approaches, rendering
incremental solutions inadequate
Serialized Phasing is Common in ULS Systems
System
infrastructure
components
developed first
Level of Abstraction
Application components
developed after infrastructure is
sufficiently mature
Software Lifecycle Timeline
Serialized Phasing is Common in ULS Systems
Level of Abstraction
System integration &
testing is performed after
application development is
finished
Integration
Surprises!!!
Software Lifecycle Timeline
Complexities of Serialized Phasing
Level of Abstraction
Still in development
Ready for testing
Complexities
• System infrastructure cannot be tested
adequately until applications are done
Development Timeline
Complexities of Serialized Phasing
Level of Abstraction
End-to-end
performance of
critical path?
System bottleneck?
Complexities
• System infrastructure cannot be tested
adequately until applications are done
• Entire system must be deployed &
configured (D&C) properly to meet endto-end QoS requirements
• Existing tools & platforms have poor
support for realistic “what if” evaluation
Development Timeline
QoS needs of components in ULS systems often unknown until late in lifecycle
Promising Approach for ULS System Challenges:
System Execution Modeling (SEM) Tools
Tools to express & validate design
rules
• Help applications & developers
adhere to system specifications at
design-time
Tools to ensure design rule
conformance
• Help properly deploy & configure
applications to enforce design rules
throughout system lifecycle
Tools to conduct “what if” analysis
• Help analyze QoS concerns prior to
completing the entire system, i.e.,
before system integration phase
Ensure
Design
Conformance
Express &
Validate
Design
Rules
Conduct
“What If”
Analysis
SEM tools should be applied continuously when developing software elements
SEM Tool Approach for Planning Aspect
Approach
• Develop Component Workload Emulator (CoWorkEr) Utilization Test Suite
(CUTS) so architects & systems engineers can conduct “what if” analysis on evolving
systems by
1. Software architects, developers, & systems engineers specify structure of an
enterprise DRE system
2. They associate necessary QoS characteristics with individual components (e.g., CPU
utilization) or system as a whole
3. Tools synthesize emulation
code & configuration
metadata to deploy system in
target environment
4. Metrics are collected &
analyzed to explore how
design alternatives from
multiple computational &
valuation perspectives affect
system QoS
CUTS integrates nicely with continuous integration servers & test-driven development
Level of Abstraction
Emulating Computational Components in CUTS
• Application components are represented as
Component Workload Emulators
(CoWorkErs)
• CoWorkErs can be interconnected by the
PICML tool to form operational strings
• CoWorkErs are programmed using the
Component Behavior Modeling Language
(CBML)
Software Lifecycle Timeline
www.dre.vanderbilt.edu/~schmidt/IEEE-RTCSA-06.pdf
www.cs.wustl.edu/~schmidt/PDF/CUTS.pdf
Level of Abstraction
Emulating Computational Components in CUTS
High-level Overview of WML’s
Modeling Structure
• Workload Modeling Language (WML) is used to
parameterize CBML actions with realistic &
executable workload
Software Lifecycle Timeline
Level of Abstraction
Emulating Computational Components in CUTS
• Workload Modeling Language (WML) is used to
parameterize CBML actions with realistic &
executable workload
• Fully parameterized models are used to generate
emulation code for the target architecture
• i.e, spec. compliant code
Software Lifecycle Timeline
www.dre.vanderbilt.edu/~schmidt/WML-book-chapter.pdf
www.cs.wustl.edu/~schmidt/PDF/QoSPML-WML.pdf
Level of Abstraction
Analyzing Systemic Performance in CUTS
• Benchmark Manager Web (BMW) Utility provides
analytical capabilities for CUTS
• e.g., showing performance of actions in each
CoWorkEr, or evaluating non-functional
concerns of mission critical operational strings
Software Lifecycel Timeline
www.dre.vanderbilt.edu/~schmidt/IEEE-ECBS-2008.pdf
Concluding Remarks
• The emergence of ULS systems
requires significant innovations &
advances in tools & platforms
• Not all technologies provide the
precision we’re accustomed to in
legacy small-scale systems
• Advances in computing technology &
related disciplines needed to address
ULS systems challenges
• Significant groundwork layed in various
R&D programs
• Much more
research needed
for ULS systems
• Currently working
with SEI, OSD, &
NSF on ULS
systems R&D