LAI Whitepaper Series
“Lean Product Development for Practitioners”
Program Management
for Large Scale Engineering Programs
Version 1.0 – December 2011
Prepared by:
Dr. Josef Oehmen
Dr. Eric Rebentisch
Kristian Kinscher
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
1 About LAI
The Lean Advancement Initiative (LAI) at MIT, together with its Educational Network (EdNet),
offers organizational members from industry, government, and academia the newest and best
thinking, products, and tools related to lean enterprise architecting and transformation. LAI is
a unique research consortium that provides a neutral forum for sharing research findings,
lessons learned, and best practices.
LAI offers:
 unique opportunities to engage with customers, suppliers, and partners to solve
problems and share organizational transformation experiences
 a portfolio of thought-provoking knowledge exchange events and meetings
 innovative enterprise transformation products, tools, and methodologies
LAI researches, develops, and promulgates practices, tools, and knowledge that enable and
accelerate enterprise transformation. LAI accelerates lean deployment through identified best
practices, shared communication, common goals, and strategic and implementation tools
honed from collaborative experience. LAI also promotes cooperation at all levels and facets of
an enterprise to eliminate traditional barriers to improving industry and government
teamwork.
The greatest benefits of lean result when the operating, technical, business, and
administrative units of an enterprise strive for enterprise-wide lean performance. LAI is
completing its fifth Enterprise Value phase, during which LAI has engaged in transforming
aerospace entities into total lean enterprises and delivered more value to all stakeholders
than would have been possible through conventional approaches.
Contact Information
Lean Advancement Initiative
Massachusetts Institute of Technology
77 Massachusetts Avenue
Building E38-642
Cambridge, MA 02139
Homepage: http://lean.mit.edu
Phone: +1 (617) 258-7628
Email: lean@mit.edu
2 About this Series
A vast amount of research has been conducted at MIT´s Lean Advancement Initiative (LAI) on
Lean Product Development in the last 15 years. For the first time, this series of papers makes
this research accessible to practitioners in a condensed form.
The aim is to provide an application-oriented, readable, concise and comprehensive overview
of the main fields of Lean Product Development. The papers follow LAI´s understanding and
Page 2
LAI Whitepaper Series
philosophy regarding Lean Management concepts and especially their integration into large
and complex Enterprise settings.
The papers draw mainly
on the research done
by
LAI.
Where
necessary to ensure a
comprehensive
presentation of a topic,
findings
of
other
researchers
and
research groups from
the field of Lean
Product Development
are integrated into the
papers.
Figure 1: Topics of the Paper Series - LAI's Three Main Areas of Lean Product
Development
The series focuses on 15 topics in three major areas of Lean Product Development that LAI
identified (see Figure 1). The processes span the space from single project to project portfolio
management. This paper addresses topics 4-6, summarizing them as Program and MultiProject Management.
2.1 I: Processes for Value-orientation
The processes for value-orientation address those types of processes that ensure a focus on
the creation of value and the elimination of waste in Lean Product Development. This covers
the areas of stakeholder needs generation, trade space exploration and decision making, as
well as the identification and handling of value and waste in the core PD processes.
2.2 II: Processes for Enterprise Integration
Enterprise Integration is one of the main challenges in developing a Lean Enterprise. Product
Development plays a central role in this integration effort, as it interfaces with all main
Enterprise processes. This therefore larger group consists of the processes of enterprise,
program and multi-project management, performance metrics and measurement, product
architecture and commonality management, risk management, IT systems, HR development
and human capital, and teams in Product Development.
2.3 III: Processes for Efficient Execution
This group addresses the challenges surrounding the efficient execution of PD processes. It
includes the relationship of PD to overall Enterprise process improvement initiatives, enabling
organizational factors within Lean PD, as well as addressing alternative Lean PD core process
principles.
Page 3
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
Table of Contents
1
About LAI ........................................................................................................................................... 2
2
About this Series ................................................................................................................................ 2
2.1 I: Processes for Value-orientation ............................................................................................. 3
2.2 II: Processes for Enterprise Integration ..................................................................................... 3
2.3 III: Processes for Efficient Execution ......................................................................................... 3
3
Summary and objective of this whitepaper ....................................................................................... 5
4
Introduction to Program and Multi-Project Management ................................................................ 5
5
Defining Program Management ........................................................................................................ 7
6
Overview of Existing Program Management Guidelines ................................................................. 10
7
Program Management within the Scope of LAI’s Research ............................................................. 15
7.1 Structure of LAI Program Management Framework ............................................................... 15
7.2 Overview of Program Management Core Competencies ....................................................... 17
8
LAI’s Insights on Program Management .......................................................................................... 22
8.1 LAI’s Insights: Program Organization ...................................................................................... 22
8.1.1
8.1.2
8.1.3
8.1.4
8.1.5
8.1.6
Integrating and leading the program organization .................................................... 22
Stakeholder engagement ........................................................................................... 23
Progress monitoring and management...................................................................... 24
Risk management ....................................................................................................... 25
Project management .................................................................................................. 28
Multi-project coordination ......................................................................................... 28
8.2 LAI’s Insights: Enterprise Management................................................................................... 30
8.2.1
8.2.2
8.2.3
8.2.4
Developing intellectual capital ................................................................................... 30
Building and transforming the program enterprise ................................................... 32
Understanding the program enterprise ..................................................................... 34
Learning and improving in the program enterprise ................................................... 36
8.3 LAI’s Insights: Scoping, planning & contracting....................................................................... 39
8.3.1
8.3.2
8.3.3
8.3.4
Definition of stakeholder needs and requirements ................................................... 39
Managing trade-offs ................................................................................................... 43
Cost estimation & life cycle costing ........................................................................... 44
Incentive alignment, contract negotiation & conclusion ........................................... 46
8.4 LAI’s Insights: Technology Integration .................................................................................... 48
8.4.1
8.4.2
Technology maturation monitoring ........................................................................... 48
Technology transition management .......................................................................... 49
8.5 LAI’s Insights: Product Design ................................................................................................. 50
8.5.1
8.5.2
8.5.3
8.5.4
8.5.5
8.5.6
8.5.7
8.5.8
9
Product Design team organization ............................................................................. 50
Integrated product and process development .......................................................... 52
Product Design and System Engineering best practices ............................................ 54
Monitoring Product Design progress ......................................................................... 55
Product architecting ................................................................................................... 56
Value Stream Optimization ........................................................................................ 59
Product Design strategy ............................................................................................. 60
Testing & prototyping ................................................................................................ 62
Literature references ....................................................................................................................... 64
Page 4
LAI Whitepaper Series
3 Summary and objective of this whitepaper
The goal of this whitepaper is to summarize the LAI research that applies to program
management. The context of most of the research discussed in this whitepaper are large-scale
engineering programs, particularly in the aerospace & defense sector.
The main objective is to make a large number of LAI publications – around 120 – accessible to
industry practitioners by grouping them along major program management activities. Our
goal is to provide starting points for program managers, program management staff and
system engineers to explore the knowledge accumulated by LAI and discover new thoughts
and practical guidance for their everyday challenges.
The whitepaper begins by introducing the challenges of programs in section 4, proceeds to
define program management in section 5 and then gives an overview of existing program
management frameworks in section 6. In section 7, we introduce a new program
management framework that is tailored towards describing the early program management
phases – up to the start of production. This framework is used in section 8 to summarize the
relevant LAI research.
The authors wish to thank the LAI consortium members for their insightful comments and
suggestions, most notably Tony Eastland, Don McAlister, Stuart Swalgen, Sean Dorey and
Dustin Ziegler; the members of the Lean in Program Management Community of Practice, as
well as our colleagues at MIT. The sole responsibility for any errors and inaccuracies remains
with the authors. The views expressed in this document are those of the authors alone.
4 Introduction to Program and Multi-Project Management
This paper focuses on large-scale, engineering development projects. It draws mostly from
insights in the aerospace and defense industry. These large-scale projects are referred to as
‘programs’. The generic term ‘project’ is used for smaller-scale efforts, e.g. the development
of sub-systems within the program. These programs lead to the formation of ‘enterprises’. An
enterprise is the interorganizational network that has the common purpose of the
development and delivery of a system. An enterprise is comprised of both public
organizations, such as the program office, as well as private organizations, such as the main
contractor and its suppliers. As a result of the interorganizational nature of enterprises, they
have distributed responsibility and leadership, and they have stakeholders with both common
and diverse interests (Stanke, 2006).
Product development in these programs occurs through a complex network of human and
physical resources, whose interactions utilize, transform and create knowledge that provides
a new solution to a customer need. Large programs, with many stakeholders and relatively
immature or unstable knowledge bases, produce risks. Some of these risks are mitigated or
avoided, but others whether known or unexpected, are realized and turn into issues that
Page 5
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
impact cost and schedule. These issues result in significant cost and schedule overruns (see
Figure 2): The F-35 Joint Strike Fighter is, for example, currently the most expensive program
of all times, with a total cost overrun of 65% or $150 billion, and an increased price per plane
of 81%, or $50 million. A program cost overrun of around 40% was the average in the last ten
years for RDT&E costs (research, development, test & evaluation). As of 2009, the combined
total cost overrun of major defense acquisition programs is estimated at $296 billion, with a
22 months average delay in delivery of the promised capability (GAO, 2010).
Breaking down the cost overrun into its main root causes is not easy, but three main areas
emerge: Planning that was inaccurate or overly optimistic (e.g. regarding technology
readiness or enterprise capabilities) leads to cost and schedule targets that cannot be
achieved. This also includes understanding the stakeholder requirements, and the stability of
these requirements. Changing stakeholder requirements can have a significant impact on cost
overrun. Planning also involves the careful setup of contracts, so that risks and profits are
fairly shared, as well as the incentives aligned properly. The second area is inefficient
execution of the program. This includes sub-optimal organizational constructs, for example
regarding the integration of program office, main contractor and suppliers. It also includes
inefficient processes, consuming more resources or time than they could and should. The
third factor is technological uncertainty that is added by immature technologies and not
handled properly. All three areas are related to program management: Program management
must ensure top-notch planning of complex engineering programs, the efficient execution of
those programs in an admittedly complex enterprise organization, as well as balance and
manage the need to include cutting-edge technology.
Figure 2: Examples of Cost and Schedule Overrun in DoD Programs (GAO, 2010, GAO, 2006, Capaccio, 2010)
Page 6
LAI Whitepaper Series
Exactly quantifying the amount of
cost overrun each factor contributes
is difficult. Analyses of the
relationship of technology maturity at
program start to the final cost
overrun of the project suggest that
immature technology is responsible
for about 30% of the total cost
overrun. The remaining 70% are
divided between the other two
factors, insufficient planning and
inefficient execution.
Figure 3: Estimation of Cost Overrun Contribution of
Different Factors (adapted from GAO, 2010)
5 Defining Program Management
Program management is a diverse field, and a number of definitions for both “program” and
“program management” exist. Table 1 gives an overview of possible definitions, in alphabetic
order of author.
Table 1: Various definitions of program and program management (in alphabetic order of author)
Author
(Andersen
Jessen, 2003)
Definition of Program and / or Program Management
and
(APM, 2006)
(Archibald, 2003)
(Brown, 2007)
(DAU, 2010a)
“A program could be a new product development, an organizational
restructuring of the company or the implementation of an advanced software
package in different departments of the company. Program management is the
effective management of all the projects under the umbrella of the program."
“Program management is the coordinated management of related projects,
which may include related business-as-usual activities that together achieve a
beneficial change of a strategic nature for an organization”
“Program: A long-term undertaking that includes two or more projects that
require close coordination.”
“Program Management is management of a group of projects and/or operations
to achieve business targets, goals, or strategies, and may or may not have a
defined end point.”
“[Program management is] the process whereby a single leader exercises
centralized authority and responsibility for planning, organizing, staffing,
controlling, and leading the combined efforts of participating/ assigned civilian
and military personnel and organizations, for the management of a specific
defense acquisition program or programs, through development, production,
deployment, operations, support, and disposal.”
Page 7
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
Author
Definition of Program and / or Program Management
(Ferns, 1991)
“A program is a group of projects that are managed in a coordinated way to gain
benefits that would not be possible were the projects to be managed
independently. […] Program management is the coordinated support, planning,
prioritization and monitoring of projects to meet changing business needs.”
“Program management is a technique that allows organizations to run multiple
related projects concurrently and obtain significant benefits from them as a
collection. Program management is a way to control project management, which
traditionally has focused on technical delivery. A group of related projects not
managed as a program are likely to run off course and fail to achieve the desire
outcome.”
“[…] program management is the definition and integration of a number of
projects to cause a broader, strategic business outcome to be achieved. Program
management is not just the sum of all project management activities but also
includes management of the risks, opportunities and activities that occur ‘in the
white space’ between projects.”
“Program Management – defined as the integration and management of a group
of related projects with the intent of achieving benefits that would not be
realized if they were managed independently. Whilst connected, this is distinct
from portfolio management.”
“[…] a Program consists of a series of interrelated projects while Program
Management is to manage all interrelated projects as a whole in a harmonic
manner which is beneficial to successfully fulfill each single project and the entire
program.”
“A program is defined as a temporary, flexible organization created to
coordinate, direct and oversee the implementation of a set of related projects
and activities in order to deliver outcomes and benefits related to the
organization’s strategic objectives. A program is likely to have a life that spans
several years. […] Program management is the coordinated organization,
direction and implementation of a dossier of projects and transformation
activities (i.e. the program) to achieve outcomes and realize benefits of strategic
importance.”
“[…] a program, led by a program manager, is a family of projects that are
strongly dependent, share common goals, and lead to a single deliverable
product or service.”
“Another case of MPM [multi-project management] is program management,
where the projects in the group are mutually dependent, share a common goal,
and lead to a single deliverable product or service. In contrast with MGMP
[management of a group of multiple projects], program management is the
centralized coordinated management of a group of goal-related projects to
achieve the program’s strategic objectives and benefits.”
“Program Management is the centralized coordinated management of a
program to achieve the program’s strategic objectives and benefits. It involves
aligning multiple projects to achieve the program goals and allows for optimized
or integrated cost, schedule, and effort.”
(Haughey, 2001)
(Hut, 2008)
(Lycett et al., 2004)
(Mao et al., 2008)
(OGC, 2007)
(Patanakul
and
Milosevic, 2008)
(Patanakul
and
Milosevic, 2009)
(PMI, 2008b)
Page 8
LAI Whitepaper Series
By comparing these definitions, five key aspects of programs and program management
emerge:
1.
2.
3.
4.
5.
A program consists of multiple, related projects. The first aspect in which all
definitions for program management found in current literature agree (except for
one) is that a program consists of multiple projects. Programs may also evolve
from single projects as they become more complex. The organization of the
projects within a program can vary depending on the purpose of the program.
Projects can be organized sequentially, run in parallel as hybrid sequential-parallel
models, or be organized as a network of interlinked projects. However, the most
common concept of the organization of projects within a program is an
environment of interlinked projects.
Long program duration. The second aspect focuses on the duration of a project or
program. Whereas projects are time constrained, programs in general do not
need to have a fixed end date. A program can only come to an end when the
deliverable (e.g., a product or service) is fully developed and handed over to the
user/customer. In addition, some authors state that a program is also responsible
for the time after handing over the product or service to the user. This may
include, for example, deployment, operations, support, and disposal.
Management economy of scale. There are many tasks and objectives in the
projects within a program that are similar. By identifying best practices among
different projects and introducing organization-wide learning, huge benefits can
be gained when projects are managed in a program instead of being managed in
isolation in a project management context.
Complex value proposition. A program focuses on multiple stakeholders and has
to satisfy a complex value proposition. While a single project may only deliver an
output such as a product, a program has to satisfy the multifaceted needs of
several stakeholders. A program could, for example, provide a transportation
solution for urban environments, while the project would deliver a train or bus for
this purpose.
Programs define their own enterprise. Based on the research undertaken by the
LAI, the formation of a program enterprise is included in the definition of program
management used in this thesis. The organization of each program is unique and
established to satisfy the needs of all stakeholders in the best way. Since a
program has its own program office and has to defend decisions against undue
influence from single stakeholders, a program has a certain degree of autonomy.
Therefore, we define program management in the context of this whitepaper as follows:
“Program management is defined as the management of multiple related projects and the
resulting enterprise over an extended period of time to deliver a complex value proposition
and utilize efficiency and effectiveness gains.”
Page 9
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
6 Overview of Existing Program Management Guidelines
A number of industry and government organizations provide guidelines for program
management. While we cannot present all relevant standards, this section will introduce
relevant documents from the US Department of Defense, the Project Management Institute,
and the UK Office for Government Commerce (OGC).
Figure 4: Program Management Elements according to Defense Acquisition Guidebook (DAU, 2010a)
The work on program management carried out at LAI stands in the context of a number of
different bodies of knowledge. First and foremost, the Department of Defense Instruction
Page 10
LAI Whitepaper Series
5000.02 describing the operation of the acquisition system (DoD, 2008). The content of the
DoDI 5000.02 is supported by additional detail and best practices outlined in the Defense
Acquisition Guidebook (DAU, 2010a), summarized in Figure 4.
In addition to the Guidebook, the Defense Acquisition University also offers a number of
courses that are relevant to program management. Among others, these include ACQ-101:
Fundamentals of Systems Acquisition Management, ACQ-201: Intermediate Systems
Acquisition Course, PMT-251: Program Management Tools, and PMT-352: Program
Management Office Course (DAU, 2010b). The course content is summarized in Figure 5.
Figure 5: Selection of DAU courses and content on Program Management (DAU, 2010b)
Two major, generic standards for program management exist: The PMI Program Management
Standard, currently in its 2nd edition (PMI, 2008b), and the OGC standard Managing Successful
Programmes, MSP (OGC, 2007).
PMI groups program management activities into 5 program management process groups and
12 knowledge areas (PMI, 2008b). The activities in each knowledge area are structured along
the process groups:


Program management process groups: Initiating process, planning process, execution
process, monitoring and control process and closing process
Knowledge areas: Integration management, scope management, time management,
cost management, quality management, human resource management,
communication management, risk management, procurement management, financial
management, stakeholder management and program governance.
Page 11
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
The OGC standard consists of
three major elements (see Figure
8):
1. Transformational Flow: Identifying a program, defining a
program,
managing
the
program tranches, delivering
the capability, realizing the
program benefits and closing a
program
2. Governance Themes: Organization, Vision, Leadership and
stakeholder
engagement,
benefits
realization
management, blueprint design
and delivery, planning and
control, business case, risk
management
and
issue
Figure 6: Structure of the OGC ‘Managing Successful
resolution,
and
quality Programmes’ standard (from OGC, 2007)
management.
3. Program management principles: Remaining aligned with corporate strategy, leading
change, envisioning and communicating a better future, focusing on the benefits and
threats to them, adding value, designing and delivering a coherent capability, and learning
from experience.
Based on the recent PMI Role
Delineation Study (PMI, 2011), a new
framework emerged to structure
program management activities: The 5
program management performance
domains (see Figure 7). The program
management performance domains
are:

Program strategy alignment:
Identifying opportunities and
benefits that achieve the
organization‘s
strategic
objectives through program
implementation
Figure 7: 5 Program Management Performance Domains
(Norman, 2011)
Page 12
LAI Whitepaper Series




Program benefits management: Defining, creating, maximizing, and sustaining the
benefits provided by programs.
Program stakeholder engagement: Capturing stakeholder needs and expectations,
gaining and maintaining stakeholder support, and mitigating / channeling opposition
Program governance: Establishing processes and procedures for maintaining proactive program management oversight and decision-making support for applicable
policies and practices throughout the entire program life cycle.
Program lifecycle management: Managing all the program activities related to
program definition, program benefits delivery and program closure
Another relevant resource for Program Management is the project management literature, as
programs often contain a large number of projects that must be guided and coordinated
thorough program management. Two major overlapping frameworks exist: the Project
Management Institute with the Project Management Body of Knowledge (PMI, 2008a), and
the PRINCE2 framework developed in the UK by the Office of Government Commerce (OGC,
2009). Fundamentally, both provide scalable and detailed project management guidelines.
Whereas the PMI approach places more emphasize on the different knowledge areas and
application processes of project management (such as scope and time management), the
PRINCE2 approach focuses more on managing project stages and their transitions. The
content of the guidelines is summarized in Table 2.
The following sections will discuss the contributions that LAI research has made to the
management of large-scale engineering programs. The structure of the LAI contributions do
not follow any particular program management standard, but will be mapped to the program
management standards and models discussed before for reference.
Page 13
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
Table 2: Comparison of Project Management Standards
Content of PMI Project Management Body Content of OGC PRINCE2 Standard (OGC,
of Knowledge (PMI, 2008a)
2009)
 Project Life Cycle and Organization: Project
phases, stakeholders, organizational
integration
 Project Management Processes: Project
initiating, planning, executing, monitoring and
closing
 Project Integration Management: Project
charter, management plan, coordination of
processes
 Project Scope Management: Requirements,
scope, WBS, verify and control scope
 Project Time Management: Define and
sequence activities, estimate resources and
duration, develop and control schedule
 Project Cost Management: Estimate costs,
determine and control budget
 Project Quality Management: Plan quality,
quality assurance and control
 Project HR Management: HR plan, acquire,
develop and manage project team
 Project Communication Management:
Stakeholders, communication plan,
information distribution and reporting,
stakeholder expectations
 Project Risk Management: Plan, identify,
analyze, mitigate, monitor and control risks
 Project Procurement Management: Plan,
conduct, administrate and close procurement
Page 14
 Project Management Principles: Continued
business justification, learning from
experience, defined roles and responsibilities,
manage by stage, manage by exceptions, focus
on product, tailor process
 Project Management Themes: Business case,
organization, quality, plans, risk, change,
progress
 Starting up a project: Project manager and
team appointment, project description,
project approach
 Initiating a project: Project planning, business
case and risk analysis, project controls, project
initiation document
 Directing a project: Authorizing initiation,
project and stages, giving ad-hoc directions,
confirming project closure
 Controlling a stage: Work packages, progress
assessment, project issues, reviewing status,
reporting, correct or escalate issues, receive
completed work package
 Managing stage boundaries: Planning stages,
updating project plan and business case,
updating risk log, report stage end
 Managing product delivery: Linking project
and team leaders by accepting, executing and
delivering work packages
 Closing a project: decommissioning project,
identify follow-on actions, project review
LAI Whitepaper Series
7 Program Management within the Scope of LAI’s Research
7.1 Structure of LAI Program Management Framework
The research summarizes in this paper focusses on the early phases of program management,
up to the start of production. Significant part of programs may lie in those life cycle phases
not addressed by this document. However, the overall program organization and enterprise
management aspects are still relevant for these phases.
Figure 8: Aggregated Structural View of Program Management
Figure 8 structures Program Management along four main activities that will be used to
summarize the related LAI research: Program Organization and Enterprise Management;
Scoping, Planning & Contracting; Technology Integration; Product Design; and Production,
Use, Service and Decommissioning. As this document focuses on the design- and
development-related phases of Program Management, the Production phase and following
life cycle phases are not within its scope. While the model provides a logical structure for
important processes, it does not represent a strict ‘timeline’ for the execution. While for
example cost estimation is important when evaluating different proposals during contracting,
it is also important to update program plans during technology development and product
design. Programs can be expected to go through several spirals of planning, technology
Page 15
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
integration and design. Each of the four main activities contains a number of core
competencies that are discussed in the following subsection. The LAI research contributions
are mapped onto these core competencies in section 7.
The framework consists of four main elements: Program organization and enterprise
management; Scoping, planning and contracting; Technology development; and Product
design. This framework covers a program from the decision for material development to the
start of production. The framework focuses on the processes and organizational practices that
are important for the successful program execution from both a government Program Office
perspective, as well as program management from the contactors perspective (see Figure 9).
Whereas some practices in the model explicitly focus on the interaction between and within
the government and the main contractor, many recommendations are also valid for the
extended program enterprise. That is, the recommendations are also valid within and for
interactions between the main contractor and the system suppliers, the system suppliers and
their suppliers and so forth.
Figure 9: Simplified Representation of Program Enterprise Stakeholders
The LAI research does not follow any particular program management standard, but can be
mapped onto the existing standards (see Table 3).
Page 16
LAI Whitepaper Series
Technology Integration
Product Design
Scoping, Planning &
Contracting
Program Organization and
Enterprise management
Table 3: Mapping of LAI Program Management Elements to Existing Standards
X
X
Managing Successful Programmes (OGC, 2007)
Transformational Flow
X
Governance
X
Program Management Principles
X
nd
PMI Program Management Standard, 2 edition (PMI, 2008b)
Program management process groups
X
Program management knowledge areas
X
PMI Program Management Performance Domains (Norman, 2011)
Program strategy alignment
X
Program benefits management
X
X
Program stakeholder engagement
X
X
Program governance
X
Program lifecycle management
X
7.2 Overview of Program Management Core Competencies
In general, the Program Management function coordinates the transformation of various
input factors, such as stakeholder needs, technology, competitive actions, process definitions,
organization, skills, manpower, allocated budget, regulations etc., into the final product, an
operational system. There are many ways to structure the program “black box”. The
representation in Figure 8 is clustered around the core competencies of a program enterprise
to successfully deliver the operational system, given a certain set of input factors, or
resources. These consist of the four main areas of Program organization and enterprise
management; Scoping, planning & contracting; Technology development; and Product design.
As the LAI research focuses on the early phases of a system’s life cycle, the phases of
production, use, service, refurbishment and decommissioning are not covered in this model.
Program Organization and Enterprise Management consists of two main parts: Program
organization (8.1) focuses on the practices to efficiently execute a program between the
government program office, the main contractor and the system suppliers. Program
Enterprise Management (8.2) addresses more generally those aspects that are important in
Page 17
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
creating, understanding and improving large and complex enterprises. Program organization
encompasses activities such as






Integrating and leading the program organization (8.1.1): This section addresses how
the main organizations of the program enterprise, i.e. program office, main contractor
and system suppliers, interact most effectively, and how this complex organization
can be managed.
Stakeholder engagement (8.1.2): Programs are always dynamic, as even the shortest
programs usually cover several years. Stakeholder needs and the program
environment continually change, and stakeholder engagement is key to integrate
changing stakeholder needs in the program, as well as allow the stakeholders to make
informed decisions on the program (e.g. cost/benefit tradeoffs for changing
requirements, renewal or extension of budgets etc.).
Progress monitoring and management (8.1.3): It addresses the need to understand,
track and manage the progress of the program and informs the corrective actions
when necessary. Earned Value Management (EVM) is one example of such a tool.
Risk management (8.1.4): This is an activity that makes the key program risks
transparent, ensures that the understanding of risks is closely linked to the decision
making process, strives to reduce uncertainty as much as possible, and account for
residual uncertainty and risk with robust processes and systems.
Project management (8.1.5): The activities include managing the execution of and
transition between the different phases of the program, as well as the management
of scope, time, budget and quality of the program (see Table 2 for a detailed
summary).
Multi-project coordination (8.1.6): In addition to the challenges of managing a single
project, program offices execute several projects in parallel, as well as contractors and
suppliers on the different levels also usually have more than one project that they
work on in parallel. This generates specific challenges, such as the allocation of
resources, especially the staffing of the different projects in parallel along their
specific life cycles.
The category of Program Enterprise Management (8.2) contains the following core
competencies:


Page 18
Developing intellectual capital (8.2.1): Companies and government organizations
must provide incentives to their employees to develop deep technical knowledge and
expertise in relevant areas. This includes encouraging and rewarding life-time careers
as technical experts, for example through technology-oriented career towards senior
engineers.
Building and transforming the program enterprise (0): Architecting the enterprise of
both public (government) and private (companies) organizations that are involved in
executing the program and creating stakeholder value. Transforming existing
LAI Whitepaper Series


enterprises along an enterprise transformation roadmap to re-align them with
stakeholder needs and optimize the value flow throughout the enterprise.
Understanding the program enterprise (8.2.3): Assessing existing program
enterprises to benchmark them against best practices and/or other program
enterprises, as well as to uncover areas for improvement.
Learning and improving in the program enterprise (Error! Reference source not
ound.): Successfully managing continuous improvement and organizational change in
complex enterprises.
During Scoping, Planning & Contracting (8.3), the user needs are translated into a legal
contract between the program office and the main contractor. As part of this, the life cycle
strategy for the entire program is planned and codified, spanning system development,
technology transition, operational use and sustainment, as well as future technology refresh
and ultimate system disposal. This step includes the following activities:




Stakeholder needs & requirements definition (8.3.1): The stakeholders in large
programs are numerous and varied, and so are the resulting needs. Stakeholders from
different areas – government, congress, users, general public, contractors etc. – and
different levels of the hierarchy – top management to shop floor – have to be taken
into account. This requires a sophisticated process for eliciting and aligning these
diverse needs.
Managing trade-offs (8.3.2): During the specification of the final requirements,
conflicting requirements have to be traded against each other. This includes for
example trade-offs regarding cost, schedule, performance and risk requirements. The
management of trade-offs must also include re-visiting and re-negotiating
requirements where necessary, in order to assure that the baseline requirements are
realistic.
Cost estimation & life cycle costing (8.3.3): Before a budget is allocated, the resource
requirements in term of cost, schedule, manpower, facilities and other resources has
to be forecasted. This involves a probability-based assessment of these resources so
an informed decision can be made in the following step regarding the type and
content of the contract.
Incentive alignment, contract negotiation & conclusion (0): Contracting under large
program risks is a very challenging task. Based on the cost (and other resources)
probability distributions, an adequate way of sharing risk (cost overrun relative to
baseline) and profit (cost savings relative to baseline) has to be developed. This
should allocate the responsibility for carrying risk consequences according the
responsibility for the risk sources, or create incentives to accept external risks. An
important part of this process is understanding and aligning incentives between the
program office and the contractors, as well as between contractor and its suppliers.
Incentives must include guidelines on how to share benefits from local and
Page 19
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
enterprise-level improvements, improvements strategies, as well as guidelines for
recovering stressed relationships.
During Technology Integration (8.4), technologies for the design phase are selected and
integrated into the program. The readiness of low-maturity technologies may be increased in
parallel technology development tracks to a level that is suitable for their introduction into
the design process. It includes the following activities:


Technology maturation monitoring (8.4.1): The main objective is to develop
technologies to a level where the main risks regarding their cost and performance
have been eliminated and they are fit to be integrated into an overall system.
However, assessing the maturity level of a technology is not a trivial task, as certainty
regarding its development status may be difficult to establish.
Technology transition management (8.4.2): This process addresses the integration of
technologies that just reached sufficient maturity levels at R&D facilities into existing
design and production processes at the main contractor and its suppliers.
In Product Design (8.5), the system integration and its components are specified to a level
that makes it fit for production. It consists of the following activities:





Page 20
Product Design team organization (Error! Reference source not found.): Product
esign is a complex task that relies heavily on the integration of deep domain
knowledge from various areas. This places high requirements not only on the
leadership of PD teams, but also on the team members and the organization that
provides them. In addition, integrated product teams for example bridge the gap
between different organizations.
Integrated product and process development (8.5.2): As requirements and systems
vary, the PD process has to be adapted accordingly. This is closely linked to the
optimization of the value stream, for example with value stream mapping and the
elimination of waste, as well as the selection of an appropriate development strategy,
such as a waterfall model or set-based design.
Product Design and System Engineering best practices (Error! Reference source not
ound.): Some organizational concepts as well as processes have been identified as
especially important for success. These are summarized in best practices that can be
used for an internal benchmarking and to identify opportunities for improvement.
Monitoring Product Design progress (8.5.4): Monitoring progress during PD can be
particularly challenging, as many iterations and re-work of “finished” components
may occur. Therefore, to allow for a better management of the process, the use of
appropriate KPIs and leading indicators is very important.
Product architecting (0): The architecture of a product has important implications on
the fulfillment of customer requirements (e.g. modularity to allow for partial and
continuous upgrades during the life cycle, decoupling of technology development
LAI Whitepaper Series



cycles etc.). It also has a significant influence on how the PD process can be organized
most effectively.
Value Stream Optimization (0): The creation of stakeholder value is the ultimate goal
in product design. Several tools from Lean Product Development are available to
model and improve the flow of value while eliminating waste. The challenge lies in the
identification of value and discerning wasteful activities from necessary, but nonvalue creating activities.
Product Design strategy (8.5.7): A number of strategies or process families can be
chosen for a design that have strong implications for all other program processes and
the way that users and customers are integrated. These include evolutionary
acquisition, waterfall model and set-based design.
Testing and prototyping (8.5.8): This activity is relevant both to technology
development as well as product design. In technology development, the testing and
prototyping would focus on single components or technologies, whereas in product
design, the entire system (or major components) would be tested. This is the main
activity to verify that a certain performance level has been reached, and component
and system requirements are met. Developing testing and prototyping protocols that
assure a necessary level of certainty while minimizing cost is a very challenging task.
Page 21
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
8 LAI’s Insights on Program Management
8.1 LAI’s Insights: Program Organization
In the following, the LAI insights regarding the different areas of program organization and
enterprise management are briefly summarized.
8.1.1 Integrating and leading the program organization
This section addresses how the main organizations of the program enterprise, i.e. program
office, main contractor and system suppliers, interact most effectively, and how this complex
organization can be managed.
Publication Title & Synopsis
Stanke, A. K. (2006) Creating High Performance Enterprises, PhD Thesis, LAI and
Massachusetts Institute of Technology.
There are a small number of exceptional programs that have managed to meet all of
their commitments in terms of cost, schedule, and technical performance
simultaneously. This work explores these programs on a case-by-case basis. Best
practices from these programs are organized into a framework of factors that
differentiate these high performing programs from the majority of others that struggle
in the same environment. These factors are: 1. Distributed Leadership Action: 1.1
Boundary spanning activity across organizations in the enterprise; 1.2Developing and
utilizing a social network; 1.3 Developing and sustaining extensive customer
interactions; 1.4 Fostering and maintaining personal accountability of plans and
outcomes; 2. Informal Structures: 2.1 Boundary spanning activity with the enterprise
environment; 2.2 Encouragement for open information sharing; 2.3 Veteran core
group to institutionalize behavior; 3. Formal Structures: 3.1 Balanced risk through work
share and teaming arrangements; 3.2 Common contract structure; 3.3 Standardized
program management practices (metrics and reporting system)
Wirthlin, J. R. (2009) Identifying Enterprise Leverage Points in Defense Acquisition
Program Performance, PhD Thesis, LAI and Massachusetts Institute of Technology.
The program management office on the government side has a decisive influence on
the performance of a program. This work explores program office management
practices. Based on interviews with senior program managers and decision makers at
the Air Force, key decisions and principles of running a program office were collected.
A simulation model was set up to explore the impact that these decisions and
principles had on the duration of a program to start of production, regarding both the
average duration as well as the variation in duration. Of the 20 possible actions,
combined actions showed the biggest impact. The top five single actions to reduce
overall program duration were 1. Ability to kill programs at milestones; 2. Guaranteed
funding stability; 3. Reduction of technical uncertainty; 4. Improve success at Critical
Design Reviews and 5: Eliminate wait time for Acquisition Panel activities.
Morgan, S. (1999) The Cost and Cycle Time Implications of Selected Contractor and Air
Force System Program Office Management Policies during the Development Phase of
Major Aircraft Acquisition Programs, Master Thesis, LAI and Massachusetts Institute of
Technology.
This work develops a different simulation model that analyzed government program
office management policies and especially workforce development and staffing policies
The focus was again the analysis on the impact on program schedule. The main
Page 22
Citation &
Download Link
(Stanke, 2006)
Download Link
(Wirthlin,
2009)
Download Link
(Morgan,
1999)
Download Link
LAI Whitepaper Series
findings were: 1. There is a strong dependence between government program office
performance and contractor performance; 2. Project compression, i.e. schedule
reduction, is difficult to achieve by increasing the workforce beyond a certain limit; 3.
High workforce turnover is detrimental to program success; 4. Low military tenure is
detrimental to program success; and 5. Government program office workforce size is
critical for success.
McNutt, R. T. (1998) Reducing DoD Product Development Time: The Role of the
Schedule Development Process, PhD Thesis, LAI and Massachusetts Institute of
Technology.
The issue of schedule overruns in programs is further explored in this work, based on
detailed interviews with key stakeholders from the Pentagon, program offices and
industry contractors. The aim was to identify the key reasons for schedule overruns
and develop recommendations for improvements. The main issues leading to long
program schedules are 1. Lack of leadership on cycle time; 2. Lack of schedule-related
information and tools; 3. Lack of schedule-based incentives; and 4. Overriding
influence of funding-related constraints. The derived recommendations are 1. Provide
clear leadership on cycle time reduction; 2. Develop and use schedule-based
information; 3. Provide incentives for cycle time reduction; and 4. Mitigate fundingbased constraints on development projects
(McNutt,
1998)
Download Link
8.1.2 Stakeholder engagement
Programs are always dynamic, as even the shortest programs usually cover several years.
Stakeholder needs and the program environment continually change, and stakeholder
engagement is key to integrate changing stakeholder needs in the program, as well as allow
the stakeholders to make informed decisions on the program (e.g. cost/benefit tradeoffs for
changing requirements, renewal or extension of budgets etc.).
Publication Title & Synopsis
McKenna, N. (2006) The Micro-foundations of Alignment among Sponsors and
Contractors on Large Engineering Projects, Cambridge, MA, Master's thesis, MIT and
LAI.
A contributing factor to the cost, schedule and performance challenge of large projects
is that the project enterprise is created by separate firms being brought together by
the project sponsor. Success requires multiple firms working together to efficiently
create complex product systems. In an attempt to improve project outcomes, sponsors
often endeavor to create “alignment” between themselves and their key contractors.
In practice, alignment has proved difficult to create and to sustain. This research
explores the policies and actions taken by firms that give rise to alignment. The large
engineering projects studied for this research were offshore oil and gas field
developments. The research revealed that alignment is founded on the collective
understanding of the project, incorporating the firm's separate interests, and inter-firm
trust. Furthermore the two antecedents of alignment act together to form a selfenforcing alignment mechanism. Six factors (system architecture, organizational
design, contract design, risk, metrics and incentives) were identified that establish the
inter-firm interactions through which collective understanding and inter-firm trust are
created. These findings are organized into a framework that guides policy selection
with a view to enabling the generation, and sustainment, of alignment.
McVey, M. E. (2002) Valuation Techniques for Complex Space Systems: An Analysis of a
Potential Satellite Servicing Market Cambridge, MA, Master's thesis, LAI and MIT.
Current financial valuation techniques fail to capture several important aspects of
Citation &
Download Link
(McKenna,
2006)
Download Link
(McVey, 2002)
Download Link
Page 23
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
technical projects, including flexibility and the interface between economics and
technology. Additionally, valuations rarely aid in the process of determining which
service or product provides value to both the client and the provider. This thesis
presents a new valuation framework that accounts for these downfalls by breaking the
valuation analysis into two distinct parts: the client’s value and the provider’s value.
The client value analysis is a necessary step in determining the provider’s value, as it
provides the basis for the revenue the provider will generate as well as an idea of
which type of service or product provides the most value to the client. As a viable
market does not exist without both a client and a provider, it is necessary to look at a
project from both perspectives. The valuation framework is used to analyze the
commercial geosynchronous satellite servicing market.
8.1.3 Progress monitoring and management
This area addresses the need to understand, track and manage the progress of the program
and informs the corrective actions when necessary. Earned Value Management (EVM) is one
example of such a tool.
Publication Title & Synopsis
Blackburn, C. D. (2009) Metrics for Enterprise Transformation, Cambridge, MA,
Master's thesis, MIT and LAI.
The objective of this thesis is to depict the role of metrics in the evolving journey of
enterprise transformation. Two case studies were performed, considering: 1. The
implementation of a bottom-up measurement system to drive transformation and 2.
The effect of a top-down corporate measurement system on the enterprise. The Lean
Advancement Initiative's Enterprise Transformation Roadmap was used as a method
for depicting how performance measurement can help enable enterprise
transformation. The implications of research in metrics for enterprise transformation
span across three areas: 1. The extensive literature reviews provide an academic
contribution for performing enterprise and measurement research; 2. A common
language and framework for exploring measurement problems is depicted for
practitioners through the case study analysis; and 3. A connection between enterprise
measurement and enterprise transformation is established to drive future
transformation success.
Whitaker, R. B. (2005) Value Stream Mapping and Earned Value Management: Two
Perspectives on Value in Product Development, Cambridge, MA, Master's thesis, MIT
and LAI.
The concepts of value and value stream are crucial to the philosophy of Lean, and a
better understanding of how these concepts relate to product development (PD) is
essential for the creation of a Lean PD strategy. This thesis focuses on value by looking
at PD processes through two different value perspectives: Product Development Value
Stream Mapping and Earned Value Management. Product Development Value Stream
Maps (PDVSMs) were created for two different PD projects, and the tasks from the
maps were analyzed for how they each create value. The official value measurement
for the two projects, Earned Value Management System data, was analyzed and
compared to the PDVSMs. This comparison of the two value perspectives proved
valuable, as it showed that despite some misalignments, they are congruent. The
comparison also highlighted several flaws in EVMS. A combined EVMS/PDVSM hybrid
management tool is proposed and discussed.
Page 24
Citation &
Download Link
(Blackburn,
2009)
Download Link
(Whitaker,
2005)
Download Link
LAI Whitepaper Series
Paduano, R. (2001) Employing Activity Based Costing and Management Practices
Within the Aerospace Industry: Sustaining the Drive for Lean, Cambridge, MA, Master's
thesis, MIT and LAI.
The case studies contained in the thesis have illustrated several aspects of Activitybased costing and management (ABCM) implementation within one of the largest
facilities of the largest aerospace and defense sector company in the United States.
What started as a tool to support the manufacturing environment by providing a link
between manufacturing performance and financial benefits or losses, has the potential
to migrate to a more strategic role, playing a central theme in the cost management
strategy of the company. Barriers to implementation include poor upper management
support, and continuous resistance in changing the culture of the employees affected
by the implementations, especially those that belong to the finance and accounting
departments. Enablers to widespread implementation are education, whereby the
company’s employees are exposed to the concepts of ABCM, understanding the
benefits to be gained from widespread implementation, and communication.
Continuous feedback provides a situation where the ABCM initiative does not get
classified as another highly publicized and short-lived management optimization
initiative. At the facility level, application of ABCM has shown an improvement in
efficiency, and a reduction of the cost of quality.
(Paduano,
2001)
Download Link
8.1.4 Risk management
This is an activity that makes the key program risks transparent, ensures that the
understanding of risks is closely linked to the decision making process, strives to reduce
uncertainty as much as possible, and account for residual uncertainty and risk with robust
processes and systems.
Publication Title & Synopsis
Oehmen, J. & Rebentisch, E. (2010) Risk Management in Lean PD. LAI Paper Series
"Lean Product Development for Practitioners". Cambridge, MA, LAI and MIT.
The two core challenges of risk management are finding the optimum balance a)
between the cost of carrying risks vs. the cost of mitigating risks and b) between a risk
that is taken with a certain development project and the return that is expected from
the project. This whitepaper summarizes the work conducted at LAI to-date on risk
management. It contains a summary description of the ISO 31000 risk management
process, a discussion of how these process steps can be adapted to development
programs, an overview of risk management methods, as well as a review of past LAI
research on risk management methods and processes, management of uncertainty,
real options theory and portfolio-level risk management.
Oehmen, J., Ben-Daya, M., Seering, W. & Al-Salamah, M. (2010) Risk Management in
Product Design: Current State, Conceptual Model and Future Research. Proceedings of
the ASME 2010 International Design Engineering Technical Conferences & Computers
and Information in Engineering Conference IDETC/CIE 2010
This publication expands the discussion of risk management methods currently
discussed in the literature. It gives an overview of the product development oriented
risk management literature along the seven process steps of 1. Communication and
consultation, 2. Establishing the context, 3. Risk identification, 4. Risk analysis, 5. Risk
evaluation, 6. Risk treatment and 7. Monitoring and review.
Citation &
Download Link
(Oehmen and
Rebentisch,
2010b)
Download Link
(Oehmen et
al., 2010)
Request at LAI
Page 25
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
Oehmen, J. & Ben-Daya, M. (2010) A Reference Model for Risk Management in Product
Development Programs. Research Paper of the MIT-KFUPM Center of Clean Water and
Energy. Cambridge and Dhahran, MIT and KFUPM.
This research report discusses in detail the risk management of complex engineering
programs. It contains an overview of several risk management process frameworks, a
detailed review of the ISO 31000 risk management guidelines, a discussion of how risk
management can be executed in different phases of a program, and how these results
can be integrated, as well as a detailed discussion of possible risks and mitigation
measures for the different phases of a program.
Oehmen, J., Seering, W.: Risk-Driven Design Processes – Balancing Efficiency with
Resilience in Product Design. In: Birkhofer, H. (ed.) The Future of Design Methodology.
Springer, London (2011)
In this book chapter, the authors argue that modern risk management should focus on
achieving four major goals: 1. Create transparency regarding risks; 2. Make decisions
based on risk knowledge; 3. Minimize uncertainty; and 4. Create a resilient
organization. The chapter links those goals to classic risk management processes but
ultimately argues that risk management and program management must become one
overall process.
The following documents are covered by the LAI Risk Management Whitepaper (see
above)
Wagner, C. (2007) Specification Risk Analysis: Avoiding Product Performance
Deviations through an FMEA-based Method. Munich and Cambridge, Technical
University of Munich and LAI.
Meeting specifications during the design phase is crucial for the later success of a
product. The research analyzes the characteristics of this design phase from a risk
management perspective. It develops 24 requirements for a method to manage the
risk of not achieving specifications, and based on these requirements, develops a risk
management tool following the FMEA process. It identifies, assesses, and ranks
product specifications that are challenging to achieve. It avoids product deficiencies
and provides a systematic approach to develop appropriate mitigation measures. Thus,
the method seeks to prevent time and cost-consuming changes at a later point.
Madachy, R. & Valerdi, R. (2010) Automating Systems Engineering Risk Assessment.
Proceedings of the 8th Conference on Systems Engineering Research, Hoboken, NJ,
March 17-19 2010.
This paper describes an automated expert system tool, Expert COSYSMO. It is a
knowledge-based method for systems engineering risk assessment and mitigation. It is
an extension of the COSYSMO cost model which supports PD project planning by
identifying, categorizing, quantifying, and prioritizing system-level risks and project
execution by providing mitigation advice. The knowledge base codifies the experience
of seasoned systems engineering practitioners to identify and quantify risks, and
provide risk mitigation advice for users to help develop their project-specific mitigation
plans.
Oehmen, J. (2005) Approaches to Crisis Prevention in Lean Product Development by
High Performance Teams and through Risk Management. Munich and Cambridge,
Technical University of Munich and LAI.
This thesis reviews the pre-ISO 31000 literature on PD risk management and develops
a process framework similar to that introduced by ISO. The publication also contains an
overview of PD-related risk management methods for every risk management process
step.
Page 26
(Oehmen and
Ben-Daya,
2010)
Request at LAI
(Oehmen and
Seering, 2011)
Request at LAI
(Wagner,
2007)
Download Link
(Madachy and
Valerdi, 2010)
Download Link
(Oehmen,
2005)
Download Link
LAI Whitepaper Series
Browning, T. R., Deyst, J. J., Eppinger, S. D. & Daniel E. Whitney (2002) Adding Value in
Product Development by Creating Information and Reducing Risk. IEEE Transactions on
Engineering Management, 49, 443-458.
This publication develops the “risk value method” to link the probability distribution of
performance outcomes in PD projects with the customer utility function. As design
work proceeds, certainty increases surrounding the ability of the evolving product
design (including its production process) to be the final design (i.e. technical
performance risk decreases). The proposal is that making progress and adding
customer value in PD equate with producing useful information that reduces
performance risk. The risk value method integrates current approaches such as
technical performance measure tracking charts and risk reduction profiles.
Bresnahan, S. M. (2006) Understanding and Managing Uncertainty in Lean Aerospace
Product Development. Master's Thesis. Cambridge, Massachusetts Institute of
Technology.
This thesis explores the role of uncertainty in lean product development, demonstrates
the relationship between risk mitigation activities and the generation of customer
value in the design and development process, and provides guidelines for completing
these activities in a manner that reduces cycle time, assures quality, and makes the
most efficient use of company resources. Product development teams that undertake
aggressive and rigorous activities to identify uncertainties and risk ultimately
encounter fewer problems and unplanned rework. These teams complete their project
at an overall lower cost than the shortsighted teams who spend less to address
uncertainty and risk, but meet greater problems later in the process. The thesis
contains a list of recommended risk-related criteria for different stage gates.
Mikaelian, T. (2009) An Integrated Real Options Framework for Model-based
Identification and Valuation of Options under Uncertainty. Cambridge, Massachusetts
Institute of Technology.
This PhD thesis focuses on flexibility as an important means of managing uncertainties
and leverages real options analysis that provides a theoretical foundation for
quantifying the value of flexibility. Complex systems and enterprises, such as those
typical in the aerospace industry, are subject to uncertainties that may lead to
suboptimal performance or even catastrophic failures if unmanaged. This work
introduces an Integrated Real options Framework (IRF) that supports holistic decision
making under uncertainty by considering a spectrum of real options across an
enterprise.
Wirthlin, J. R., Seering, W. & Rebentisch, E. (2008) Understanding Enterprise Risk
Across an Acquisition Portfolio: A Grounded Theory Approach. Seventh National
Symposium on Space Systems Engineering & Risk Management, Los Angeles, CA,
February 26-29, 2008.
This publication analyses the current state of the art regarding portfolio level risk
management in Air Force acquisition programs. Data collected from portfolio
managers working at multiple levels of the system suggest that most are unable to
articulate the risk carried by their portfolio of product development activities or what
this means to them. However, the same interviews suggest they strongly desire this
capability. From a review of the applicable literature in the areas of risk, product
development (acquisition) and product portfolio management, portfolio-level risk
applications are found to be sparse and ill-conceived. While portfolio leaders are
expected to live within the resources available, they have few effective levers of
control to influence portfolio performance.
(Browning et
al., 2002)
Download Link
(Bresnahan,
2006)
Download Link
(Mikaelian,
2009)
Download Link
(Wirthlin et
al., 2008)
Download Link
Page 27
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
8.1.5 Project management
The activities include managing the execution of and transition between the different phases
of the program, as well as the management of scope, time, budget and quality of the program
(see Table 2 for a detailed summary).
There is no specific LAI work in project management as such. The following references are to
two of the standard project management processes, the Project Management Institute (PMI)
guidelines, as well as the PRINCE2 project management framework.
Publication Title & Synopsis
Project Management Institute (2008): A guide to the project management body of
knowledge (PMBOK guide), Drexel Hill, PA.
The PMI guidelines address the following topics: Project Life Cycle and Organization;
Project Management Processes; Project Integration Management; Project Scope
Management; Project Time Management; Project Cost Management; Project Quality
Management; Project HR Management; Project Communication Management; Project
Risk Management; and Project Procurement Management.
Office of Government Commerce (2009): Managing Successful Projects with PRINCE2,
London, The Stationary Office.
The PRINCE2 project management guidelines contain the following topics: Project
Management Principles; Project Management Themes; Starting up a project; Initiating
a project; Directing a project; Controlling a stage; Managing stage boundaries;
Managing product delivery; and Closing a project.
Citation &
Download Link
(PMI, 2008a)
Link to PMI
(OGC, 2009)
Link to OGC
8.1.6 Multi-project coordination
In addition to the challenges of managing a single project, program offices execute several
projects in parallel, as well as contractors and suppliers on the different levels also usually
have more than one project that they work on in parallel. This generates specific challenges,
such as the allocation of resources, especially the staffing of the different projects in parallel
along their specific life cycles.
Publication Title & Synopsis
Herweg, G. M. & Pilon, K. E. (2001) System Dynamics Modeling for the Exploration of
Manpower Project Staffing Decisions in the Context of a Multi-Project Enterprise,
Master Thesis, LAI and Massachusetts Institute of Technology. and:
Wright, M. R. (2003) Strategies for dealing with instabilities in a complex, multi-project
product development system engineering environment, Master Thesis, LAI and
Massachusetts Institute of Technology.
Project decisions may be made on a project to project basis and often neglect to
account for the complex interactions that exist between projects. Total workload and
resource-usage are often not stable. In most environments, resources are limited, and
thus have to be dynamically allocated according an ever-changing overall project
situation. Often the current decision process results in a less than optimal return to the
corporation. This return may not only be measured financially, but also in terms of
organizational capability or intellectual capital. These works explore the effects of
manpower decisions as well as other decisions in a multiple engineering project
setting. Based on case studies, system dynamics models were developed to explore
Page 28
Citation &
Download Link
(Herweg and
Pilon, 2001)
Download Link
(Wright, 2003)
Download Link
LAI Whitepaper Series
different options in regard to decision making in a multi-project environment. The
recommendations are: 1. Short-term: 1.1 Limit overtime; 1.2 Keep project priorities
stable; 1.3 Outsourcing; 2. Mid-term: 2.1 Develop detailed plans to ramp-up and rampdown staffing levels at the begin and end of projects; 2.2 Cancel troubled projects early
in their lifecycle; 3. Long-term: 3.1 Quality improvement; 3.2 Assess new projects also
based on the intellectual capital that they generate; 3.3 Schedule buffer between
projects
Page 29
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
8.2 LAI’s Insights: Enterprise Management
8.2.1 Developing intellectual capital
Companies and government organizations must provide incentives to their employees to
develop deep technical knowledge and expertise in relevant areas. This includes encouraging
and rewarding life-time careers as technical experts, for example through technology-oriented
career towards senior engineers.
Publication Title & Synopsis
Davidz, H. L. (2006) Enabling Systems Thinking to Accelerate the Development Senior
Systems Engineers, Cambridge, MA, PhD Thesis, LAI and MIT.
As engineering systems become more complex, the roles involved in developing and
managing such systems also become more complex. Thus, there is increasing interest
in educating and training engineering professionals to think more systemically. In
particular, there is an increasing need to accelerate the development of senior systems
engineers. In a field study, 205 interviews were conducted in 10 companies. Senior
systems engineers were studied, as well as proven stellar systems thinkers
interviewed. The best practices for industry to develop systems engineers include: 1.
structure systems thinking interventions to emphasize experiential learning; 2. offer
systems programs to teach systems skills and systems thinking; 3. filter and foster
identified individual characteristics in systems organizations; 4. provide an
environment supportive to the development of systems thinking; and 5. clearly
communicate how strength of systems thinking is assessed.
Herweg, G. M. & Pilon, K. E. (2001) System Dynamics Modeling for the Exploration of
Manpower Project Staffing Decisions in the Context of a Multi-Project Enterprise,
Master Thesis, LAI and Massachusetts Institute of Technology.
This work has already been discussed in section 8.1.6. It is also relevant to the
development of intellectual capital: project experience is an important contributor to
the skill development of the workforce. A successful long-term staffing strategy
therefore has to balance direct project objectives (return on investment) with an
overall, project-spanning goal of developing and maintaining a highly skilled workforce.
Andrew, W. G. (2001) Do Modern Tools Utilized in the Design and Development of
Modern Aircraft Counteract the Impact of Lost Intellectual Capital within the
Aerospace Industry?, Cambridge, MA, Master Thesis, LAI and MIT.
The new design experience of Post-World War II Aerospace Engineers was
approximately 6-12 new design aircraft per career. In contrast, an aerospace engineer
starting his career today may experience only one, maybe two new aircraft designs
during their career. Anecdotal evidence has been published linking this trend to
problems experienced in many recent aircraft programs. Counter arguments cite rapid
advances in design, manufacturing and information technologies have compensated
for some or all of declining experience base. An analysis of the program performance
metrics revealed that the predecessor programs indeed outperformed the more recent
programs. Examination of the case studies showed a strong correlation between
intellectual capital metrics and the performance metrics for programs. This reemphasizes the importance that must be placed on developing a skilled workforce.
Siegel, L. R. (2004) Measuring and Managing Intellectual Capital in the U.S. Aerospace
Industry, Cambridge, MA, Master Thesis, LAI and MIT.
Intellectual capital consists of knowledge-based assets – including people,
relationships, tools, and processes – that create value for a firm and its clients. In this
research, a new conceptual framework is developed for understanding the role of
Page 30
Citation &
Download Link
(Davidz, 2006)
Download Link
(Herweg and
Pilon, 2001)
Download Link
(Andrew,
2001)
Download Link
(Siegel, 2004)
Download Link
LAI Whitepaper Series
intellectual capital in new product development. The framework develops a dynamic
model of the three forms of intellectual capital – human capital, structural capital, and
relational capital – and identifies mechanisms for knowledge transfer, organizational
learning, and value creation. The framework is based on case studies of seven product
development projects at different US aerospace firms. The thesis contains a selfassessment tool that managers can use to measure and assess the health of their
intellectual capital base.
Forseth, C. E. (2002) The Pursuit of Acquisition Intrapreneurs, Cambridge, MA,
Research Report, LAI and MIT.
This research focused on identifying Acquisition Intrapreneurs, viewed and defined for
the purpose of this research as, individuals within the acquisition profession who take
direct responsibility for turning ideas into products through assertive risk taking. A
comprehensive survey was conducted and asked the acquisition workforce to respond
to a number of statements regarding their current jobs, opportunities for
advancement, future job interests and supervisor. Short term recommendations for
increased innovation include specific funding and reporting for PCOs and engineers. A
long term plan to develop future acquisition leaders, CGOs, is proposed through a
framework for agile acquisition Jobs, Environment, Training and Support. This
recommendation includes operational tours, an environment of failure-tolerant
leadership, localized training through base-level Acquisition Center of Excellence
offices and a rewards structure similar to the operational “W”, or whiskey slots. The
proposal tied the survey results and research theories together by providing a system
to recognize, train, track and reward acquisition intrapreneurs.
Lamb, C. M. T. (2009) Collaborative Systems Thinking: An exploration of the
mechanisms enabling team systems thinking, Cambridge, MA, PhD thesis, LAI and MIT.
Within the aerospace industry, an aging workforce places those with the most systems
experience near retirement at a time when fewer new programs exist to provide
systems experience to the incoming generation of aerospace engineers and leaders. It
is therefore important to look at teams of aerospace engineers as a new unit of
systems knowledge and thinking. By understanding more about how teams engage in
collaborative systems thinking (CST), organizations can better determine which types
of training and intervention will lead to greater exchanges of systems-level knowledge
within teams. Based on numerous interviews, case studies and surveys, a model was
developed to differentiate collaborative systems thinking (CST) teams from non-CST
teams. These are 1. The right team balance between individual and consensus decision
making; 2. The median number of past program experiences on a team; 3. Sufficient
real-time group interactions; 4. An overall creative environment; and 5. A realistic
program schedule.
(Forseth,
2002)
Download Link
(Lamb, 2009)
Download Link
Page 31
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
8.2.2 Building and transforming the program enterprise
Architecting the enterprise of both public (government) and private (companies) organizations
that are involved in executing the program and creating stakeholder value. Transforming
existing enterprises along an enterprise transformation roadmap to re-align them with
stakeholder needs and optimize the value flow throughout the enterprise.
Publication Title & Synopsis
Nightingale, D., Srinivasan, J.: Beyond the Lean Revolution: Achieving Successful and
Sustainable Enterprise Transformation. AMACOM, New York (2011)
This book summarizes the past LAI research on enterprise transformation. Enterprise
transformation begins with the big picture: What are the strategic objectives? How is
the enterprise per forming against those objectives? How should it be? Who are the
stakeholders and what do they value? Then it moves forward toward an audacious
vision of the enterprise's future. This book provides a roadmap for achieving
sustainable, bottom-line results, delivering value to stakeholders, and reaching that
future vision. The main process steps that are covered are: ensure senior leadership
commitment; assess the enterprise's current state; analyze stakeholder values;
develop a future vision; and, create a plan for transformation. From inception to
implementation, this book provides a framework for bridging the gap from mere
change to genuine transformation.
Nightingale, D., Stanke, A. & Bryan, F. T. (2008) Enterprise Strategic Analysis and
Transformation (ESAT) - Version 2.0, Cambridge, MA, LAI Guide.
The Enterprise Strategic Analysis and Transformation (ESAT) methodology provides a
means for the senior leadership team to understand their enterprise, create an
actionable vision for the future, plan the transformation and govern the execution. The
emphasis on enterprise thinking forces enterprises to transition from local lean efforts
to Lean as an enterprise philosophy and is at the core of the ESAT transformation
method. The Enterprise Strategic Analysis and Transformation (ESAT) methodology
presented in this guide serves as an integrated, analytical framework for diagnosing
and improving overall enterprise performance. The emphasis of ESAT on
understanding the enterprise value streams, the value flow between key stakeholders
and the enterprise, and interactions both within and across the enterprise, enables the
identification of enterprise wastes and opportunities for improvement. Using the
quantitative and qualitative data gathered as part of executing the ESAT, the
enterprise leadership team can then create a future state vision, an actionable
transformation plan, and put into place a governance structure to support and drive
enterprise transformation. Equally important is the shared mental model that the
senior leadership team creates during the execution of the ESAT, as it ensures that
there is sustained leadership buy-in for the long transformation journey ahead.
Stanke, A., Nightingale, D. & Bryan, F. T. (2008) Enterprise Strategic Analysis and
Transformation (ESAT) - Facilitator’s Guide - Version 2.0, Cambridge, MA, LAI Guide.
The ESAT facilitator’s guide is an accompanying document to the general ESAT guide
discussed above. It provides an overview of the ESAT process, but it is assumed that all
facilitators are comfortably familiar with the process as described in the ESAT Guide
document and the ESAT KEE Modules (both instructional and facilitation). This
Page 32
Citation &
Download Link
(Nightingale
and
Srinivasan,
2011)
Order online
(Nightingale et
al., 2008)
Download Link
(Stanke et al.,
2008)
Download Link
LAI Whitepaper Series
document provides a general introduction to the facilitation techniques suggested for
ESAT. It also provides a step-by-step outline for taking a team through the analysis. For
each ESAT event/major time block, there is an overview and a detailed description in
the following format. Throughout the guide are instructions for particular templates as
well as examples of illustrative cases.
Stanke, A. K. (2006) Creating High Performance Enterprises, PhD Thesis, LAI and
Massachusetts Institute of Technology.
There are a small number of exceptional programs that have managed to meet all of
their commitments in terms of cost, schedule, and technical performance
simultaneously. This work explores these programs on a case-by-case basis. Best
practices from these programs are organized into a framework of factors that
differentiate these high performing programs from the majority of others that struggle
in the same environment. These factors are: 1. Distributed Leadership Action: 1.1
Boundary spanning activity across organizations in the enterprise; 1.2Developing and
utilizing a social network; 1.3 Developing and sustaining extensive customer
interactions; 1.4 Fostering and maintaining personal accountability of plans and
outcomes; 2. Informal Structures: 2.1 Boundary spanning activity with the enterprise
environment; 2.2 Encouragement for open information sharing; 2.3 Veteran core
group to institutionalize behavior; 3. Formal Structures: 3.1 Balanced risk through work
share and teaming arrangements; 3.2 Common contract structure; 3.3 Standardized
program management practices (metrics and reporting system)
Glazner, C. G. (2006) Enterprise Integration Strategies Across Virtual Extended
Enterprise Networks: A Case Study of the F-35 Joint Strike Fighter Program Enterprise,
Cambridge, MA, Master's thesis, LAI and MIT.
Increasingly, integration efforts have moved beyond the boundaries of the core or
focal enterprise serving as the prime contractor or system integrator to span the entire
value chain, to form virtual extended enterprises. While integration offers many
benefits to enterprises, a high degree of integration is not always desirable or
advantageous in a limited duration virtual extended enterprise composed of
autonomous companies. The objective of this research is to explore the extent to
which a focal enterprise, such as a prime contractor or system integrator, should
consider integration across its virtual extended enterprise, identify major barriers to
integration, and define key enablers of integration overcoming these barriers. Analysis
focuses on the extent of integration based on the characteristics of the virtual
extended enterprise, such as the duration and scope of the program in question,
product system architecture, the organizational architecture, and the external
environment. In particular, three key conceptual dimensions of integration are
developed and explored—technological integration, strategic integration, and
organizational integration. This framework is applied in an in-depth case study of
integration strategies on the virtual extended enterprise of the F-35 Joint Strike Fighter
(JSF) Program. The knowledge gained from the case study is used to make
recommendations for the development of integration strategies for future programs.
Oehmen, J. & Rebentisch, E. (2010a) Compilation of Lean Now! Project Reports,
Cambridge, MA, LAI Report.
(Stanke, 2006)
Download Link
(Glazner,
2006)
Download Link
(Oehmen and
Rebentisch,
2010a)
Download Link
The aim of Lean Now was to focus on the process interfaces between these enterprise
stakeholders to eliminate barriers that impede progress. Any best practices developed
would be institutionalized throughout the Air Force and the Department of Defense.
The industry members of LAI agreed to help accelerate the government-industry
transformation by donating lean Subject Matter Experts (SMEs) to mentor, train, and
facilitate the lean events of each enterprise. This compilation contains project briefings
Page 33
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
illustrating the positive effects of Lean Now! in the following programs / processes: 1.
F/A-22, 2. F-16, 3. Global Hawk, 4. Turbine Engine Development, and 5. Purchase
Request Process. In addition, the compilation contains the Lean Now! workshop
material.
Jobo, R. S. (2003) Applying the Lessons of “Lean Now!” To Transform the US Aerospace
Enterprise - A study guide for government lean transformation, Cambridge, MA, LAI
Report.
This research report summarizes the key learnings from the Lean Now initiative. From
the observations of three Lean Now prototype programs, a common methodology for
implementing lean can be developed. This methodology has three distinct phases: 1)
Set-up Phase, 2) Planning Phase, 3) Execution and Follow-through. Within each phase
are distinct steps that must occur in order for the lean initiative to be successful.
(Jobo, 2003)
Download Link
8.2.3 Understanding the program enterprise
Assessing existing program enterprises to benchmark them against best practices and/or other
program enterprises, as well as to uncover areas for improvement.
Publication Title & Synopsis
LAI (2001) Lean Enterprise Self-Assessment Tool (LESAT) Version 1.0.
The Lean Enterprise Self-Assessment Tool helps organizations to determine their
progress and maturity during a lean transformation. This document describes the
LESAT tool and provides a structure and implementation reference for the selfassessment process. Extensive field-testing in more than 20 companies in both
countries demonstrated the tool’s utility, effectiveness and ease of use. It addresses
the lean-related maturity of an organization in the following areas: Section I – Lean
Transformation/Leadership: I.A Enterprise Strategic Planning (3 Lean practices); I.B
Adopt Lean Paradigm (4 Lean practices); I.C Focus on the Value Stream (4 Lean
practices); I.D Develop Lean Structure and Behavior (7 Lean practices); I.E Create and
Refine Transformation Plan (3 Lean practices); I.F Implement Lean Initiatives (2 Lean
practices); I.G Focus on Continuous Improvement (5 Lean practices); Section II – LifeCycle Processes: II.A Business Acquisition and Program Management (4 Lean practices);
II.B Requirements Definition (2 Lean practices); II.C Develop Product and Process (3
Lean practices); II.D Manage Supply Chain (3 Lean practices); II.E Produce Product (2
Lean practice); II.F Distribute and Service Product (4 Lean practices); Section III –
Enabling Infrastructure: III.A Lean Organizational Enablers (5 Lean Practices); III.B Lean
Process Enablers (3 Lean Practices)
Nightingale, D. J. & Mize, J. H. (2001) Development of a Lean Enterprise
Transformation Maturity Model. Information, Knowledge, Systems Management, 3,
15-30.
This journal publication provides background information to the Lean Enterprise SelfAssessment Tool (LESAT). It explains its integration into an enterprise transformation
process, discusses a number of other approaches to organizational assessment,
explains the rationale behind the design of the LESAT tool and gives a brief
introduction to the tool itself..
Ferdowsi, B. & Stanke, A. (2002) F-16 Case Study Report - Lean Effects on Aerospace
Programs (LEAP) Project, Cambridge, MA, LAI Report.
This case study describes the F-16 Enterprise. The continuous, evolutionary
development strategy of the F-16 has substantially increased the functionality of the
Page 34
Citation &
Download Link
(LAI, 2001)
Download Link
(Nightingale
and Mize,
2001)
Request at LAI
(Ferdowsi and
Stanke, 2002)
Request at LAI
LAI Whitepaper Series
system while the dimensions of the airframe have remained constant. This report
follows the F-16 journey of improvement over the last decade. Since an increase in
focus on cost and quality in the early 1990s, there has been gradual but continuous
progress along this journey. Most remarkable are some of the more recent changes,
following the implementation of lean principles and practices across the program.
there has been less than three percent increase in the price of the Contractor
Furnished Equipment portion of the aircraft in the last ten years. Not only has the price
been kept nearly constant, but Lockheed Martin also have a record of 100+ months of
consecutive on-time deliveries. They have developed and produced over 100 different
type versions of this aircraft (i.e. tailored configurations) for their various customers.
All of these achievements have been made despite fluctuating demand heavily
contingent on international sales and a substantial drop in production rate from 200+
to 24 aircraft per year.
Ferdowsi, B. & Haggerty, A. (2002) 737 Fuselage Case Study Report - Lean Effects on
Aerospace Programs (LEAP) Project, Cambridge, MA, LAI Report.
This case study describes the Boeing 737 “Next Generation” Enterprise. The plant
ramped up production from 10 to 28 planes per month, which in itself is remarkable,
but in this time, several other lean principles were set in motion, leading to great
improvements. For example, flow time in the factory was reduced by 21 percent, while
creating capacity to add work content from the Boeing final assembly plant in Renton,
WA. Furthermore from 1998 to 2000, labor hours per unit were reduced by nearly 50
percent. Unit cost was reduced by 25 percent over the same period.
Davidz, H. & Nightingale, D. (2002) AMRAAM Case Study Report - Lean Effects on
Aerospace Programs (LEAP) Project, Cambridge, MA, LAI Report.
This case study describes the production of the AMRAAM-120. One of the most
interesting observations is the cooperative relationship Raytheon has with their U.S.
Government customer. The accomplishments of the AMRAAM program were made
possible by using lean methods to develop an improved, trusting relationship with
their U.S. Government customer. With strong, insightful leadership from both the
contractor and the government, this cooperation enabled an even higher level of lean
and even more cost reduction. The Agile program and Raytheon Six Sigma have
created a culture where the six sigma/lean tools are institutionalized. Strong
leadership at all levels has provided the motivation and support for continued change.
People are empowered at all levels, and there is teamwork across the organization.
Derleth, J., Hitchings, S. & Rebentisch, E. (2003) Atlas Case Study Report - Lean Effects
on Aerospace Programs (LEAP) Project, Cambridge, MA, LAI Report.
This case study describes the Atlas expendable launch vehicle. Along with its successful
launch performance, the Atlas program has also achieved significant business and
operating success, which this case study attempts to document. Due to different
market factors, Lockheed Martin not only needed to reduce its costs, but also to
increase its capacity to produce Atlas launchers, as well as a 50% reduction in order-todelivery cycle time. Lockheed Martin achieved these goals by implementing lean
principles that allowed them to reduce cycle time, man hours needed for assembly,
overhead costs, and other forms of waste.
Stanke, A. K. (2006) Creating High Performance Enterprises, PhD Thesis, LAI and
Massachusetts Institute of Technology.
This thesis contains three relevant case studies: The F/A-18E/F case study on the
(Ferdowsi and
Haggerty,
2002)
Download Link
(Davidz and
Nightingale,
2002)
Request at LAI
(Derleth et al.,
2003)
Request at LAI
(Stanke, 2006)
Download Link
Page 35
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
robust enterprise, the JDAM case study on the agile enterprise, as well as the case
study on the X-35 as an example for a flexible enterprise (see section 0 for more detail
on this publication).
Cowap, S. A. (1998) Economic Incentives in Aerospace Weapon Systems Procurement,
Master Thesis, LAI and Massachusetts Institute of Technology.
A number of incentive-related best practices are benchmarked in this thesis. It
contains case studies on two Air Force ammunition programs, C-130J case study, as
well as a case study on the C-17 program. Please refer to section Error! Reference
ource not found. for more details.
Labedz, C. S. & Harvey, R. K. (2006) Letterkenny Army Depot: Finance Innovations
Support Lean Six Sigma Success.
(Cowap, 1998)
Download Link
(Labedz and
Harvey, 2006)
Download Link
As a result of significant dollar savings to the Army and U.S. taxpayers, Letterkenny
Army Depot received widespread public recognition in 2005. The depot received a
public sector Shingo Prize for applying Lean principles and tools to its PATRIOT missile
system recapitalization program. While Letterkenny was Leaning its production
systems, the depot implemented two innovative and effective financial incentive
systems: one to reward employees, the other to reward customers. The reward
systems were innovative because they occurred in a not-for-profit organization and
effective because they motivated customers, employees, and unions to embrace Lean.
The case describes the organizational conditions leading to these innovations and the
responses to them among its customers, unions and headquarters.
8.2.4 Learning and improving in the program enterprise
Successfully managing continuous improvement and organizational change in complex
enterprises.
Publication Title & Synopsis
Bozdogan, K. (2010) Towards An Integration Of The Lean Enterprise System, Total
Quality Management, Six Sigma And Related Enterprise Process Improvement
Methods, Cambridge, MA, MIT LAI / ESD Working Paper (ESD-WP-2010-05), scheduled
for publication in Encyclopedia of Aerospace Engineering.
In recent years, a growing number of firms in many industries have adopted Lean
Thinking, Six Sigma and related enterprise process improvement initiatives to achieve
continuous performance improvement and, often, to bring about enterprise-wide
change and transformation. This paper presents a critical comparative review of these
initiatives to identify the similarities and differences among them, highlight how they
relate to each other, and assess their strengths and weaknesses. The paper strives to
provide an improved understanding of these initiatives to enable enterprises develop
better-informed change and transformation strategies. More specifically, the paper
focuses on Lean Thinking, Total Quality Management (TQM), Six Sigma, Theory of
Constraints (TOM), Agile Manufacturing, and Business Process Reengineering. A main
conclusion of the paper is that Lean Thinking provides an overarching intellectual
architecture for the various systemic change initiatives, wherein they share common
roots, augment each other in significant ways, and generally comprise mutuallycomplementary approaches. The various differences among them are dwarfed by their
common characteristics. Taken together, under the overall umbrella of Lean Thinking,
they are rapidly merging into a unified framework for bringing about fundamental
enterprise-wide change.
Page 36
Citation &
Download Link
(Bozdogan,
2010)
Download Link
LAI Whitepaper Series
Roth, G. (2008) The Order and Chaos of the Learning Organization. In Cummings, T.
(Ed.) Handbook of Organization Development. Newbury, CA, Sage.
This book chapter summarizes the most important perspectives on organizational
learning: 1. Organizational learning as accumulated experience; 2. Organizational
learning as adaptation to environmental changes; 3. Organizational learning as
assumption sharing; and 4. Organizational learning as developing the knowledge base.
The different concepts are integrated with the broader field of management research
into into a framework of the Learning Organization that integrates diverse concepts
such as team organization, management by objectives, strategic planning, value chain
management, quality management , core competencies and cultural change. The goal
is to develop an organization that continually expands its capacity to create its own
future.
Roth, G. (2006) Distributing Leadership Practices for Lean Transformation. Reflections The SoL Journal on Knowledge, Learning, and Change, 7, 15-31
Many organizations have achieved impressive results in various aspects of their
business through lean transformation. Few firms, however, sustain those initial results,
and many struggle to bring the results down to a bottom-line impact. This article links
research literature on change management with lean case studies and presents a form
of distributed leadership that facilitates lean transformation. Distributing leadership
practices is one of five capabilities identified for successful lean enterprise change: 1.
Rethinking organizational boundaries; 2. Installing innovation sets; 3. Pushing and
pulling change; 4. Seeking growth opportunities; and 5. Distributing leadership
practices
Deardorff, S. J. K. (2003) Institutionalizing Change in Aerospace Process and Product
Settings, Cambridge, MA, Master's thesis, MIT and LAI.
This thesis is an examination of the methods used to introduce and sustain change to
help answer the reasons why some organizations are successful at adapting and some
are not. In order to affect beneficial change, strategies to promote transformations
must recognize the political, economic, and social issues, as well as the capabilities of
the organizations. The research is based on four separate case studies of organizations
that produce, or manage the delivery of, a technically complex good or service.
More commitment from lower level leadership and a wider availability of best practice
documentation corresponds to less regression to former practices. More formal
training led to wider diffusion of change, but availability of documentation did not
correspond to greater diffusion or adoption of new practices. However, better
availability of documentation did correspond to less regression. The strongest defenses
against regression were not supporting the old process or making permanent or semipermanent physical changes to the work area.
The recommendations include using small, organic change offices in business areas
that do not have a history of mature change practices to use as models, setting aside a
percentage of realized savings for human resources investment, and time-in-grade
provisions for management positions to allow for greater continuity.
Cohen, J. L. (2005) United States Air Force Air Logistics Centers: Lean Enterprise
Transformation and Associated Capabilities, Cambridge, MA, Master's thesis, MIT and
LAI
Lean enterprise transformation entails a complementary set of initiatives and efforts
executed over a substantial period of time, in a consistent and coordinated manner, at
all levels of the enterprise. It builds upon ordinary organizational change in that a
broader set of people and functions will be affected, and non-traditional approaches
and mental models will continue to be exercised. Between 2003 and 2005, three US Air
Force Air Logistics Centers (ALCs) initiated lean enterprise transformation efforts. A set
(Roth, 2008)
Request at LAI
(Roth, 2006)
Request at LAI
(Deardorff,
2003)
Download Link
(Cohen, 2005)
Download Link
Page 37
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
of 12 lean enterprise change capabilities is proposed: 1. A leadership team with a
shared mental model; 2. Planning and implementation process; 3. Balanced and
cascading system of metrics; 4. Standardized processes and a team to track them; 5.
Compensation and reward system; 6. Developing and deploying vision and mission; 7.
Communication and public relations abilities; 8. Change management; 9. Cadre of
change agents; 10. Context for local experimentation; 11. Continued learning and selfrenewal; and 12. Training and education.
Shoepe, T. V. (2006) In Pursuit of Understanding Lean Transformation - Capturing Local
Change Journeys in a DoD Field Environment, Cambridge, MA, LAI Research Paper.
Guided by the Secretary of Defense, both the USAF and USN have adopted “lean”
principles as a compass to help guide the transformational journeys of their acquisition
programs. This study describes two of the US Air Force and US Navy enterprise-level
continuous process improvement programs and the ways in which each contribute to
local results in a field case setting. Additionally, this study tests the applicability of
ongoing LAI lean enterprise change theory in the context of a DoD environment. Three
noteworthy results were discovered: 1. Both the Navy and USAF have been using lean
principles for years. However, they have traditionally been limited to the
manufacturing floors and USAF Air Logistics Centers. 2. On the foundation of these
successes, both organizations are placing the strategic leadership, vision,
infrastructure, and processes in place to proliferate continuous improvement
throughout their enterprises. 3. With minor tailoring, the ongoing change management
theory development at LAI is applicable to evaluating enterprise change in the context
of a DoD field case.
Hemann, J. M. (2005) Improving Complex Enterprises with System Models, Cambridge,
MA, Master's thesis, MIT and LAI.
Air Force sustainment operations are the focus of an intensive internal effort to
improve performance and reduce costs. Past improvement initiatives have often failed
to produce the intended results. Exploratory research was conducted at an Air Logistics
Center to study how improvements are executed. Two conclusions are drawn from this
research. The first is that changing sustainment operations is a problem of high
dynamic and behavioral complexity. The second conclusion is that system models are
well suited to coordinating change because they provide insight into how a
complicated system can be managed and improved. Three key findings support these
conclusions. First, there is significant correlation between categories of unavailable F16 aircraft such that reductions in one category are associated with increases in
another. Second, an analysis of change efforts in two parts of the ALC shows that
systemic influences, such as the inability to reinvest in improvements, are hindering
change initiatives in one part of the ALC. The third finding is that a model of
sustainment operations suggests that independent improvement initiatives are
outperformed by coordinated efforts driven with an understanding of systemic
interactions.
Page 38
(Shoepe,
2006)
Download Link
(Hemann,
2005)
Download Link
LAI Whitepaper Series
8.3 LAI’s Insights: Scoping, planning & contracting
During Scoping, Planning & Contracting, the user needs are translated into a legal contract
between the program office and the main contractor. As part of this, the life cycle strategy for
the entire program is planned and codified, spanning system development, technology
transition, operational use and sustainment, as well as future technology refresh and ultimate
system disposal. This step includes the following activities:
8.3.1 Definition of stakeholder needs and requirements
The stakeholders in large programs are numerous and varied, and so are the resulting needs.
Stakeholders from different areas – government, congress, users, general public, contractors
etc. – and different levels of the hierarchy – top management to shop floor – have to be taken
into account. This requires a sophisticated process for eliciting and aligning these diverse
needs.
Publication Title & Synopsis
Rebentisch, E. (1996) Preliminary Observations on Program Instability, Cambridge, MA,
LAI Whitepaper LEAN 96-03.
This white paper provides details about research on program instability. Its objective is
to discuss high-level findings detailing: 1) the relative contribution of different factors
to a program’s overall instability; 2) the cost impact of program instability on
acquisition programs; and 3) some strategies recommended by program managers for
overcoming and/or mitigating the negative effects of program instability on their
programs.
The government managers of military acquisition programs rated annual budget or
production rate changes, changes in requirements, and technical difficulties as the
three top contributors, respectively, to program instability. When asked to partition
actual variance in their program’s planned cost and schedule to each of these factors,
it was found that the combined effects of unplanned budget and requirement changes
accounted for 5.2% annual cost growth and 20% total program schedule slip.
Program management practices involving the integration of stakeholders from
throughout the value chain into the decision making process were rated the most
effective at avoiding program instability. The use of advanced information technologies
was rated the most effective at mitigating the negative impact of program instability.
Dare, R. E. (2003) Stakeholder Collaboration in Air Force Acquisition: Adaptive Design
Using System Representations. Cambridge, MA, PhD thesis, LAI and MIT.
Air Force development of new or evolutionary weapon systems is a complex endeavor
due to the involvement of many stakeholders and the presence of considerable
uncertainty in the acquisition environment. The ability to adapt a weapon system while
it is still being designed affords a means to respond to this complexity. Eight case
studies were conducted on Air Force development programs. Data were collected on
collaborative practices and patterns of adaptability demonstrated during design. The
research placed an emphasis on usage of “system representations” such as prototypes
and beta software releases that acted as a form of boundary object to facilitate
knowledge sharing across organizational boundaries. This research indicates that the
pressing need for Air Force programs to be able to adapt in today’s uncertain
acquisition environment can be addressed to a significant degree through the usage of
effective system representations in conjunction with supporting patterns of
Citation &
Download Link
(Rebentisch,
1996)
Download Link
(Dare, 2003)
Download Link
Page 39
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
stakeholder interaction. Specific recommendations for Air Force acquisition policy
makers and practitioners are: 1. Make system representations and adaptability part of
acquisition planning; 2. Involve the operational user in the design phase; 3. Create
effective system representations; 4. Make effective use of system representations; and
5. Create a “zone of novelty”, i.e. a mix of flexibility and structure for the program.
Wirthlin, J. R. (2000) Best Practices in User Needs / Requirements Generation,
Cambridge, MA, Master’s thesis, LAI and MIT.
This research develops a process framework for the early phases of product
development, from the initial recognition of a customer need to the decision launch a
development program. The framework focuses on the development of the
requirements that are needed to decide the business case. Detailed case studies of
military and commercial organizations, as well as within the Air Force, were conducted
and best practices in the early product development phases identified. These include
identification of requirements, screening of requirements, concept development,
business case development, as well as organizational and business enablers. Also
included is an instrument to assess the maturity of an organization’s early phases of its
product development process.
Walton, M. A. (1999) Identifying the Impact of Modeling and Simulation in the
Generation of System Level Requirements, Cambridge, MA, Master's thesis, LAI and
MIT.
Requirements generation is an influential time in the evolution of the program. It
allocates 70% of the life-cycle cost of a program and is responsible for a large
percentage of the system errors and cost overruns. This research develops a
framework to describe the current state of requirements generation and then focuses
on the use of modeling and simulation within the process. It is shown that although
modeling and simulation tools are being used extensively in requirements generation
in many programs throughout the DoD, their effectiveness is largely undocumented
and areas of high leverage are unknown. Research results also indicate that a more
effective use of modeling and simulation within requirements generation could be
achieved with increased tool interoperability and easier tool validation and
verification. Finally, the ability to perform more iterations early and the benefits as a
boundary-spanning tool for communication are discussed as the two main benefits of
modeling and simulation.
Gillespie, D. M. (2009) Mission Emphasis and the Determination of Needs for New
Weapon Systems, Cambridge, MA, PhD thesis, LAI and MIT.
Efforts to understand the determination of needs of new weapon systems must take
into account inputs and actions beyond the formally documented requirements
generation process. This study analyzes three recent historical cases of fighter aircraft
development to identify decisions made independently from the documented
requirements process, about the need for new systems. The primary inputs to those
decisions are identified, and a qualitative model for understanding the undocumented
inputs, and their role in determining weapon system needs, is presented. By analyzing
data across the cases, which span a period of significant change in fighter design, the
concept of a Dominant Mission Emphasis (DME) is introduced. The DME is defined as
that mission which receives the most emphasis from the majority of participants in the
needs determination process, and which the majority of other missions support, either
directly or indirectly. Cues are identified which suggest the need to re-examine the
DME. The strength of a DME can be measured by qualitative and quantitative
indicators, including such things as verbal statements, military doctrine, intellectual
and academic writings, organization within the military, resources committed, and
Page 40
(Wirthlin,
2000)
Download Link
(Walton,
1999)
Download Link
(Gillespie,
2009)
Download Link
LAI Whitepaper Series
promotion decisions.
Cameron, B. G., Crawley, E. F., Loureiro, G. & Rebentisch, E. S. (2008) Value
flowmapping: Using networks to inform stakeholder analysis. Acta Astronautica, 62.
Stakeholder theory has garnered significant interest from the corporate community,
but has proved difficult to apply to large government programs. A detailed value flow
exercise was conducted to identify the value delivery mechanisms among stakeholders
for the current Vision for Space Exploration. We propose a method for capturing
stakeholder needs that explicitly recognizes the outcomes required of the value
creating organization. The captured stakeholder needs are then translated into input–
output models for each stakeholder, which are then aggregated into a network model.
Analysis of this network suggests that benefits are infrequently linked to the root
provider of value. Furthermore, it is noted that requirements should not only be
written to influence the organization’s outputs, but also to influence the propagation
of benefit further along the value chain. A number of future applications of this model
to systems architecture and requirement analysis are discussed.
Loureiro, G., Crawley, E., Catanzaro, S. & Rebentisch, E. (2006) From Value to
Architecture - Ranking the Objectives of Space Exploration (IAC-06-D1.4.2).
Proceedings of the International Astronautical Congress (IAC 2006), Valencia, Spain, 2 6 October 2006.
This paper describes and discusses a process for identifying and ranking the objectives
of space exploration. Objectives were generated and structured. Initial objectives were
then classified as stakeholder intent objectives, outcomes to stakeholders and assets
to be provided by the exploration enterprise to stakeholders. Assets derived objectives
at lower level and these were organized into disjoint sets. Objectives were then related
to stakeholder groups. Objectives within each stakeholder group were ranked using
the Kano model. Architectures were, then, ranked based on their relationship to the
ranked objectives. Architectures were ranked, separately, per stakeholder value, per
cost, per risk and per policy robustness. There was no one overall metrics composed by
each stakeholder group value, cost, risk and policy robustness. The paper exemplifies
the process with the actual objectives, their resulting ranking per stakeholder group
and the use of the ranked objectives to rank candidate architectures generated. The
paper discusses the preferred architectures per stakeholder group and concludes with
a critical analysis of the process.
Ippolito, B. J. (2000) Identifying Lean Practices for Deriving Software Requirements,
Cambridge, MA, Master's thesis, LAI and MIT.
(Cameron et
al., 2008)
Request at LAI
(Loureiro et
al., 2006)
Request at LAI
(Ippolito,
2000)
Download Link
Based on detailed case studies, this research analysis the process for deriving software
requirements from the standpoint of lean management. Ten aerospace software
upgrades were analyzed at both an enterprise level and an organizational level to
identify the presence of Lean practices. At the enterprise level, metrics typically used
to measure enterprise performance (Flow Time, Stakeholder Satisfaction, Quality Yield,
and Resource Utilization) were found to be appropriate for the software requirement
process but not adequately implemented. An organizational analysis observed five of
the twelve Lean practices as effectively implemented. These are 1. Implement
integrated product and process development; 2. Maximize stability in a changing
environment; 3. Assure seamless information flow; 4. Develop relationship based on
mutual trust and commitment; and 5. Continuous focus on the customer. The research
also identified opportunities to implement three more Lean practices: 1. Identify and
optimize enterprise flow; 2. Optimize capability and utilization of people; and 3. Make
decisions at the lowest possible level.
Page 41
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
Downen, T. D. (2005) A Multi-Attribute Value Assessment Method for the Early
Product Development Phase With Application to the Business Airplane Industry,
Cambridge, MA, PhD Thesis, MIT and LAI.
The early phase of product development is critical to the success of enterprises. In
addition, it has been argued that the concept of consumer value is central to effective
product development. In this research, a new product value assessment method is
established for the fuzzy front-end of business airplane development. A recentlydeveloped multi-attribute value method, based on Taguchi’s loss function approach to
quality assessment, is modified and extended in this study and applied for the first
time to the domain of business aviation. A comprehensive 40-year historical product
database is developed for use in testing and evaluating the method, referred to as the
Relative Value Index (RVI), enabling the scope of value method appraisal to be
expanded to an industry-wide examination over a significant time span. This
methodology is a useful advance in the methods to extract objective findings from
historical industry market activities. Marking the first independent review of the loss
function-based value method, this study finds that the Relative Value Index is superior
to existing value methods at retaining simplicity of implementation and minimal data
requirements while maintaining a firm grounding in economics and consumer choice
theory.
Dickerson, C. & Valerdi, R. (2010) Using Relational Model Transformations to Reduce
Complexity in SoS Requirements Traceability: Preliminary Investigation, Loughborough,
UK, 5th IEEE International Conference on Systems of Systems Engineering, June 2010.
The principles and methods of Model Driven Architecture are applied to the problem
of requirements traceability for a System of Systems (SoS). Model transformations of
operational threads are used to reduce the complexity of modeling mission
requirements and their flow into the architecture of the SoS. The allocation of
requirements to operational mission threads (OMTs) rather than to individual systems
reduces the complexity of the requirements tracing. Relational transformations
provide a mathematically based formalism for model transformations that permit
precise computation of the transformation of operational threads into threads of
systems allocated from the SoS. Connectivity requirements for the SoS are also
exposed in this way and the number of permissible system threads are seen to
correspond directly to the number of permissible transformations. The principles and
methods are illustrated by an elementary case study for sensor fusion.
Matty, D.: Stakeholder Salience Influence on Bureaucratic Program Enterprise Value
Creation. Ph.D. Thesis, Massachusetts Institute of Technology, Cambridge, MA (2010)
The government acquisition system has been studied extensively. Applying traditional
system engineering methods have not improved performance, but developed a highlycomplex bureaucracy that is viewed as inflexible, unscalable, unreliable, and (recently)
unsustainable. With this seemingly intractable challenge, this work uses the synergy of
integrating approaches based on engineering, management, and social sciences to
develop a new framework to help understand the policy resistance of many previous
unsuccessful initiatives.
This research seeks to develop a dynamic enterprise engineering system framework
using case study methodology to integrate three widely adopted but disparate
frameworks by evaluating the influence relationships. Informed by the enterprise
architecture, this new framework seeks to incorporate stakeholder salience and its
dynamic influence on value creation as an endogenous factor in the context of the
bureaucratic program enterprise of DOD acquisition. This work not only proposes an
intermediate level theory but also provides insights for policy implications.
Page 42
(Downen,
2005)
Download Link
(Dickerson
and Valerdi,
2010)
Request at LAI
(Matty, 2010)
Download Link
LAI Whitepaper Series
8.3.2 Managing trade-offs
During the specification of the final requirements, conflicting requirements have to be traded
against each other. This includes for example trade-offs regarding cost, schedule, performance
and risk requirements. The management of trade-offs must also include re-visiting and renegotiating requirements where necessary, in order to assure that the baseline requirements
are realistic.
Publication Title & Synopsis
Derleth, J. E. (2003) Multi-Attribute Tradespace Exploration and its Application to
Evolutionary Acquisition, Cambridge, MA, Master's thesis, LAI and MIT.
The concept of Evolutionary Acquisition (EA) provides many challenges to a system
engineer, especially regarding the definition and trading-off of requirements. MultiAttribute Tradespace Exploration (MATE) is a tool that can provide some focus on a
project in EA. MATE is a method of developing models to simulate the product user’s
preferences for the attributes of a design. Once these preferences are well known,
they can be used to guide the design choice. The design choice is further guided by the
creation of system level computer models that represent the design choices available
to the engineer. These choices are then varied systematically to create a “tradespace”
of possible designs. This tradespace exhaustively enumerates all of the possible design
choices for the engineer. Then, through the preference models previously developed,
each possible design is ranked in order of user utility and cost. The result can be
graphed, giving a visual representation of the utility and cost of literally thousands of
architectures in a single glance. This research is based on a case study of the Small
Diameter Bomb (SDB) and shows that MATE is a useful tool for a systems engineer
working on an EA system. There are several benefits to the use of MATE in EA,
including: 1. a better understanding of the end user’s desires and requirements for the
system; 2. the ability to optimize the system for the first evolution; 3. the possibility of
understanding what will become optimal in later evolutions; 4. quick redesign time if
circumstances or preferences change; and 5. further insight into systems level
considerations.
Spaulding, T. J. (2003) Tools for Evolutionary Acquisition: A Study of Multi-Attribute
Tradespace Exploration (MATE) Applied to the Space Based Radar, Cambridge, MA,
Master's thesis, LAI and MIT.
In this research, the Multi-Attribute Tradespace Exploration (MATE) process was
applied to the Space Based Radar (SBR), a space system under study by the United
States Air Force. A system-level model of possible SBR architectures was created using
data and analysis from previous high-level studies. Competing designs were evaluated
through MATE’s universal utility metric. The MATE model was qualitatively compared
against a high-level design study and MATE’s advantages were noted, specifically its
ability to trace modeling assumptions and present a holistic view of the space of
competing designs. A quantitative comparison revealed significant differences
between MATE’s recommended system design and that of the comparison high-level
study. The potential for a simplification of the MATE method was explored through the
use of several approximations to revealed user preferences. It was shown that while a
linear or subjective approximation to utility curves resulted in excessive errors,
approximation to weighting relationships did not. Finally, MATE’s potential
applicability to the Air Force acquisition process was studied. In general MATE was
shown to be useful to any acquisition effort that derives its benefit from a networked
approach and is of sufficient technical complexity as to make tradeoff decisions
Citation &
Download Link
(Derleth,
2003)
Download Link
(Spaulding,
2003)
Download Link
Page 43
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
opaque to casual analysis. Specifically, MATE was shown to be useful in the analysis of
alternatives process as well as an aid to early milestone sourcing decisions.
Ross, A. M. (2003) Multi-Attribute Tradespace Exploration with Concurrent Design as a
Value-Centric Framework for Space System Architecture and Design, Cambridge, MA,
Master's thesis, LAI and MIT.
This research investigates incorporating decision theory methods into the early design
processes to streamline communication of wants and needs among stakeholders and
between levels of design. Communication channeled through formal utility interviews
and analysis enables engineers to better understand the key drivers for the system and
allows for a broad and more thorough exploration of the design tradespace. MultiAttribute Tradespace Exploration (MATE), an evolving process incorporating decision
theory into model and simulation-based design, has been applied to several space
system projects. The conclusions of these studies indicate that this process can
improve the quality of communication to more quickly resolve project ambiguity, and
enable the engineer to discover better value designs for multiple stakeholders. MATE
is also being integrated into a concurrent design environment to facilitate the transfer
of knowledge of important drivers into higher fidelity design phases. MATE with
Concurrent Design (MATE-CON) couples decision makers more closely to the design,
and most importantly, maintains their presence between formal reviews.
Tang, V. (2006) Corporate Decision Analysis: An Engineering Approach, Cambridge,
MA, PhD thesis, LAI and MIT.
This thesis explores corporate decisions and their solutions under uncertainty using
engineering methods. The proposition is that as in an engineering system, corporate
problems and their potential solutions deal with the behavior of systems. Since
systems can be studied with experiments, this research uses Design of Experiments
(DOE) to understand the behavior of systems within which decisions are made and to
estimate the consequences of candidate decisions as scenarios To test this engineering
approach to decision analysis, four experiments are performed. The first two are a set
of simulations using a company surrogate. Using a progression of experiments, two
major corporate decisions are simulated. Simulation data show that there is support
for the validity of the decision analysis method. Then two in situ experiments are
performed: with a manufacturing company and with a technology services company.
Findings from these company experiments also support the validity and efficacy of the
decision analysis method.
(Ross, 2003)
Download Link
(Tang, 2006)
Download Link
8.3.3 Cost estimation & life cycle costing
Before a budget is allocated, the resource requirements in term of cost, schedule, manpower,
facilities and other resources has to be forecasted. This involves a probability-based
assessment of these resources so an informed decision can be made in the following step
regarding the type and content of the contract.
Publication Title & Synopsis
Valerdi, R. (2005) The Constructive Systems Engineering Cost Model (COSYSMO), Los
Angeles, CA, PhD Thesis, USC.
This dissertation presents a parametric model that can help people reason about their
decisions related to systems engineering. COSYSMO, the Constructive Systems
Engineering Cost Model, is an “open” model that contains eighteen parameters: four
size drivers and fourteen effort multipliers. It is built on a framework similar to its wellknown predecessor, COCOMO II, and integrates accepted systems engineering
standards to define its scope. The four size drivers are: Number of System
Page 44
Citation &
Download Link
(Valerdi, 2005)
Download Link
LAI Whitepaper Series
Requirements, Number of Major Interfaces, Number of Critical Algorithms, and
Number of Operational Scenarios. The fourteen effort multipliers are: Requirements
understanding, Architecture understanding, Level of service requirements, Migration
complexity, Technology risk, Documentation to match life cycle needs, Number and
diversity of platforms, Number of recursive levels in the design, Stakeholder team
cohesion, Personnel or team capability, Personnel experience or continuity, Process
capability, Multisite coordination, and Tool support.
Valerdi, R., Rieff, J. E. & Wang, G. (2007) Lessons Learned From Industrial Validation of
COSYSMO, San Diego, CA, 17th INCOSE Symposium, June 2007.
The development of COSYSMO has been an ongoing collaboration between industry,
government, and academia since 2001. Now that the development phase of the model
is completed we take a retrospective view of lessons learned during the ongoing
validation phase of the model and present new lessons learned that should help cost
model developers, academic researchers, and practitioners develop and validate
similar approaches. These lessons include the need for more specific counting rules, an
approach to account for reuse in systems engineering, and strategies for model
adoption in organizations.
Valerdi, R. (2010) Heuristics for Systems Engineering Cost Estimation. IEEE Systems
Journal.
This paper provides thirty one heuristics that have been inspired by the development
and application of a systems engineering cost estimation model. The objective of this
paper is to present such heuristics in a simple manner so that they can benefit systems
engineering researchers and practitioners. They fall into the four categories of model
development heuristics, model calibration heuristics, model usage heuristics and
estimation heuristics.
Czerwonka, S. P. (2000) Avionics Life-Cycle Forecasting Model, Cambridge, MA,
Master's thesis, MIT and LAI.
(Valerdi et al.,
2007)
Download Link
(Valerdi, 2010)
Download Link
(Czerwonka,
2000)
Download Link
Sustaining aging aircraft fleets is a major challenge. Avionics are of particular concern
due to rising problems with reliability and obsolescence as these systems age. If these
problems are not addressed before an avionics component reaches the end of its life
cycle, aircraft availability can suffer and sustainment costs can rise dramatically. This
thesis presents a computational model that analyzes component performance metrics
to forecast the time that an avionics component will reach the end of its life cycle. The
purpose of the model is to provide aircraft program managers with a means of
identifying critical components in advance of the onset of aging difficulties so as to
allow corrective measures to be taken which can prevent unnecessary hardships.
Page 45
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
8.3.4 Incentive alignment, contract negotiation & conclusion
Contracting under large program risks is a very challenging task. Based on the cost (and other
resources) probability distributions, an adequate way of sharing risk (cost overrun relative to
baseline) and profit (cost savings relative to baseline) has to be developed. This should allocate
the responsibility for carrying risk consequences according the responsibility for the risk
sources, or create incentives to accept external risks. An important part of this process is
understanding and aligning incentives between the program office and the contractors, as well
as between contractor and its suppliers. Incentives must include guidelines on how to share
benefits from local and enterprise-level improvements, improvements strategies, as well as
guidelines for recovering stressed relationships.
Publication Title & Synopsis
Dorey, S.: Enhancing cost realism through risk-driven contracting: Designing incentive
fees based on probabilistic cost estimates. Research report, Lean Advancement
Initiative, Massachusetts Institute of Technology, March 2011 (2011).
A risk-driven contract structure is proposed to enhance the cost realism of competitive
proposals for the Engineering and Manufacturing Development (EMD) phase of the
acquisition lifecycle. An economic theory framework is employed to discuss how the
cost-plus contracts typically used during this phase have inadvertently motivated
contractors to provide optimistic cost estimates in an attempt to win competitive
source selections. By directly mapping probabilistic cost estimates to profit
distributions, risk-driven contracts offer a structured method to impose more cost risk
sharing on contractors during EMD. Holding contractors accountable for their cost
estimates and cost performance should enhance the realism of their cost proposals
and ultimately reduce the cost growth that continues to plague the defense acquisition
system.
Cowap, S. A. (1998) Economic Incentives in Aerospace Weapon Systems Procurement,
Master Thesis, LAI and Massachusetts Institute of Technology.
Incentives are an instrument to reconcile conflicting goals between program office and
contractor, or also between contractor and supplier. By linking the achievement of a
goal that is important for one party with the achievement of a goal with another party,
the severity of conflicting objectives may be reduced. It can also be used to make the
achievement of a specific goal desirable to both parties. This work describes case
studies that show that programs successfully use incentives by tailoring them to the
specific goals and risks, and describes the tailoring process. It also contains examples of
incentives, such as 1. Finance-related incentives: 1.1 Increased profit rate; 1.2 Sharing
of cost saving benefits; 1.3 Reimbursement for investments; 1.4 Average unit
procurement price; 1.5 Annual re-negotiation of firm fixed price contract; 2. Riskrelated incentives: 2.1 Transfer of uncertain cost elements to program office; 2.2
Limited financial liability; 2.3 Limited future legal liability; 3. Process-related incentives:
3.1 Configuration control; 3.2 Variance in performance specifications; and 3.3 Granting
of waivers.
Kirtley, A. L. (2002) Fostering Innovation across Aerospace Supplier Networks, Master
Thesis, LAI and Massachusetts Institute of Technology.
Suppliers play a central role in every program. They account for a large share of the
total cost and technology content of major programs. Also, a significant share of
technological innovation across many industries takes place at the interface between
customer companies and their suppliers. Based on two case studies, this work
develops recommendations on how to incentives suppliers for increased innovation.
Page 46
Citation &
Download Link
(Dorey, 2011)
Request at LAI
(Cowap, 1998)
Download Link
(Kirtley, 2002)
Download Link
LAI Whitepaper Series
These incentives to supplier innovation are: 1. Differentiation of supplier relationships;
2. Excellent communication; 3. Sharing risk and investments; 4. Incentive mechanisms
in contract structures; 5. Training and support; and 6. Appropriate levels of
competition. Identified barriers to supplier innovation include: 1. Concern for secrecy
and other communication barriers; 2. Uncertainty about the program’s future; and 3.
Government contracting practices failing to reward innovative efforts.
Mandelbaum, J., Kaplan, W. S., McNutt, R. T. & Rebentisch, E. (2001) Incentive
Strategies for Defense Acquisitions, Washington, D.C., Department of Defense.
(Mandelbaum
et al., 2001)
Request at LAI
Incentives should exist in every business arrangement because they maximize value for
all parties. DoD needs to adopt strategies that attract, motivate, and reward
contractors to encourage successful performance. Using commercial practices will
enhance DoD's ability to attract nontraditional contractors. This guide amplifies
existing policy regarding use of incentives in defense acquisitions. It explores costbased and non-cost-based incentive strategies. It clearly defines use of performance
objectives or product functionality vs. detailed requirements to seek best value
acquisitions. It answers the following questions: 1. Why are we concerned with
contractual incentives? 2. What elements contribute to an effective incentive
strategy?, 3. How can we build and maintain an effective environment for a successful
business relationship?, 4. How can we build the acquisition business case? And 5. How
can we build an incentive strategy that maximizes value?
Page 47
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
8.4 LAI’s Insights: Technology Integration
During Technology Integration, technologies for the design phase are selected and integrated
into the program. The readiness of low-maturity technologies may be increased in parallel
technology development tracks to a level that is suitable for their introduction into the design
process. It includes the following activities:
8.4.1 Technology maturation monitoring
The main objective is to develop technologies to a level where the main risks regarding their
cost and performance have been eliminated and they are fit to be integrated into an overall
system. However, assessing the maturity level of a technology is not a trivial task, as certainty
regarding its development status may be difficult to establish.
Publication Title & Synopsis
Kenley, C. R. & Creque, T. R. (1999) Predicting Technology Operational Availability
Using Technical Maturity Assessment. Systems Engineering, 2, 198-211.
This paper reports research on the relationship of technology readiness and remaining
time to technology introduction for nuclear stabilization technology. A technical
maturity assessment method was performed by systems engineers in collaboration
with a senior advisory panel composed of team members from different Department
of Energy sites and from different engineering and science disciplines. Various
stabilization technologies were assessed annually as to their relative maturity and
availability for use in stabilizing nuclear materials. After 3 years of assessments, several
of the technologies are now components of operational systems. A regression analysis
of the historical assessments was performed, and it was concluded that the numerical
technical maturity score produced by a team of experts provides a powerful predictor
of the time remaining until the operational application of technologies.
Tang, V. & Otto, K. N. (2009) Multifunctional Enterprise Readiness: Beyond the Policy
of Build-Test-Fix Cyclic Rework. Proceedings of the ASME 2009 International Design
Engineering Technical Conferences & Design Theory and Design IDETC/DTM 2009 ,
August 30 - September 2, 2009, San Diego, California, 1-9.
NASA, the US Government and many companies attempt to manage the development
and launch of new technology using Technology Readiness Levels, TRLs. Unfortunately,
TRLs as generally defined are outdated and flawed, based on the extent of prototype
or hardware use in the field. Urgency in improving TRL levels drives early release of
hardware before it is ready, and initiates cyclic rounds of debugging and fixing failures
in the field or laboratory. Such a build-test-fix approach to product development is
now well documented to be inefficient and wasteful. We present updated definitions
of technology readiness levels (TRLs) based on the lean and design-for-six-sigma
product design methodology, a radical departure from the “build-test-fix”
methodology of conventional TRLs. We argue that the iterative build-test-fix approach
of cyclic rework is costly to product development, as well as, downstream
manufacturing and services. We call our updated TRL the L-TRL, for Lean TRL.
Consistent with our L-TRL, we also present updated definitions for Manufacturing
Readiness Levels (MRLs) to address lean and six-sigma manufacturing principles. Hence
we call them L-MRL. We address a void in the literature and unveil definitions for
service readiness levels (SRLs).
Page 48
Citation &
Download Link
(Kenley and
Creque, 1999)
Download Link
(Tang and
Otto, 2009)
Request at LAI
LAI Whitepaper Series
8.4.2 Technology transition management
This process addresses the integration of technologies that just reached sufficient maturity
levels at R&D facilities into existing design and production processes at the main contractor
and its suppliers.
Publication Title & Synopsis
Shroyer, E. (2002) Lean Transition of Emerging Industrial Capability (LeanTEC), Final
Report, Cooperative Agreement F33615-97-2-5153, U.S. Air Force and Boeing
Company.
Technology transition from Research and Development to Product is not done as
effectively as it should be in either government or industry. Industry invests an average
of 3.5% of sales in R&D ($264B in FY2000). Of the projects that are expected to
transition into production, only 20% to 60% do transition. Of those that transition, 60%
are either late, have changes after transition, do not meet technical goals, or do not
meet cost goals, while 5% of the projects experience all of these inefficiencies.
Conservative monetary estimates for losses to industry are over $80B per year in
waste and over $300B per year in lost savings. This research documents the result of
surveys and best practice analysis in the aerospace and defense sector. A large number
of recommendations regarding an efficient technology transition process are grouped
into eight main process steps: 1. Establish technology transition process; 2. Create
enabling environment for technology transition; 3. Select technology transition project
portfolio; 4. Form team and execute technology transition projects; 5. Develop project
charter and plans; 6. Establish communication protocols; 7. Promote efficient plan
execution; and 8. Conduct formal reviews
Pomponi, R. A. (1998) Organizational structures for technology transition: rethinking
information flow in the integrated product team, Cambridge, MA, PhD thesis, LAI and
MIT.
This research examines how organizational structure affects the implementation of
new technology within an IPPD environment, focusing on information flow among
integrated product teams (IPTs). Four unique information structures were identified in
the first phase of the research, termed the management chain, focal point linkage,
focal point inclusion, and network structures. The results of four in-depth case studies
indicate that organizational structures which promote the flow of information to and
from the IPT level contribute to a more effective technology transition process, as
evidenced by the ability to deal effectively with organizational conflict, to coordinate
across functional groups, and to plan for the future use of the technology in the firm.
These structures were also shown to increase the ability of the manufacturing
technology to positively influence the product design. Focal inclusion links between
IPTs and a centralized technology transition function are recommended as a means to
accomplish company-wide strategic goals within the focused development activity of
the IPPD environment.
Citation &
Download Link
(Shroyer,
2002)
Download Link
(Pomponi,
1998)
Download Link
Page 49
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
8.5 LAI’s Insights: Product Design
In Product Design, the system integration and its components are specified to a level that
makes it fit for production. It consists of the following activities:
8.5.1 Product Design team organization
Product Design is a complex task that relies heavily on the integration of deep domain
knowledge from various areas. This places high requirements not only on the leadership of PD
teams, but also on the team members and the organization that provides them. In addition,
integrated product teams for example bridge the gap between different organizations.
Publication Title & Synopsis
Klein, J., Cutcher-Gershenfeld, J. & Barrett, B. (1997) Implementation Workshop: High
Performance Work Organizations, Cambridge, MA, LAI Report RP97-02-34.
This report summarize key insights and learning by representatives from a cross
section of organizations who are on the journey of implementing lean principles in
order to achieve high performance work systems. The overall goal of the workshops
and case studies was to facilitate learning about the implementation of high
performance work organizations among LAI consortium members. Three case study
presentations helped initiate the discussion. The two unionized cases were selected
based on recommendations from the UAW and the IAM, while the Texas Instruments
case was selected for the length of experience operating as a team-based work system.
The key lessons learned address the areas of 1. Leadership, 2. Skill Development, 3.
Measuring Progress, 4. Rewards and Incentives, and 5. Diffusion.
Bernstein, J. I. (2000) Multidisciplinary Design Problem Solving on Product
Development Teams, Cambridge, MA, PhD thesis, LAI and MIT.
This research studied how engineers from different specialties interpret and
communicate about technical design problems while working on product development
teams. Data was collected on 98 cases via interviews with engineers. The most
important finding of this study was that engineers from different specialties do
interpret the same problem differently. Specifically, two engineers were likely to
evaluate the benefits or drawbacks of a potential solution using different sets of
criteria. Thus, some design disputes were the result not of mutually exclusive needs
but of a failure to recognize the different ways in which engineers were evaluating
solutions to the problem. Furthermore, data collected during this study illustrated that
in some cases these differences were the result of engineers addressing related, but
unique problems. Therefore, a solution to one engineer’s problem often created a new
problem for another engineer on the team. A second conclusion of this study was that
how design tools were used had a greater impact on a team’s problem solving abilities
than what tool was used. In this context, design tools included objects such as real or
“virtual” prototypes as well as processes like simulations and tests. The results of this
investigation suggested that such tools offered their greatest benefits when they were
used in a participatory fashion in which a large fraction of a team shared in their use.
Additionally, the more elements of a problem’s context that were captured in a design
tool, the greater its utility. Under such conditions, team members were able to create
a shared evaluation system to judge potential solutions to the problem they were
confronting, thereby facilitating problem resolution.
Page 50
Citation &
Download Link
(Klein et al.,
1997)
Download Link
(Bernstein,
2000)
Download Link
LAI Whitepaper Series
Bresman, P. H. M. (2004) Learning Strategies and Performance in Organizational
Teams, Cambridge, MA, PhD thesis, MIT and LAI.
This research addresses the subject of team learning strategies and their performance
effects in three areas. The first part explores team learning in an inductive study of six
teams in one large pharmaceutical firm. Many of these teams engage in vicarious team
learning — the activities by which a team learns key aspects of its task from the similar
experiences of others outside the team — rather than experiential team learning. The
nature of vicarious team learning is detailed in a model including three component
processes: identification, translation, and application. The second part reviews the
literature on team learning and concludes that it has largely been treated as a uniform
construct. It is proposed that teams learn by deploying at least three different
strategies: experiential learning, contextual learning, and vicarious learning. The third
part examines different team learning strategies, and vicarious learning in particular,
as a means to understanding learning and performance differences across teams.
Vicarious learning is conceptualized as an integral part of how teams learn. A field
study of 43 teams in the pharmaceutical industry is used to develop and test the
construct and shows that vicarious learning is positively associated with performance.
Susman, G. & Petrick, I. (1995) Product Development Team Effectiveness, Cambridge,
MA, LAI Whitepaper 95-06.
This study explores the relative influence of function managers and team leaders in
managing integrated product development teams. It was prompted by the results of an
earlier study which suggested that the most successful high risk projects had a 50/50
balance of influence in decision-making between function managers and team leaders.
This study also suggested that in the most successful low risk projects, the balance of
influence shifted heavily toward team leaders. The data from the current study suggest
that too much influence toward team goals in daily decision-making may be counterproductive; however, the optimal point appears higher than 50/50 in the direction of
team influence. Strong team leadership was shown to be important to project success
in the earlier study. One of the ways in which strong team leadership may manifest
itself is through influence over evaluating performance and determining rewards.
Team leaders of successful projects also appear to form good working relationships
with function managers who have different backgrounds and responsibilities than they
do. Team leaders of successful projects focus more on resolving technical and
production issues with function managers than on resolving priority and resource
issues. They resolve such issues through compromise and problem-solving rather than
relying on common superiors to resolve them or not resolving them at all. Finally, the
optimal balance between goals, and influence over evaluation and rewards may vary
over project phase.
Browning, T. R. (1996) Systematic IPT Integration in Lean Development Programs,
Cambridge, MA, Master's thesis, LAI and MIT.
(Bresman,
2004)
Download Link
(Susman and
Petrick, 1995)
Download Link
(Browning,
1996)
Download Link
This research investigates the use of different integrative mechanisms for Integrated
Product Teams in lean product development. Program integration of cross-functional,
upstream/downstream nature is required at three distinct levels: (1) within IPTs, (2)
between IPTs but within higher level team groupings, and (3) between IPTs and higher
level team groupings in a program at large. This work focuses on the second and third
levels, realms of IPT interdependence, and categorizes several integrative mechanisms
(IMs) that facilitate interteam integration. IMs are strategies and tools for effectively
coordinating actions across team interfaces. The research is based on five case studies
of different programs. The nine integrative mechanisms that are discussed are: 1.
Page 51
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
Systems engineering and interface optimization; 2. Improved information and
communication technologies; 3. Training; 4. Col-location; 5. Town meetings / team
building; 6. Manager mediation; 7. Participant mediation; 8. Interface management
groups; and 9. Interface contracts and scorecards.
Oehmen, J. (2005) Approaches to Crisis Prevention in Lean Product Development by
High Performance Teams and through Risk Management. Munich and Cambridge,
Technical University of Munich and LAI.
This thesis investigates factors that define high performance teams. Based on an
extensive literature review, 49 factors are identified that have been used to describe
high performance team. A system theory-based method is then used to cluster these
factors into four groups, enablers, drivers, critical elements and indicators of high
performance in teams. The clustering is verified by interviews in a product
development organization. The factors in the different categories are discussed.
Utter, D. A. (2007) Performing Collaborative, Distributed Systems Engineering (CDSE):
Lessons Learned from CDSE Enterprises, Cambridge, MA, Master's thesis, LAI and MIT.
Previous research has demonstrated that the design practices of distributed design
teams differ from those of traditional, co-located teams. However, many companies
today are performing collaborative, distributed systems engineering (CDSE) using
systems engineering (SE) processes and methods developed for traditional SE
environments and are therefore encountering many issues. Successful SE practices are
difficult to carry-out when performed by a traditional, colocated enterprise. The
addition of geographic distribution and cross-company or intra-company collaboration
in SE presents a myriad of social and technological challenges that necessitate new and
different SE methods for success. In an attempt to benchmark the current state of
CDSE practices in industry, this research presents the collection of CDSE lessons
learned and success factors gathered from two case studies carried out at two
aerospace and defense companies. The case studies examine many different factors
that pertain to the companies’ current CDSE efforts, including collaboration scenarios;
collaboration tools; knowledge and decision management; SE practices and processes;
SE process improvements; SE culture; SE project management, SE organization; and SE
collaboration benefits and motivation.
(Oehmen,
2005)
Download Link
(Utter, 2007)
Download Link
8.5.2 Integrated product and process development
As requirements and systems vary, the PD process has to be adapted accordingly. This is
closely linked to the optimization of the value stream, for example with value stream mapping
and the elimination of waste, as well as the selection of an appropriate development strategy,
such as a waterfall model or set-based design.
Publication Title & Synopsis
Pomponi, R. A. (1998) Organizational structures for technology transition: rethinking
information flow in the integrated product team, Cambridge, MA, PhD thesis, LAI and
MIT.
This research examines how organizational structure affects the implementation of
new technology within an IPPD environment, focusing on information flow among
integrated product teams (IPTs). Four unique information structures were identified in
the first phase of the research, termed the management chain, focal point linkage,
focal point inclusion, and network structures. The results of four in-depth case studies
indicate that organizational structures which promote the flow of information to and
from the IPT level contribute to a more effective technology transition process, as
evidenced by the ability to deal effectively with organizational conflict, to coordinate
Page 52
Citation &
Download Link
(Pomponi,
1998)
Download Link
LAI Whitepaper Series
across functional groups, and to plan for the future use of the technology in the firm.
These structures were also shown to increase the ability of the manufacturing
technology to positively influence the product design. Focal inclusion links between
IPTs and a centralized technology transition function are recommended as a means to
accomplish company-wide strategic goals within the focused development activity of
the IPPD environment.
Hernandez, C. M. (1995) Challenges and Benefits to the Implementation of Integrated
Product Teams on Large Military Procurements, Cambridge, MA, Master's thesis, LAI
and MIT.
This thesis analysis the challenges surrounding the implementation of integrated
product teams (IPT) in military programs. It looks at this issue for four, ongoing, major
aircraft developments; B2 Bomber, C17 Transport, F/A-18 E/F Fighter, and F22 Fighter.
These four programs are reviewed and contrasted to commercial business practices to
bring out structural differences that may act as barriers to IPT Implementation. Several
areas were identified that impede its implementation. These areas include: training,
team budget control, and the need for balance between teams and functions. In
addition, details of how benefits can be derived from the IPT concept are discussed.
These are reduced development time, team consensus decision making and improved
communication. Current methods being used to measure these benefit are presented.
Lucas, M. V. (1996) Supplier Management Practices of the Joint Direct Attach Munition
Program, Cambirdge, MA, Master's thesis, LAI and MIT.
The Joint Direct Attack Munition (JDAM) program reveals changes in the model for
supplier relationships in the defense aerospace industry that have been accompanied
by unprecedented results. Changes in decision-making, program structure, and
organizational culture occurred as the result of reform measures and the product
development administration of the program. The changes implemented by the
Government as well as the innovative supplier management practices of the prime
contractor showed progress in the general model for supplier relationships towards a
more collaborative, team-oriented partnership. The linkages of the suppliers and the
supplier designs resulted in innovations that changed the system architecture.
Bozdogan, K., Deyst, J., Hoult, D. & Lucas, M. (1998) Architectural innovation in product
development through early supplier integration. R&D Management, 28, 163-173.
(Hernandez,
1995)
Download Link
(Lucas, 1996)
Download Link
(Bozdogan et
al., 1998)
Download Link
This paper explains how an important opportunity exists to pro-actively integrate
suppliers at an early stage in the concept exploration and definition stages of product
development. Research suggests that the concept of architectural innovation can be
extended so that product features are matched with the associated specialized
technical skills of partners in the development team. In addition to the establishment
of integrated product teams, key enablers include: long-term commitment to
suppliers; co-location; joint responsibility for design and configuration control;
seamless information flow; and retaining flexibility in the definition of the system
configuration.
Page 53
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
8.5.3 Product Design and System Engineering best practices
Some organizational concepts as well as processes have been identified as especially
important for success. These are summarized in best practices that can be used for an internal
benchmarking and to identify opportunities for improvement.
Publication Title & Synopsis
Knoblinger, C.: PDSAT – A New Product Development Self-Assessment Tool. Technical
University of Munich and Lean Advancement Initiative (http://lean.mit.edu), Munich
and Cambridge, MA (2011)
One approach companies can use to improve their product development performance
is self-assessment to optimize their organization and processes. This thesis summarizes
the current literature on PD-related self-assessment tools and derives tool
requirements from an industry focus group (US aerospace and defense industry) as
well as from interviews at a major American defense contractor. A gap analysis
comparing these requirements to the previously identified tools is performed. The
thesis concludes with the presentation of a new holistic self-assessment framework to
be used in PD organizations. The framework includes a self-assessment questionnaire
with 91 metrics, a formalized 9-step implementation process, tool customization
guidelines, and mappings between the structure of the questionnaire and relevant
process improvement approaches such as CMMI, Malcom Baldrige National Quality
Award, Lean Management, and Six Sigma.
Gordon, M., Musso, C., Rebentisch, E. & Gupta, N. (2009) The Path to Developing
Successful New Products. The Wall Street Journal, November 30, 2009.
This article summarizes the results of a product development best practice survey
carried out by LAI and McKinsey. It was found—after surveying more than 300
employees at 28 companies across North America and Europe—that the businesses
with the best product-development track records do three things better than their
less-successful peers: They create a clear sense of project goals early on, they nurture a
strong project culture in their workplace, and they maintain close contact with
customers throughout a project's duration.
Boehm, B., Valerdi, R. & Honour, E. (2008) The ROI of Systems Engineering: Some
Quantitative Results for Software-Intensive Systems. Systems Engineering, 11, 221234.
This paper presents quantitative results on the return on investment of systems
engineering from an analysis of the 161 software projects. The analysis shows that,
after normalizing for the effects of other cost drivers, the cost difference between
projects doing a minimal job of software systems engineering—as measured by the
thoroughness of its architecture definition and risk resolution—and projects doing a
very thorough job was 18% for small projects and 92% for very large software projects
as measured in lines of code. The paper also presents applications of these results to
project experience in determining “how much up front systems engineering is enough”
for baseline versions of smaller and larger software projects, for both ROI-driven
internal projects and schedule-driven outsourced systems of systems projects.
Oppenheim, B.W., Murman, E.M., Secor, D.A.: Lean Enablers for Systems Engineering.
Systems Engineering 14(1), 29-55 (2011).
The emerging field of Lean Systems Engineering (LSE) is the application of Lean
principles, practices, and tools to SE and to the related aspects of enterprise
management (EM) in order to enhance the delivery of value (which is defined as
flawless delivery of product or mission with satisfaction of all stakeholders) while
Page 54
Citation &
Download Link
(Knoblinger,
2011)
Download Link
(Gordon et al.,
2009)
Download Link
(Boehm et al.,
2008)
Request at LAI
(Oppenheim
et al., 2011)
Download Link
LAI Whitepaper Series
reducing waste. The Lean Enablers for Systems Engineering is a comprehensive
checklist of nonmandatory practices and recommendations formulated as “do’s” and
“don’t’s” of SE, and containing tacit knowledge (collective wisdom) on how to prepare
for, plan, execute, and practice SE and EM using Lean Thinking. Each enabler has the
potential to enhance program value and reduce waste. The Enablers are formulated as
a web-based addendum to the current SE Handbook published by the International
Council for Systems Engineering (INCOSE), and do not repeat the practices made
therein, which are regarded as sound.
8.5.4 Monitoring Product Design progress
Monitoring progress during PD can be particularly challenging, as many iterations and re-work
of “finished” components may occur. Therefore, to allow for a better management of the
process, the use of appropriate KPIs and leading indicators is very important.
Publication Title & Synopsis
Roedler, G., Rhodes, D. H., Schimmoller, H. & Jones, C. (2010) Systems Engineering
Leading Indicators Guide (Version 2.0), INCOSE-TP-2005-001-03.
This guide describes the INCOSE systems engineering leading indicators. A leading
indicator is a measure for evaluating the effectiveness of a how a specific activity is
applied on a project in a manner that provides information about impacts that are
likely to affect the system performance objectives. The leading indicators are: 1.
Requirements Trend, 2. System Definition Change Backlog Trend, 3. Interface Trends,
4. Requirements validation trends, 5. Requirements verification tests, 6. Work product
approval trends, 7. Review action closure trends, 8. Risk exposure trends, 9. Risk
treatment trends, 10. Technology maturity trends, 11. Technical measurement trends,
12. Systems engineering staffing & skills trends, 13. Process compliance trends, 14.
Facility and equipment availability trends, 15. Defect or error trends, 16. System
affordability trends, 17. Architecture trends, and 18. Schedule and cost pressure.
Rhodes, D. H., Valerdi, R. & Roedler, G. J. (2009) Systems Engineering Leading
Indicators for Assessing Program and Technical Effectiveness. Systems Engineering, 12,
21-35.
This paper discusses a 3-year initiative to transform classical systems engineering (SE)
measures into leading indicators, including the resulting guidance information that has
been developed and future research directions. Contrary to simple status oriented
measures typically used on most projects, leading indicators are intended to provide
insight into the probable future state, allowing projects to improve the management
and performance of complex programs before problems arise. This paper discusses the
motivations and collaborative development of the SE leading indicators. It defines the
leading indicator construct, introduces the initial set of 13 indicators, and provides
guidance for implementation, analysis, and interpretation of these indicators. This
work serves as a foundation for industry implementation and for further research to
improve and expand the set of indicators, including development of a better
understanding of how to best implement and use the leading indicators in a given
program context.
Citation &
Download Link
(Roedler et al.,
2010)
Download Link
(Rhodes et al.,
2009)
Request at LAI
Page 55
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
8.5.5 Product architecting
The architecture of a product has important implications on the fulfillment of customer
requirements (e.g. modularity to allow for partial and continuous upgrades during the life
cycle, decoupling of technology development cycles etc.). It also has a significant influence on
how the PD process can be organized most effectively.
Publication Title & Synopsis
Bador, D. (2007) Improving the Commonality Implementation in the Cockpit of
Commercial Aircraft, Cambridge, MA, Master's thesis, LAI and MIT.
Judiciously implementing commonality across a range of products yields important
benefits in product development. Thus, measuring the quality of commonality
implementation is extremely beneficial for aircraft manufacturers. This thesis analyses
the concept of commonality and divides it into three constructs that can help
understand all of its aspects: 1. standardization, 2. reusability and 3. modularity. This
work then presents a set of metrics measuring each of these aspects, from the point of
view of the manufacturer and of the customer. The appropriateness of this set of
metrics is tested in a case study analyzing the efficiency of commonality
implementation in the cockpit of two well-known commercial aircraft families: the
Airbus A320 family and the Boeing 737 family. This thesis further describes what
additional analysis should be performed to validate the set of metrics for broader
applications. After documenting the efficiency of the set of metrics, this thesis analyses
the current practices of commonality management in commercial aviation.
Boas, R. C. (2008) Commonality in Complex Product Families: Implications of
Divergence and Lifecycle Offsets, Cambridge, MA, PhD thesis, LAI and MIT.
This dissertation leverages field research and a simple cost model to examine
commonality in the context of complex product families. The core research effort was
focused on conducting seven case studies of complex product families (aircraft,
automobiles, satellites, and capital equipment). While the case studies provided a
wealth of general insights, the studies were focused on examining divergence and
lifecycle offsets, two critical topics that influence the benefits and penalties of
commonality. Divergence refers to the tendency for commonality to reduce with time,
for both beneficial and non-beneficial reasons. Lifecycle offsets refer to temporal
differences between the lifecycle phases of product family members. Lifecycle offsets
alter the potential benefits and penalties of commonality and their apportionment to
individual products. Additionally, key factors identified during the literature review and
case studies were translated into a simple two-product cost model of development
and production in order to demonstrate key research insights in a more analytical
manner.
Nuffort, M. R. (2001) Managing Subsystem Commonality, Cambridge, MA, Master's
thesis, LAI and MIT.
Common systems satisfy the requirements of multiple platforms and meet designated
architecture, performance, life cycle cost, and interface standards. The analysis draws
on eight case studies of both commercial and military aerospace organizations. While
quantitative data on the benefits and costs of commonality in the defense aerospace
industry is difficult to obtain, the case studies suggest that commonality can
significantly reduce subsystem ownership costs by reducing both the cost of
acquisition and the cost of operations and support. Subsystem commonality also
increases mission effectiveness by reducing cycle time, improving reliability and
availability, and guarding against diminishing manufacturing sources. The work
Page 56
Citation &
Download Link
(Bador, 2007)
Download Link
(Boas, 2008)
Download Link
(Nuffort,
2001)
Download Link
LAI Whitepaper Series
indicates that commonality in the aerospace industry generally makes the most sense
at the subsystem level, where different requirements are easier to reconcile and the
strategy can have a significant impact on the logistics and supply systems. A common
organization that manages across platforms, such as a product center of excellence,
appears to offer the greatest potential advantage from commonality. The research
reinforces the notion that strategies focusing only on the benefits of commonality in
the acquisition phase of the life cycle are missing a significant portion of the holistic
advantages of commonality from a system-level perspective. Platforms that deploy
together and share common subsystems will likely have dramatically lower operations
and support costs than platforms that might be common on the manufacturing floor
but not in the field.
Silva, L. M. (2001) A Partitioning Methodology for Helicopter Avionics System with a
focus on Life Cycle Cost, Cambridge, MA, Master's thesis, LAI and MIT.
As one of the key responsibilities of a system architect, the decisions made with
respect to how an avionics system is partitioned play a significant role in the system’s
Life Cycle Cost (LCC). Despite this, most of the decomposition methods available focus
on managing the complexity of the system with respect to the architect’s ability to
understand the system. A framework was developed in order to quantify and compare
the reliability and maintainability of five different helicopter avionics system
architectures. The information was extracted from actual data collected during the
maintenance of the five helicopter systems used by the US Government over a period
of two years. Based on these five case studies, a set of partitioning criteria was
developed that can be used in future programs in order to improve the LCC of the
system. The methodology provides a process whereby the analysis of maintenance
data can assist the system architect in making better architecting decisions. The
process also identified some additional factors that have an impact on the LCC. The
most compelling was that of legacy subsystems that offer significant problems with
respect to their partitioning due to the fact that they are more entrenched in the
current avionics market.
Mahé, V. R. (2008) A Survey of Front End Modularity as an Automotive Architecture
and its Ability to Deliver Value, Cambridge, MA, Master's thesis, MIT.
The partitioning of a system dictates the creative space for a designer or engineer. This
thesis analyzes how using a new automotive architecture known as a Front End
Module (FEM) can affect a limited specific subset of stakeholders. Through the use of
interviews of subject matter experts, literature research and the use of System Design
Management tools, an in depth analysis was done on the FEM and how it affects the
craftsmanship, damageability and assembly attributes. It was shown how the
craftsmanship attribute can be improved through the strategic use of FEM's to allow
for a feed-forward system where build data are incorporated into upcoming FEM
builds. Even with this advantage, the FEM architecture will not negatively impact the
damageability attribute or assembly attribute if the proper design cues and strategies
are followed. The FEM was also evaluated as an architecture itself through the specific
and targeted intent and beneficiary breakdown. The analysis included an
Object/Process Mapping analysis where it was proposed that the true customer of the
automotive front end is not the individual that purchased the vehicle but rather the
visual society as a whole. Finally, a managerial approach was taken for the analysis of
the inherent supplier relationship that is required with using this FEM architecture.
Interviews were conducted with two suppliers of OEM's and their common road blocks
were analyzed such as lack of holistic thinking or failure to understand the role of the
system integrator.
(Silva, 2001)
Download Link
(Mahé, 2008)
Download Link
Page 57
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
Beckert, M. T. (2000) Organizational Characteristics for Successful Product Line
Engineering, Cambridge, MA, Master's thesis, LAI and MIT.
The evaluation of strategic, political, and cultural characteristics at four organizations
implementing product line engineering (PLE) was conducted. The objective of this
research was to identify non-technical characteristics that attribute to successful
product line engineering efforts. A set of questions structured around defined
strategic, political, and cultural characteristics was generated for formal evaluation of
each organization, and then a series of interviews within each organization was
performed to gather case study data. Two of the four organizations studied were
identified as being more successful with product line engineering, and, thus, were used
as the basis for comparison. A comparative analysis of the organizations was
performed to observe noteworthy characteristics that attributed to success. These
include: 1. Strategic plans clearly defined goals relating to the development of product
lines; 2. Metrics were used that applied specifically to product line engineering; 3. PLE
strategies were implemented uniformly across the organization; 4. The smallest
percent of projects for each organization utilized the new design strategy; 5.
Organizing resources around platforms, using modular system architectures, and
implementing initiatives to standardize components facilitated resource and
technology sharing; and 6. By defining and enforcing product line strategies, senior
management enabled successful product line engineering
Cunningham, T. W. (1998) Chains of Function Delivery: A Role for Product Architecture
in Concept Design, Cambridge, MA, PhD thesis, LAI and MIT.
This research intends to improve three areas of team performance in concept design
and architecting: the team’s understanding and recognition of the product
architecture, the team’s ability to document integration issues and risks, and the
team’s ability to judge whether a product concept is worthy of further pursuit. In this
context, two themes are explored. First, a systematic procedure for identifying
integration issues in mechanical assemblies is developed. The procedure captures
chains: graphical representations of how product-level attributes are delivered,
depicted on a hierarchy of the physical elements. Chains, which differ for each concept
and decomposition, show 1) which functions are delivered in integral fashion, and 2)
which elements play a role in delivering each function. Represented graphically, chains
can help the multi-disciplinary team relate the diverse technical and non-technical
influences on decomposition and architecture. Metrics can then be applied to reveal
the integration issues and integration risk associated with many, potentially conflicting
chains. The research applies established part locating mathematics in the procedure,
expanding the utility of this existing idea by applying it to concept design. Second, the
thesis develops a method for conducting a chain analysis of candidate concepts and
decompositions, creating a framework for trade-offs in the context of architecture. The
method and chain procedure and metrics are tested on the Lockheed Martin Joint
Strike Fighter military aircraft.
Page 58
(Beckert,
2000)
Download Link
(Cunningham,
1998)
Download Link
LAI Whitepaper Series
8.5.6 Value Stream Optimization
The creation of stakeholder value is the ultimate goal in product design. Several tools from
Lean Product Development are available to model and improve the flow of value while
eliminating waste. The challenge lies in the identification of value and discerning wasteful
activities from necessary, but non-value creating activities.
Publication Title & Synopsis
Oehmen, J. & Rebentisch, E. (2010) Waste in Lean Product Development. LAI Paper
Series "Lean Product Development for Practitioners". Cambridge, MA, LAI and MIT.
This whitepaper summarizes the LAI research to-date on waste in lean product
development. It contains an introduction to the concept of value and waste in Lean PD.
This is followed by a detailed discussion of the eight types of waste in Lean PD: 1. Over
production of information; 2. Over processing of information; 3. Miscommunication of
information; 4. Stockpiling of information; 5. Generating defective information; 6.
Correcting information; 7. Waiting of people; and 8. Unnecessary movement of people.
Next, the whitepaper explains the process of Value Stream Mapping and gives a
detailed example. The whitepaper concludes with a summary of the remaining
relevant LAI literature in the areas of 1. Types of waste and waste identification; 2. PD
value and value stream mapping; and 3. Application of PD value stream mapping.
The following documents are a selection of the many documents covered by the LAI
Waste in Lean PD Whitepaper (see above)
Pessôa, M. V. P. (2008) Weaving the waste net: a model to the product development
system low performance drivers and its causes, Cambridge, MA, LAI White Paper 0801.
This report is based on an extensive review of LAI and non-LAI literature to develop an
overview of different types of waste. The descriptions in Section 4 are based on this
body of work. His structure also included two more types of waste, ’wishful thinking’
and ‘external events’. In the overview presented here, these are not treated as types
of PD waste as such, but as possible underlying root causes. His work also includes a
method to prioritize different types of waste according to the extent by which they
influence and are influenced themselves by other types of waste, as well as an
approach to identify system-level root causes of wastes in order to prioritize
improvement efforts.
Kato, J. (2005) Development of a Process for Continuous Creation of Lean Value in
Product Development Organizations, Master Thesis. LAI and Massachusetts Institute of
Technology.
This thesis presents a very detailed empirical analysis of three projects, conducting a
value stream analysis, waste identification and waste quantification. His work
therefore belongs to three categories: Defining and identifying different types of
waste, extending the method of value stream analysis, and conducting and
documenting a very detailed value stream analysis (also see the example in Section 0).
In addition to the elements of his work already part of this overview, of special interest
are the additional notations that he develops to discern different types of waste
graphically in a value stream map, as well as the detailed root cause analyses that he
performs for several types of waste.
McManus, H. (2005) Product Development Value Stream Mapping (PDVSM) Manual,
Cambridge, MA, Lean Advancement Initiative (LAI) at MIT.
Citation &
Download Link
(Oehmen and
Rebentisch,
2010c)
Download Link
(Pessôa, 2008)
Download Link
(Kato, 2005)
Download Link
(McManus,
2005)
Download Link
Page 59
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
This document provides a detailed to guide to perform a value stream mapping and
analysis in a product development system. A value stream mapping aims at optimizing
the flow of value through a PD system by identifying and removing waste. It consist of
four main phases: 1. Preparatory phase; 2. Mapping of the current value stream; 3.
Identification of waste; and 4. Process improvement.
8.5.7 Product Design strategy
A number of strategies or process families can be chosen for designs that have strong
implications for all other program processes and the way that users and customers are
integrated. These include evolutionary acquisition, waterfall model and set-based design.
Publication Title & Synopsis
Ferdowsi, B. (2003) Product Development Strategies in Evolutionary Acquisition,
Cambridge, MA, Master's thesis, LAI and MIT.
The Air Force has decided to implement an Evolutionary Acquisition strategy, meaning
the development process focuses on delivering incremental capabilities through short
increments or spirals. The hypothesis of this research is that the decision for or against
Evolutionary Acquisition for a program should be based on attributes of the product,
the program goals, the uncertainties, and stakeholder involvement. A number of case
studies were performed specifically targeting programs identified as evolutionary.
From the case studies a number of key recommendations were made for program
managers and policy makers implementing Evolutionary Acquisition, as well as a
notional illustration of product development selection. Specifically, the research found
that programs with high user requirements uncertainty tended towards more
iteration, while programs with significant technical and performance goals had less
iteration and greater planning. In programs where rapid delivery was the goal and
uncertainties were relatively low, incremental strategies were found to be most
applicable. Recommendations included implementing open architecture systems
through owning system interfaces, managing stakeholder expectations through system
representations and internal change agents, and providing stable funding and
contingency funds for rapid change implementation. Additional recommendations
were made on implementing testing and logistics in highly iterative programs.
Tondreault, J. P. (2003) Improving the Management of System Development to
Produce More Affordable Military Avionics Systems, Cambridge, MA, Master's thesis,
LAI and MIT.
This thesis aims to improve the management of system development to deliver more
affordable systems. This research is different than most literature published on
affordability because it focuses on design innovation as opposed to product
development and manufacturing efficiency through Lean, Six-Sigma or other
techniques. Four areas are investigated: the nature of development focus during each
design iteration, the role of requirements, managing lifecycle cost as a design
requirement and effective integration of downstream knowledge into the design. A
model for developing requirements that strikes a better balance between performance
and lifecycle cost is suggested – treating lifecycle cost as a design requirement and
explicitly focusing on understanding the cost-performance trade space before
developing requirements. A product development model is suggested – focusing on
achieving lifecycle cost goals first and using iterations to grow performance can lead to
lower cost solutions. Both the requirements development and product development
models require leveraging prior knowledge, technology and capability. The
requirements model requires high knowledge of system cost drivers and achievable
Page 60
Citation &
Download Link
(Ferdowsi,
2003)
Download Link
(Tondreault,
2003)
Download Link
LAI Whitepaper Series
performance. The product development model requires low technical risk allowing the
team to focus on affordability first without running unacceptable levels of
performance risk. Methods for increasing the effective integration of downstream
knowledge are also discussed.
Roberts, C. J. (2003) Architecting Evolutionary Strategies Using Spiral Development for
Space Based Radar, Cambridge, MA, Master's thesis, LAI and MIT.
This thesis investigates the use of Spiral Development strategies for a space system.
Spiral Development was originally developed in the software industry, and a consensus
has not yet emerged about its applicability to space systems, for example the Space
Based Radar (SBR). In addition, space systems typically have not been developed with
explicit considerations of architectural modularity and scalability, features necessary to
enable evolutionary acquisition strategies.
A stakeholder based system architecting process known as Multi-Attribute Tradespace
Exploration (MATE) provides a framework for combining computer based modeling
and simulation of complex engineering systems with decision theory.
Qualitative comparisons between software systems and SBR in the context of spiral
development suggest a fundamental difference in system evolution and risk mitigation
strategies, yet found the Spiral Development and Evolutionary Acquisition framework
to be suitable to SBR.
Policy recommendations for improving cost accounting, key stakeholder participation,
and evolutionary options analysis were made based on findings of the research.
Stagney, D. B. (2003) The Integrated Concurrent Enterprise, Cambridge, MA, Master's
thesis, LAI and MIT.
Within the context of large-scale projects, this thesis analyzes several of the current
and emerging business processes that have been introduced in order to improve upon
the decomposition method. Techniques such as QFD, DSM, IPPD, MDO, ICE and MATECON are analyzed by way of a three-dimensional Concurrent Engineering framework.
An in-depth case study based on 15-months of work shows how the implementation of
Integrated Concurrent Engineering (ICE) can dramatically improve the quality and
speed of the design process, and can promote innovation and learning. Team metrics
are presented and analyzed. As measured theoretically and practically, no single
approach discussed in the paper brings a satisfactory resolution to the challenges
identified. Despite their initial successes, even the case study team failed to strike a
balance between the technology, people and process elements that must be
systematically managed in order to create and sustain excellence in any complex
undertaking. The structure of the team’s corporation – the financial, organizational
and human resource processes – have stalled the case study team just short of an
enterprise-wide breakthrough. Finally, radical improvements in business productivity
will not be achieved through the incremental improvements discussed above. Rather,
the very nature of the corporation must be re-thought and re-born. The thesis
presents a vision for the Integrated Concurrent Enterprise along with a concept of
operations and detailed implementation guide.
Bernstein, J. I. (1998) Design Methods in the Aerospace Industry: Looking for Evidence
of Set-Based Practices, Cambridge, MA, Master's thesis, LAI and MIT.
(Roberts,
2003)
Download Link
(Stagney,
2003)
Download Link
(Bernstein,
1998)
Download Link
Set-based concurrent engineering (SBCE) seems to offer advantages over more
traditional techniques. This research had three goals: 1) to develop a clear
understanding of the definition of SBCE and to contrast that definition with other
theories, 2) to assess the “set-basedness” of the aerospace industry, and 3) based on
the assessment, to propose a model for implementing SBCE within an aerospace
Page 61
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
development project. While set-based concurrent engineering consists of a wide
variety of design techniques, the basic notions can be stated in two principles: 1)
engineers should consider a large number of design alternatives, i.e., sets of designs,
which are gradually narrowed to a final design, and 2) in a multidisciplinary
environment, engineering specialists should independently review a design from their
own perspectives, generate sets of possible solutions, and then look for regions of
overlap between those sets to develop an integrated final solution. This research
found that while no company’s design process completely fulfilled both of these
criteria, many set-based techniques are used within the aerospace industry. Building
on some of the observed industry practices, a design process model is proposed which
combines concepts from lean manufacturing, such as “flow” and “pull,” to implement
set-based concurrent engineering.
8.5.8 Testing & prototyping
This activity is relevant both to technology development as well as product design. In
technology development, the testing and prototyping would focus on single components or
technologies, whereas in product design, the entire system (or major components) would be
tested. This is the main activity to verify that a certain performance level has been reached,
and component and system requirements are met. Developing testing and prototyping
protocols that assure a necessary level of certainty while minimizing cost is a very challenging
task.
Publication Title & Synopsis
Weigel, A. L. (2000) Spacecraft System-Level Integration and Test Discrepancies:
Characterizing Distribution and Costs, Cambridge, MA, Master's thesis, MIT and LAI.
The goal of this research is to characterize the distribution and costs of spacecraft
discrepancies found at the system level of integration and test, as well as understand
the implications of those distributions and costs for the spacecraft enterprise as a
whole. Spacecraft discrepancies are analyzed in this work on the basis of the following
categories: 1. the spacecraft mission, 2. the spacecraft subsystem where the
discrepancy occurred, 3. the date of the discrepancy occurrence, 4. the discrepancy
report open duration, 5. the immediate action taken to fix the discrepancy
(disposition), 6. the root cause of the discrepancy, 7. the long-term corrective action
prescribed to prevent the discrepancy from happening again on future spacecraft, 8.
the labor time spent on the discrepancy, and 9. the cycle time lost due to the
discrepancy. A statistical analysis forms the basis for research findings at the enterprise
level in the areas of quality yield, resource utilization, stakeholder satisfaction and flow
time. Recommendations to enterprise stakeholders for increasing the value derived
from system-level integration and test follow from the enterprise-level findings.
Carreras, C. E. (2002) Opportunities for Lean Thinking in Aircraft Flight Testing and
Evaluation, Cambridge, MA, Master's thesis, LAI and MIT.
This paper investigates whether Lean principles can be applied to aircraft flight testing
and evaluation to help meet these goals. Specific objectives are to identify
opportunities for the implementation of Lean thinking and establish a framework for
structured implementation of Lean principles and practices. This study focuses on
seven aircraft programs: 737-NG, 767-400, Hawker Horizon, F-22, F/A-18E/F, C-130J,
and the T-6A. The programs are analyzed from a programmatic viewpoint to identify
where lean practices are currently being used and how lean thinking could further
Page 62
Citation &
Download Link
(Weigel, 2000)
Download Link
(Carreras,
2002)
Download Link
LAI Whitepaper Series
improve the overall flight testing process. Opportunities were identified in:
coordination of the systems engineering value stream, coordination with other test
aircraft and necessary support functions, and management of the daily test operations.
Deonandan, I., Valerdi, R., Lane, J. A. & Macias, F. (2010) Cost and Risk Considerations
for Test and Evaluation of Unmanned and Autonomous Systems of Systems,
Loughborough, UK, 5th IEEE International Conference on Systems of Systems
Engineering, June 2010.
The evolutionary nature of Unmanned and Autonomous systems of systems (UASoS)
acquisition needs to be matched by evolutionary test capabilities yet to be developed.
As part of this effort we attempt to understand the cost and risk considerations for
UASoS Test and Evaluation (T&E) and propose the development of a parametric cost
model to conduct trade-off analyses. This paper focuses on understanding the need for
effort estimation for UASoS, the limitations of existing cost estimation models, and
how our effort can be merged with the cost estimation processes. We present the
prioritization of both technical and organizational cost drivers. We note that all drivers
associated with time constraints, integration, complexity, understanding of
architecture and requirements are rated highly, while those regarding stakeholders
and team cohesion are rated as medium. We intend for our cost model approach to
provide management guidance to the T&E community in estimating the effort required
for UASoS T&E.
Hess, J., Agarwal, G., Deonandan, I., Kenley, B., Mikaelian, T. & Valerdi, R. (2010)
Normative and Descriptive Models for Test & Evaluation of Unmanned and
Autonomous Systems of Systems, Chicago, IL, 20th INCOSE Symposium, July 2010.
The United States Department of Defense has purchased and deployed many
unmanned autonomous systems and will continue to do so at an ever-increasing rate
in the years to come. Our research aims to address a number of questions posed by
members of DOD’s acquisition workforce that are related to testing and evaluation of
these systems. This paper defines a small set of normative (ideal) and descriptive
(actual) frameworks for test and evaluation. Our research will use these models as a
step toward identifying the best prescriptive advice to give the DOD through PATFrame
(Prescriptive and Adaptive Testing Framework), the decision support system we are
developing. We describe tests strategies used in the DARPA Grand Challenge
competition and the SPHERES test bed and suggest ways in which the DOD can learn
from these experiences.
Hess, J. T. & Valerdi, R. (2010) Test and Evaluation of a SoS using a Prescriptive and
Adaptive Testing Framework, Loughborough, UK, , 5th IEEE International Conference
on Systems of Systems Engineering, June 2010.
Testers need the ability to adapt test planning on the order of days and weeks.
PATFrame will use its reasoning engine to prescribe the most effective strategies for
the situation at hand. Strategies in this context include methods of experimental
designs, test schedules and resource allocation. By facilitating rapid planning and replanning, the PATFrame reasoning engine will enable users to use information learned
during the test process to improve the effectiveness of their own testing rather than
simply follow a preset schedule. This capability is particularly attractive in the domain
of Systems of Systems testing because the complexity of test planning and scheduling
make frequent re-planning by hand infeasible.
(Deonandan
et al., 2010)
Request at LAI
(Hess et al.,
2010)
Request at LAI
(Hess and
Valerdi, 2010)
Request at LAI
Page 63
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
9 Literature references
Research publications by LAI (see links in tables) are available on our website at
http://lean.mit.edu or can be requested at LAI.
Andersen, E. S. & Jessen, S. A. (2003) Project maturity in organisations. International Journal
of Project Management, 21, 457-461.
Andrew, W. G. (2001) Do Modern Tools Utilized in the Design and Development of Modern
Aircraft Counteract the Impact of Lost Intellectual Capital within the Aerospace
Industry?, Cambridge, MA, Master Thesis, LAI and MIT.
APM (2006) Association for Project Management (APM): Body of Knowlegde, High Wycombe,
UK, APM.
Archibald, R. D. (2003) Managing high-technology programs and projects: A complete,
practical, and proven approach to managing large-scale projects with emphasis on
those involving advanced technology, Hoboken, NJ, John Wiley & Sons Inc; Wiley.
Bador, D. (2007) Improving the Commonality Implementation in the Cockpit of Commercial
Aircraft, Cambridge, MA, Master's thesis, LAI and MIT.
Beckert, M. T. (2000) Organizational Characteristics for Successful Product Line Engineering,
Cambridge, MA, Master's thesis, LAI and MIT.
Bernstein, J. I. (1998) Design Methods in the Aerospace Industry: Looking for Evidence of SetBased Practices, Cambridge, MA, Master's thesis, LAI and MIT.
Bernstein, J. I. (2000) Multidisciplinary Design Problem Solving on Product Development
Teams, Cambridge, MA, PhD thesis, LAI and MIT.
Blackburn, C. D. (2009) Metrics for Enterprise Transformation, Cambridge, MA, Master's
thesis, MIT and LAI.
Boas, R. C. (2008) Commonality in Complex Product Families: Implications of Divergence and
Lifecycle Offsets, Cambridge, MA, PhD thesis, LAI and MIT.
Boehm, B., Valerdi, R. & Honour, E. (2008) The ROI of Systems Engineering: Some Quantitative
Results for Software-Intensive Systems. Systems Engineering, 11, 221-234.
Bozdogan, K. (2010) Towards An Integration Of The Lean Enterprise System, Total Quality
Management, Six Sigma And Related Enterprise Process Improvement Methods,
Cambridge, MA, MIT LAI / ESD Working Paper (ESD-WP-2010-05), scheduled for
publication in Encyclopedia of Aerospace Engineering.
Bozdogan, K., Deyst, J., Hoult, D. & Lucas, M. (1998) Architectural innovation in product
development through early supplier integration. R&D Management, 28, 163-173.
Bresman, P. H. M. (2004) Learning Strategies and Performance in Organizational Teams,
Cambridge, MA, PhD thesis, MIT and LAI.
Bresnahan, S. M. (2006) Understanding and Managing Uncertainty in Lean Aerospace Product
Development. Master's Thesis. Cambridge, Massachusetts Institute of Technology.
Brown, J. T. (2007) The Handbook of Program Management, New York, McGraw-Hill
Professional.
Browning, T. R. (1996) Systematic IPT Integration in Lean Development Programs, Cambridge,
MA, Master's thesis, LAI and MIT.
Browning, T. R., Deyst, J. J., Eppinger, S. D. & Daniel E. Whitney (2002) Adding Value in Product
Development by Creating Information and Reducing Risk. IEEE Transactions on
Engineering Management, 49, 443-458.
Cameron, B. G., Crawley, E. F., Loureiro, G. & Rebentisch, E. S. (2008) Value flowmapping:
Using networks to inform stakeholder analysis. Acta Astronautica, 62.
Page 64
LAI Whitepaper Series
Capaccio, T. (2010) Lockheed F-35’s Projected Cost Now $382 Billion, Up 65 Percent.
Bloomberg Businessweek, June 01, 2010.
Carreras, C. E. (2002) Opportunities for Lean Thinking in Aircraft Flight Testing and Evaluation,
Cambridge, MA, Master's thesis, LAI and MIT.
Cohen, J. L. (2005) United States Air Force Air Logistics Centers: Lean Enterprise
Transformation and Associated Capabilities, Cambridge, MA, Master's thesis, MIT and
LAI.
Cowap, S. A. (1998) Economic Incentives in Aerospace Weapon Systems Procurement, Master
Thesis, MIT and LAI.
Cunningham, T. W. (1998) Chains of Function Delivery: A Role for Product Architecture in
Concept Design, Cambridge, MA, PhD thesis, LAI and MIT.
Czerwonka, S. P. (2000) Avionics Life-Cycle Forecasting Model, Cambridge, MA, Master's
thesis, MIT and LAI.
Dare, R. E. (2003) Stakeholder Collaboration in Air Force Acquisition: Adaptive Design Using
System Representations. Cambridge, MA, PhD thesis, LAI and MIT.
DAU (2010a) Defense Acquisition Guidebook, https://dag.dau.mil, Defense Acquisition
University.
DAU (2010b) Public Access Course Material of the Defense Acquisition University,
https://myclass.dau.mil/webapps/portal/frameset.jsp?tab_id=_54_1,
Defense
Acquisition University.
Davidz, H. & Nightingale, D. (2002) AMRAAM Case Study Report - Lean Effects on Aerospace
Programs (LEAP) Project, Cambridge, MA, LAI Report.
Davidz, H. L. (2006) Enabling Systems Thinking to Accelerate the Development Senior Systems
Engineers, Cambridge, MA, PhD Thesis, LAI and MIT.
Deardorff, S. J. K. (2003) Institutionalizing Change in Aerospace Process and Product Settings,
Cambridge, MA, Master's thesis, MIT and LAI.
Deonandan, I., Valerdi, R., Lane, J. A. & Macias, F. (2010) Cost and Risk Considerations for Test
and Evaluation of Unmanned and Autonomous Systems of Systems, Loughborough,
UK, 5th IEEE International Conference on Systems of Systems Engineering, June 2010.
Derleth, J., Hitchings, S. & Rebentisch, E. (2003) Atlas Case Study Report - Lean Effects on
Aerospace Programs (LEAP) Project, Cambridge, MA, LAI Report.
Derleth, J. E. (2003) Multi-Attribute Tradespace Exploration and its Application to
Evolutionary Acquisition, Cambridge, MA, Master's thesis, LAI and MIT.
Dickerson, C. & Valerdi, R. (2010) Using Relational Model Transformations to Reduce
Complexity in SoS Requirements Traceability: Preliminary Investigation,
Loughborough, UK, 5th IEEE International Conference on Systems of Systems
Engineering, June 2010.
DoD (2008) Operation of the Defense Acquisition System, Department of Defense Instruction
(DoDI) 5000.02, Department of Defense.
Dorey, S. (2011) Enhancing cost realism through risk-driven contracting: Designing incentive
fees based on probabilistic cost estimates. Research report, Lean Advancement
Initiative, Massachusetts Institute of Technology, March 2011.
Downen, T. D. (2005) A Multi-Attribute Value Assessment Method for the Early Product
Development Phase With Application to the Business Airplane Industry, Cambridge,
MA, PhD Thesis, MIT and LAI.
Ferdowsi, B. (2003) Product Development Strategies in Evolutionary Acquisition, Cambridge,
MA, Master's thesis, LAI and MIT.
Page 65
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
Ferdowsi, B. & Haggerty, A. (2002) 737 Fuselage Case Study Report - Lean Effects on
Aerospace Programs (LEAP) Project, Cambridge, MA, LAI Report.
Ferdowsi, B. & Stanke, A. (2002) F-16 Case Study Report - Lean Effects on Aerospace Programs
(LEAP) Project, Cambridge, MA, LAI Report.
Ferns, C. (1991) Developments in programme management. International Journal of Project
Management, 9, 148-156.
Forseth, C. E. (2002) The Pursuit of Acquisition Intrapreneurs, Cambridge, MA, Research
Report, LAI and MIT.
GAO (2006) Defense Acquisitions - Major Weapon Systems Continue to Experience Cost and
Schedule Problems under DOD’s Revised Policy (GAO 06-368), Washington, D.C.,
United States Government Accountability Office.
GAO (2010) Defense Acquisitions - Managing Risk to Achieve Better Outcomes (GAO 10-374T),
Washington, D.C., United States Government Accountability Office.
Gillespie, D. M. (2009) Mission Emphasis and the Determination of Needs for New Weapon
Systems, Cambridge, MA, PhD thesis, LAI and MIT.
Glazner, C. G. (2006) Enterprise Integration Strategies Across Virtual Extended Enterprise
Networks: A Case Study of the F-35 Joint Strike Fighter Program Enterprise,
Cambridge, MA, Master's thesis, LAI and MIT.
Gordon, M., Musso, C., Rebentisch, E. & Gupta, N. (2009) The Path to Developing Successful
New Products. The Wall Street Journal, November 30, 2009.
Haughey, D. (2001) A perspective on programme management. Project Smart Website.
Hemann, J. M. (2005) Improving Complex Enterprises with System Models, Cambridge, MA,
Master's thesis, MIT and LAI.
Hernandez, C. M. (1995) Challenges and Benefits to the Implementation of Integrated Product
Teams on Large Military Procurements, Cambridge, MA, Master's thesis, LAI and MIT.
Herweg, G. M. & Pilon, K. E. (2001) System Dynamics Modeling for the Exploration of
Manpower Project Staffing Decisions in the Context of a Multi-Project Enterprise,
Master Thesis, LAI and Massachusetts Institute of Technology.
Hess, J., Agarwal, G., Deonandan, I., Kenley, B., Mikaelian, T. & Valerdi, R. (2010) Normative
and Descriptive Models for Test & Evaluation of Unmanned and Autonomous Systems
of Systems, Chicago, IL, 20th INCOSE Symposium, July 2010.
Hess, J. T. & Valerdi, R. (2010) Test and Evaluation of a SoS using a Prescriptive and Adaptive
Testing Framework, Loughborough, UK, , 5th IEEE International Conference on
Systems of Systems Engineering, June 2010.
Hut, P. M. (2008) How Program Management Differs From Project Management.
Ippolito, B. J. (2000) Identifying Lean Practices for Deriving Software Requirements,
Cambridge, MA, Master's thesis, LAI and MIT.
Jobo, R. S. (2003) Applying the Lessons of “Lean Now!” To Transform the US Aerospace
Enterprise - A study guide for government lean transformation, Cambridge, MA, LAI
Report.
Kato, J. (2005) Development of a Process for Continuous Creation of Lean Value in Product
Development Organizations, Master Thesis. LAI and Massachusetts Institute of
Technology.
Kenley, C. R. & Creque, T. R. (1999) Predicting Technology Operational Availability Using
Technical Maturity Assessment. Systems Engineering, 2, 198-211.
Kirtley, A. L. (2002) Fostering Innovation across Aerospace Supplier Networks, Master Thesis,
LAI and Massachusetts Institute of Technology.
Page 66
LAI Whitepaper Series
Klein, J., Cutcher-Gershenfeld, J. & Barrett, B. (1997) Implementation Workshop: High
Performance Work Organizations, Cambridge, MA, LAI Report RP97-02-34.
Knoblinger, C. (2011) PDSAT – A New Product Development Self-Assessment Tool, Munich and
Cambridge, MA, Technical University of Munich and Lean Advancement Initiative
(http://lean.mit.edu).
Labedz, C. S. & Harvey, R. K. (2006) Letterkenny Army Depot: Finance Innovations Support
Lean Six Sigma Success.
LAI (2001) Lean Enterprise Self-Assessment Tool (LESAT) Version 1.0.
Lamb, C. M. T. (2009) Collaborative Systems Thinking: An exploration of the mechanisms
enabling team systems thinking, Cambridge, MA, PhD thesis, LAI and MIT.
Loureiro, G., Crawley, E., Catanzaro, S. & Rebentisch, E. (2006) From Value to Architecture Ranking the Objectives of Space Exploration (IAC-06-D1.4.2). Proceedings of the
International Astronautical Congress (IAC 2006), Valencia, Spain, 2 - 6 October 2006.
Lucas, M. V. (1996) Supplier Management Practices of the Joint Direct Attack Munition
Program, Cambridge, MA, Master's thesis, LAI and MIT.
Lycett, M., Rassau, A. & Danson, J. (2004) Programme management: a critical review.
International Journal of Project Management, 22, 289-299.
Madachy, R. & Valerdi, R. (2010) Automating Systems Engineering Risk Assessment.
Proceedings of the 8th Conference on Systems Engineering Research, Hoboken, NJ,
March 17-19 2010.
Mahé, V. R. (2008) A Survey of Front End Modularity as an Automotive Architecture and its
Ability to Deliver Value, Cambridge, MA, Master's thesis, MIT.
Mandelbaum, J., Kaplan, W. S., McNutt, R. T. & Rebentisch, E. (2001) Incentive Strategies for
Defense Acquisitions, Washington, D.C., Department of Defense.
Mao, P., Cheng, H. & Xu, Q. (2008) Innovation of Large-Scale Project Multi-Programme
Construction Management Mode. IN MAO, P., CHENG, H. & XU, Q. (Eds.). IEEE.
Matty, D. (2010) Stakeholder Salience Influence on Bureaucratic Program Enterprise Value
Creation. Ph.D. Thesis, Massachusetts Institute of Technology, Cambridge, MA.
McKenna, N. (2006) The Micro-foundations of Alignment among Sponsors and Contractors on
Large Engineering Projects, Cambridge, MA, Master's thesis, MIT and LAI.
McManus, H. (2005) Product Development Value Stream Mapping (PDVSM) Manual,
Cambridge, MA, Lean Advancement Initiative (LAI) at MIT.
McNutt, R. T. (1998) Reducing DoD Product Development Time: The Role of the Schedule
Development Process, PhD Thesis, LAI and Massachusetts Institute of Technology.
McVey, M. E. (2002) Valuation Techniques for Complex Space Systems: An Analysis of a
Potential Satellite Servicing Market Cambridge, MA, Master's thesis, LAI and MIT.
Mikaelian, T. (2009) An Integrated Real Options Framework for Model-based Identification
and Valuation of Options under Uncertainty. Cambridge, Massachusetts Institute of
Technology.
Morgan, S. (1999) The Cost and Cycle Time Implications of Selected Contractor and Air Force
System Program Office Management Policies during the Development Phase of Major
Aircraft Acquisition Programs, Master Thesis, LAI and Massachusetts Institute of
Technology.
Nightingale, D. & Srinivasan, J. (2011) Beyond the Lean Revolution: Achieving Successful and
Sustainable Enterprise Transformation, New York, AMACOM.
Nightingale, D., Stanke, A. & Bryan, F. T. (2008) Enterprise Strategic Analysis and
Transformation (ESAT) - Version 2.0, Cambridge, MA, LAI Guide.
Page 67
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
Nightingale, D. J. & Mize, J. H. (2001) Development of a Lean Enterprise Transformation
Maturity Model. Information, Knowledge, Systems Management, 3, 15-30.
Norman, E. (2011) Introduction to Program Management. Presentation at the Lean Program
Management Workshop at the PMI Global Congress, Dallas-Fort Worth, TX, October
22 2011.
Nuffort, M. R. (2001) Managing Subsystem Commonality, Cambridge, MA, Master's thesis, LAI
and MIT.
Oehmen, J. (2005) Approaches to Crisis Prevention in Lean Product Development by High
Performance Teams and through Risk Management. Munich and Cambridge,
Technical University of Munich and LAI.
Oehmen, J. & Ben-Daya, M. (2010) A Reference Model for Risk Management in Product
Development Programs. Research Paper of the MIT-KFUPM Center of Clean Water and
Energy. Cambridge and Dhahran, MIT and KFUPM.
Oehmen, J., Ben-Daya, M., Seering, W. & Al-Salamah, M. (2010) Risk Management in Product
Design: Current State, Conceptual Model and Future Research. Proceedings of the
ASME 2010 International Design Engineering Technical Conferences & Computers and
Information in Engineering Conference IDETC/CIE 2010
Oehmen, J. & Rebentisch, E. (2010a) Compilation of Lean Now! Project Reports, Cambridge,
MA, LAI Report.
Oehmen, J. & Rebentisch, E. (2010b) Risk Management in Lean PD. LAI Paper Series "Lean
Product Development for Practitioners". Cambridge, MA, LAI and MIT.
Oehmen, J. & Rebentisch, E. (2010c) Waste in Lean Product Development. LAI Paper Series
"Lean Product Development for Practitioners". Cambridge, MA, LAI and MIT.
Oehmen, J. & Seering, W. (2011) Risk-Driven Design Processes – Balancing Efficiency with
Resilience in Product Design. IN BIRKHOFER, H. (Ed.) The Future of Design
Methodology. London, Springer.
OGC (2007) Managing Successful Programmes, London, Office of Government Commerce, The
Stationary Office.
OGC (2009) Managing Successful Projects with PRINCE2, London, Office of Government
Commerce, The Stationary Office.
Oppenheim, B. W., Murman, E. M. & Secor, D. A. (2011) Lean Enablers for Systems
Engineering. Systems Engineering, 14, 29-55.
Paduano, R. (2001) Employing Activity Based Costing and Management Practices Within the
Aerospace Industry: Sustaining the Drive for Lean, Cambridge, MA, Master's thesis,
MIT and LAI.
Patanakul, P. & Milosevic, D. (2008) A competency model for effectiveness in managing
multiple projects. The Journal of High Technology Management Research, 18, 118131.
Patanakul, P. & Milosevic, D. (2009) The effectiveness in managing a group of multiple
projects: Factors of influence and measurement criteria. International Journal of
Project Management, 27, 216-233.
Pessôa, M. V. P. (2008) Weaving the waste net: a model to the product development system
low performance drivers and its causes, Cambridge, MA, LAI White Paper 08-01.
PMI (2008a) A guide to the project management body of knowledge (PMBOK guide), Drexel
Hill, PA, Project Management Institute.
PMI (2008b) The Standard for Program Management, Newton Square, PA, Project
Management Institute.
Page 68
LAI Whitepaper Series
PMI (2011) The PgMP role delineation study. http://www.pmi.org/Certification/ProjectManagement-Professional-PgMP/Updates-to-PgMP-Certification-Exam/PgMP-RoleDelineation-Study.aspx.
Pomponi, R. A. (1998) Organizational structures for technology transition : rethinking
information flow in the integrated product team, Cambridge, MA, PhD thesis, LAI and
MIT.
Rebentisch, E. (1996) Preliminary Observations on Program Instability, Cambridge, MA, LAI
Whitepaper LEAN 96-03.
Rhodes, D. H., Valerdi, R. & Roedler, G. J. (2009) Systems Engineering Leading Indicators for
Assessing Program and Technical Effectiveness. Systems Engineering, 12, 21-35.
Roberts, C. J. (2003) Architecting Evolutionary Strategies Using Spiral Development for Space
Based Radar, Cambridge, MA, Master's thesis, LAI and MIT.
Roedler, G., Rhodes, D. H., Schimmoller, H. & Jones, C. (2010) Systems Engineering Leading
Indicators Guide (Version 2.0), INCOSE-TP-2005-001-03.
Ross, A. M. (2003) Multi-Attribute Tradespace Exploration with Concurrent Design as a ValueCentric Framework for Space System Architecture and Design, Cambridge, MA,
Master's thesis, LAI and MIT.
Roth, G. (2006) Distributing Leadership Practices for Lean Transformation. Reflections - The
SoL Journal on Knowledge, Learning, and Change, 7, 15-31.
Roth, G. (2008) The Order and Chaos of the Learning Organization. IN CUMMINGS, T. (Ed.)
Handbook of Organization Development. Newbury, CA, Sage.
Shoepe, T. V. (2006) In Pursuit of Understanding Lean Transformation - Capturing Local
Change Journeys in a DoD Field Environment, Cambridge, MA, LAI Research Paper.
Shroyer, E. (2002) Lean Transition of Emerging Industrial Capability (LeanTEC), Final Report,
Cooperative Agreement F33615-97-2-5153, U.S. Air Force and Boeing Company.
Siegel, L. R. (2004) Measuring and Managing Intellectual Capital in the U.S. Aerospace
Industry, Cambridge, MA, Master Thesis, LAI and MIT.
Silva, L. M. (2001) A Partitioning Methodology for Helicopter Avionics System with a focus on
Life Cycle Cost, Cambridge, MA, Master's thesis, LAI and MIT.
Spaulding, T. J. (2003) Tools for Evolutionary Acquisition: A Study of Multi-Attribute
Tradespace Exploration (MATE) Applied to the Space Based Radar, Cambridge, MA,
Master's thesis, LAI and MIT.
Stagney, D. B. (2003) The Integrated Concurrent Enterprise, Cambridge, MA, Master's thesis,
LAI and MIT.
Stanke, A., Nightingale, D. & Bryan, F. T. (2008) Enterprise Strategic Analysis and
Transformation (ESAT) - Facilitator’s Guide - Version 2.0, Cambridge, MA, LAI Guide.
Stanke, A. K. (2006) Creating High Performance Enterprises, PhD Thesis, LAI and
Massachusetts Institute of Technology.
Susman, G. & Petrick, I. (1995) Product Development Team Effectiveness, Cambridge, MA, LAI
Whitepaper 95-06.
Tang, V. (2006) Corporate Decision Analysis: An Engineering Approach, Cambridge, MA, PhD
thesis, LAI and MIT.
Tang, V. & Otto, K. N. (2009) Multifunctional Enterprise Readiness: Beyond the Policy of BuildTest-Fix Cyclic Rework. Proceedings of the ASME 2009 International Design
Engineering Technical Conferences & Design Theory and Design IDETC/DTM 2009 ,
August 30 - September 2, 2009, San Diego, California, 1-9.
Tondreault, J. P. (2003) Improving the Management of System Development to Produce More
Affordable Military Avionics Systems, Cambridge, MA, Master's thesis, LAI and MIT.
Page 69
J. Oehmen, E. Rebentisch and K. Kinscher: Program Management
Utter, D. A. (2007) Performing Collaborative, Distributed Systems Engineering (CDSE): Lessons
Learned from CDSE Enterprises, Cambridge, MA, Master's thesis, LAI and MIT.
Valerdi, R. (2005) The Constructive Systems Engineering Cost Model (COSYSMO), Los Angeles,
CA, PhD Thesis, USC.
Valerdi, R. (2010) Heuristics for Systems Engineering Cost Estimation. IEEE Systems Journal.
Valerdi, R., Rieff, J. E. & Wang, G. (2007) Lessons Learned From Industrial Validation of
COSYSMO, San Diego, CA, 17th INCOSE Symposium, June 2007.
Wagner, C. (2007) Specification Risk Analysis: Avoiding Product Performance Deviations
through an FMEA-based Method. Munich and Cambridge, Technical University of
Munich and LAI.
Walton, M. A. (1999) Identifying the Impact of Modeling and Simulation in the Generation of
System Level Requirements, Cambridge, MA, Master's thesis, LAI and MIT.
Weigel, A. L. (2000) Spacecraft System-Level Integration and Test Discrepancies:
Characterizing Distribution and Costs, Cambridge, MA, Master's thesis, MIT and LAI.
Whitaker, R. B. (2005) Value Stream Mapping and Earned Value Management: Two
Perspectives on Value in Product Development, Cambridge, MA, Master's thesis, MIT
and LAI.
Wirthlin, J. R. (2000) Best Practices in User Needs / Requirements Generation, Cambridge, MA,
Masters thesis, LAI and MIT.
Wirthlin, J. R. (2009) Identifying Enterprise Leverage Points in Defense Acquisition Program
Performance, PhD Thesis, LAI and Massachusetts Institute of Technology.
Wirthlin, J. R., Seering, W. & Rebentisch, E. (2008) Understanding Enterprise Risk Across an
Acquisition Portfolio: A Grounded Theory Approach. Seventh National Symposium on
Space Systems Engineering & Risk Management, Los Angeles, CA, February 26-29,
2008.
Wright, M. R. (2003) Strategies for dealing with instabilities in a complex, multi-project
product development system engineering environment, Master Thesis, LAI and
Massachusetts Institute of Technology.
Page 70