Measuring Engineering Quality in Airplane
Development
by
Ammar Asfour
B.Sc., University of Texas, Austin, 2008
Submitted to the Department of Aeronautics and Astronautics and the MIT Sloan School
of Management in partial fulfillment of the requirements for the degrees of
wo
Master of Science in Aeronautics and Astronautics
and
C=)
LL0
Z
Master of Business Administration
2
Uj J
in conjunction with the Leaders for Global Operations Program at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
June 2015
@
Ammar Asfour, MMXV. All rights reserved.
The author hereby grants to MIT permission to reproduce and to distribute publicly paper and electronic
copies of this thesis document in whole or in part in any medium now known or hereafter created.
Signature redacted
. . . . ./.
. .. .. .. ..
.
A uthor .....
Department of Aeronautics and Astrona
s and the MIT Sloan School of Management
May 8 2015
Signature redacted
............
C ertified by .... . . . . . . . . . . . .WO - - - - - - - - - - Olivier de Weck, Thesis Supervisor
and Engineering Systems Division
Astronautics
and
Aeronautics
of
Professor
Certified by.
Sianature redacted
6)
Approved by .......
...............................-
Bradley Morrison, Thesis Supervisor
Senior Lecturer, MIT Sloan School of Management
Signature redacted.
Paulo C. Lozano
Chair, Aeronautics and Astronautics Gra uate Program Committee
A
Approved by.............
Signature
redacted
. .... ..
..........
Maura Herson
Director, MBA Program, MIT Sloan School of Management
<
Cif.
Measuring Engineering Quality in Airplane Development
by
Ammar Asfour
Submitted to the Department of Aeronautics and Astronautics and the MIT Sloan School
of Management on May 8, 2015, in partial fulfillment of the requirements for the degrees of
Master of Science in Aeronautics and Astronautics
and
Master of Business Administration
Abstract
This project was motivated by the desire to apply quality metrics to the multiple stages of the
airplane development process at Boeing Commercial Airplanes. This project first identified
integration and process discipline as most critical towards final quality of the engineering
work. Integration, defined as the path and connectivity between teams and activities, was
studied by analyzing performance of a small engineering support team. To understand the
effects of early stage quality on later stages, i.e. process discipline, a system dynamics model
was developed focusing on the design and development of components with suppliers. The
case study regarding integration focused on the engineering work as a four-steps process:
Inputs, Engineering Activities, Output and Customer Review. All unplanned reworked
deliverables of a 5-7 members engineering team were analyzed. The study tracked the process
step at which the error was first caused. The results found that 21% of unplanned engineering
rework was caused due to inadequate delivery of inputs to the requested engineering work.
Furthermore, the 21% of unplanned engineering rework had the highest hours per reworked
deliverable of any stages. Over all, 75% of engineering rework was due mainly to the process
rather than the actual technical engineering work. The system dynamic modeling achieved
two main results: showcased the necessity to simplify the process, and the importance of
accounting for iterations in engineering. Through the group-modeling discussions with the
process owner, it was evident the need to provide clear checkpoints and reviews of the
engineering work. Furthermore, discovering engineering rework within a given stage has the
same effect as delivering first-pass engineering quality. This project provided a methodology
to work with engineering teams to measure their quality performance. Furthermore, it
has the potential to show the thresholds of quality from one stage to another in Airplane
Development.
Thesis Supervisor: Olivier de Weck
Title: Professor of Aeronautics and Astronautics and Engineering Systems Division
Thesis Supervisor: Bradley Morrison
Title: Senior Lecturer, MIT Sloan School of Management
2
Acknowledgments
First, I would like to express my gratitude to the Boeing Airplane Development Quality
Organization and its leadership for making this opportunity possible. While at Boeing, this
project was received with the utmost courteous and supportive attitudes and there are many
without whom this work could have not been completed. Thank you.
Then, I would like to thank my advisors for providing the guidance and support through
a challenging journey. Their insight allowed me to navigate a sea of complex managerial and
engineering issues to understand and analyze.
The Leaders for Global Operations program made this opportunity possible. The LGO
Network, my MIT family, has inspired me every day to challenge myself. My classmates
have made this journey through the LGO program a unique and an enriching one.
Most importantly, my sincere and deepest thank to my family to whom I am eternally
indebted for their love and faith. My parents, Ahmad and Huda, my sister, Yasmin, and my
brother, Omar were my biggest supporters. Thank you. Shukran.
3
Contents
1 Introduction
2
3
4
9
1.1
Problem Statem ent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
1.2
Project Motivation and Company Background . . . . . . . . . . . . . . . . .
11
1.3
Rework in New Product Development . . . . . . . . . . . . . . . . . . . . . .
12
1.4
Study Approach and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
Critical Factors of Engineering Quality in Airplane Development
15
2.1
Engineering Quality: The Magnitude of the Challenge . . . . . . . . . . . . .
16
2.2
Product's Complexity Impact on Product Development . . . . . . . . . . . .
17
2.3
Hypothesis: Integration and Process Discipline . . . . . . . . . . . . . . . . .
20
Integration in Airplane Development Process
22
3.1
. . . . . . . . . . . . . . . . . . . . . . . .
23
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
Case Study Scope and Approach
3.1.1
Scope
3.1.2
A pproach
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
3.2
Case Study Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
3.3
Conclusions and Recommendations
27
. . . . . . . . . . . . . . . . . . . . . . .
Applying System Dynamics to Airplane Development Process
29
4.1
System Dynamics in Product Development . . . . . . . . . . . . . . . . . . .
29
4.1.1
Applications of System Dynamics in Product Development . . . . . .
30
4.1.2
System Dynamic Model Structure for Product Development
. . . . .
32
. . . . . . . . . . . . . . . . . . . . . . . . .
37
4.2
Modelling Approach and Scope
4
4.3
4.4
5
4.2.1
Approach
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
4.2.2
Scope
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
4.2.3
The Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
R esults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
. . . . . . . . . . . .
39
4.3.1
Quality of the Requirement Development Stage
4.3.2
Quality of the Design Stage
. . . . . . . . . . . . . . . . . . . . . . .
41
4.3.3
Quality of Both Stages . . . . . . . . . . . . . . . . . . . . . . . . . .
42
. . . . . . . . . . . . . . . . . . . . . . .
42
4.4.1
Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
4.4.2
Recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
45
Conclusions and Recommendations
47
Conclusions
A Interview Questions and Sample Answers
49
A.1
Interview Guidelines
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
49
A.2
Samples of Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
50
5
List of Figures
1-1
High view of the airplane development process at Boeing Commercial Airplanes 10
1-2
The additional costs and time delays for both the 787Dreamliner and 747-81
programs at Boeing. [6][3][11][23][24]
. . . . . . . . . . . . . . . . . . . . . .
12
1-3
The main stages and phases of the project supporting this thesis . . . . . . .
14
2-1
An analysis of number of engineering rework in Airplane Systems of Airplane
B program.
The hours and number of changes are normalized within this
table. The table emphasizes the number of changes occurring at a design and
requirem ents level.
2-2
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The percentage of non-conformities that required engineering change requests.
It is calculated as an average of each 10 airplanes
2-3
17
. . . . . . . . . . . . . . .
18
A study comparing the number of releases of design work related to the EFA
. . . . . . . . . . . . . . . . . .
18
3-1
The steps followed in studying the integration related case. . . . . . . . . . .
24
3-2
The results of the case study. The data here is normalized. . . . . . . . . . .
25
4-1
deWeck's system dynamic model.[8] . . . . . . . . . . . . . . . . . . . . . . .
31
4-2
The foundation of Cooper's models, the rework loop. [14] . . . . . . . . . . .
32
4-3
The causal loops presented by Cooper which tells the story of project man-
function within different airplane programs.
agem ent at Flour. [14]
4-4
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
Multiple phases representation of any product development project as shown
by Ford's 1997 work. [12] . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
33
4-5
The basic stock and flow structure of a single product development phase as
shown in Ford's 1997. [12]
4-6
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
Quality factors affecting the basic single product development phase in Ford's
1997 work. This graph also shows the internal concurrences. [12] . . . . . . .
4-7
34
35
The addition of upstream and downstream quality effects. This figure shows
how a single product development phase connects with other phases through
stocks of tasks to be coordinated. [12] . . . . . . . . . . . . . . . . . . . . . .
36
4-8
The stages represented in the model built for this case
38
4-9
The results of varying quality factors in components requirement stage while
. . . . . . . . . . . .
quality is constant in other stages. This figure shows the effects on time of
completion.
The time of completion is normalized against an ideal quality
case of zero errors.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40
4-10 The results of varying quality factors in components requirement stage while
quality is constant in other stages. This figure shows the effects on amount of
work needed. The amount of work is normalized against an ideal quality case
of zero errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
41
4-11 The results of considering schedule pressures while varying quality factors in
components requirement stage with quality is constant in other stages. This
figure shows the effects on time of completion.
The time of completion is
normalized against an ideal quality case of zero errors.
. . . . . . . . . . . .
42
4-12 The results of considering schedule pressures while varying quality factors in
components requirement stage with quality is constant in other stages. This
figure shows the effects on amount of work needed. The amount of work is
normalized against an ideal quality case of zero errors.
. . . . . . . . . . . .
43
4-13 The results of varying quality factors the design stage while quality is constant
in other stages. This figure shows the effects on time of completion. The time
of completion is normalized against an ideal quality case of zero errors.
7
. . .
43
4-14 The results of varying quality factors in the design stage while quality is
constant in other stages.
This figure shows the effects on amount of work
needed. The amount of work is normalized against an ideal quality case of
zero errors.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
4-15 The results of varying quality factors in both the components requirement
stage and the design stage. This figure shows the effects on time of completion.
The time of completion is normalized against an ideal quality case of zero errors. 44
8
Chapter 1
Introduction
The first chapter will introduce the problem statement, the motivation and the company
background, the scope, and approach of the study.
1.1
Problem Statement
Airplane Development process entails a long process and years of engineering development.
For example, the development of 787-8 took 7 years since the program launch in 2004 [11],
the Airbus A350XWB was launched in 2004 and finally delivered in 2015. [17]. Moreover,
even airplanes that are new variants of established designs take 3-4 years. The development
of the Airbus A330NEO which is a variant on the A330 is anticipated to take 3-4 years. [2].
The complication, magnitude, and scope of new plane development dictate such a lengthy
time.
The Boeing 777 Aircraft is constituted of 3,000,000 parts provided by 500 global
suppliers. [5]
It's therefore not uncommon for integration and testing between parts and systems to
happen years after the initial requirements, design, and engineering details completed. Figure
1-1 shows, at a high level, the Airplane Development process adopted at Boeing Commercial
Airplanes. This figure shows a the conceptual approach of Boeing's airplane development.
The actual development process is a detailed multiple stages processes. At the end of each
stage there is a review of particular goals, metrics, and milestones. The v-model shows how
the process begins with a very high-level, customer focused approach.
9
At that stage the
..n
Product
-
Definition
Transition to full
production
Certification
Requirements
Detailed Design L
Individual
components
Integration and
Testing
Component level
development
level
Time
Figure 1-1: High view of the airplane development process at Boeing Commercial Airplanes
customer needs are defined, and the product's answer to those needs is proposed. The next
stage begins the actual overall decision of airplane design. For example, will the airplane
have two or four engines. Then the actual requirements for the airplane and consequently the
different systems begins. This point is the most detailed point of the development process;
the engineering teams are working developing the particular hydraulic systems for the tires,
for example. Once the different systems are designed, prototyped, and individually tested,
the integration and testing of the large sub-assemblies begin. Finally, the whole airplane is
integrated together and goes through testing which leads to the FAA certification signaling
that the airplane is ready to move into production.
During the integration and testing phase several issues may appear regarding the technical and engineering work that would require rework of such design. In this instance, the
engineering work is considered as "poor quality." Given the months, if not years, of work
put in by multiple teams and functions addressing an engineering rework become costly and
timely. Furthermore, the increase in cost and time are exponential due to the significant
effort put in place for change management.
Engineering Quality is defined as delivering the engineering work per the customer re'The actual process is a proprietary detailed process that entails several stages and reviews at the end of
each stage.
10
quired performance to the consumer (manufacturing/production) on time and on budget.
The Airplane Development Quality Organization was established to improve the quality of
the Engineering work. The motivation of this project is to first understand the most critical levers for engineering quality, and second to measure the quality of engineering such as
to detect poor engineering quality using leading indicators to eliminate poor quality from
propagating to later stages.
1.2
Project Motivation and Company Background
Boeing is facing a fierce competition from Airbus. As a result of this competition, a surge in
new Airplane Development programs occurred. As of Jan 2015, Boeing has four new airplanes
in the development phase. Consequently, Boeing Commercial Airplanes developed a new
organization, Airplane Development Organization.
This new organization was established
to focus on three things: breaking the cost curve, shortening the development time-line, and,
finally, standardizing processes and tools of the development process.
The industry competition and market pressure amplified the necessity to face, head-on,
recent challenges of new airplane developments.
Both the 747-81 and the 787 Dreamliner
were faced with obstacles and challenges that created significant delays of their development
schedule. Figure 1-2 shows the significant cost of development in addition to the delays of
both programs.
11
Figure 1-2: The additional costs and time delays for both the 787Dreamliner and 747-81
programs at Boeing. [6][3][11][23][24]
1.3
Rework in New Product Development
In this section, we review the academic literature to discuss whether engineering change and
rework is prevalent to new product development across industries.
This thesis is focused on a project addressing challenges faced by Boeing in product development. However, engineering rework is a challenge in the aerospace industry in-specific
and in product development at large. This section presents examples of product development rework challenges connecting this work to the larger discussion of rework in product
development.
Kato [18] utilized extensive value stream mapping to study the product development
at a railway vehicles manufacturer.
His goal of the study was to apply continuous lean
improvement to the product development process. He identified nine indicators to measure
waste, i.e. non-value added time, in the product development process. Of these indicators,
rework has the second highest rate of occurrence and the largest average wasted time per
occurrence among the indicators. Furthermore, he noticed that the average time spent on
one occurrence of rework increased significantly in later stages of the project.
Repenning studies the phenomena of firefighting in product development, which he defines
12
as " ... the allocation of scarce resources to solve unanticipated problems or fires ... "[22] The
study of a solving problems in product development processes is an indicator of the scale and
magnitude of such challenges. He builds system dynamic model with the stock, Design Tasks
Awaiting Rework, at the center of the model. His conclusions highlight that multi-phases
product development environments have an equilibrium points beyond which organizations
may seem to be stuck in "firefighting" mode. This insight highlights the importance and
significance of avoiding a firefighting mode to begin with.
Academic literature also discusses that many companies have poor capacity to develop
successful plans to develop new products.[10] "...Their processes are highly iterative and
depend on the enthusiasm and expertise of the individuals involved.
They often succeed
in spite of their plans, not because of them ... " This supports the importance of this work
towards understanding improvements of new product development.
1.4
Study Approach and Scope
This study tackled a very large scope in the Airplane Development process. It was, therefore, critical to define a hypothesis through qualitative analysis and understanding of the
challenges. At first, interviews with over 15 Engineers and Leaders took place to understand
some of the challenges related to the development process. Along this process, a few examples and case studies were discussed to better define the hypothesis. A hypothesis was
defined and validated with the stakeholders. It was stated that Engineering Quality can be
measured by defining and measuring process discipline as well as integration.
Figure 1-3
shows the project approach.
Thereafter, the project diverted into two case-studies to prove the hypothesis. A study
of integration relied on an analysis of all deliverables of a small engineering support team.
Integration is defined as the pathway of connections between activities. Therefore, to measure integration adequately, each team, i.e. activity, must have a clear measurement and
definition of it's input and output. The second case study uses a system dynamics modeling
to understand the effect of poor engineering quality carried to later stages in the development
process consequently highlighting how to measure process discipline quality.
13
Identifying the challenge and developing a hypothesis
System Dynamic Modeling
Developed a system dynamic
model to understand the ripple
effect of quality between
different process stages
Data Analysis
Reviewed the work of a small
engineering team to understand
the sources of rework
Results and Opportunities
Figure 1-3: The main stages and phases of the project supporting this thesis
14
Chapter 2
Critical Factors of Engineering
Quality in Airplane Development
In Engineering Systems, Quality is the most frequently mentioned of all the '-lity' attributes.
[9] However, "Quality" definition varies and depends on the context and organization. Furthermore, it is critical to note that quality in manufacturing is significantly more studied and
understood than quality of engineering development. In the context of this study, Quality
of engineering deliverables focused on the elimination of unplanned engineering rework.
This thesis focuses its study on the development of Airplane Systems. It was suggested
by the stakeholders of this project as a focus of the study. Airplane Systems entail many
functions within such as Hydraulics, Electrical, Avionics, Mechanical, etc. For confidentiality
reasons specific Systems will not be named.
This chapter identifies two critical factors of Quality through stakeholders' input and
analysis of data.
The first section of this chapter asks the question whether engineering
quality is a challenge. The second section proposes the hypothesis of his thesis.
15
2.1
Engineering Quality: The Magnitude of the Challenge
The design and development of any new product requires iteration.
The occurrence of
unplanned iterations poses the biggest challenge. To identify whether this is an issue, we
interviewed over 15 of Airplane Systems senior managers and engineers. Then we studied
non-conformities reported on a the production line relating to the production of Airplane A.
Stakeholders' input referenced the relationship between the different functions. For example, the following statements were repeated several times " this [other team] never give us
complete deliverables," or "we finished our work, then the [other team] told us they changed
their design " Furthermore, there were multiple references to schedule schedule "we have to
find a way to meet schedule," or "we didn't have the resources to meet our timeline." These
quotes can be of any place and all too common in any workplace. They don't indicate an
Engineering Quality Challenge.
A previous study performed regarding all rework hours for Airplane Systems of Airplane
B, figure 2-1, showed the significant hours and cost contributed to engineering rework. The
scope of this relates to only Airplane Systems. Airplane Systems group includes all systems
such as electrical, avionics, hydraulics, etc on the airplane. This study was performed by a
senior engineer of airplane B program. It was done through categorization and estimation
of rework hours of each individual engineering rework.
An analysis was performed of non-conformities logged on production line of airplane A,
figure 2-2.
Non-conformities are logged when any occurrence or incident take place that
does not allow the front line worker to complete a given job on the assembly of airplane A.
These non-conformities can range from as simple as a small tool indenture left on a screw,
to as major as a request for an engineering change.
A request for an engineering change
means that the design of a particular aspect needs to be changed so that the mechanic can
complete the job. Figure 2-2 shows percentage of a non-conformities requiring engineering
rework relevant to all non-conformities.
The percentage of engineering rework peaks to
reach almost 100% of all non conformities between aircrafts 130 and 140.
the magnitude of the challenge.
This signals
Engineering rework was still being requested even after
16
Design
28
31,136
Requirements
26
21,836
System Integration
4
4,277
Supplier/Partners
2
1,113
All numbers are normalized within this table
Figure 2-1: An analysis of number of engineering rework in Airplane Systems of Airplane
B program. The hours and number of changes are normalized within this table. The table
emphasizes the number of changes occurring at a design and requirements level.
producing 140 aircrafts. Furthermore,while the overall non-conformities number decreased
significantly as more aircrafts were produced (trend not shown in this figure), the percentage
of engineering rework continued to increase. This indicates that engineering rework persisted
longer than production errors.
Another case study of the same EFA function, figure 2-3, was performed across multiple airplane programs. The following figure shows the percentage of unplanned releases of
engineering designs across different airplane programs.
2.2
Product's Complexity Impact on Product Development
This thesis identifies and focuses on two factors that are most critical to engineering rework
in airplane development. In the previous section, we review the impact of engineering rework challenge. This section emphasizes the magnitude of the challenge by discussing the
impact of complexity on product development with the assumption that Airplanes are very
complex products. However, academic literature identified complexity of a product as one
17
100.0%
90.0%
80,0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
10
20
30
130
100 110 120
90
80
70
60
50
40
NUMBER OF AIRPLANE PRODUCED IN ORDER OF THEIR PRODUCTION
-%
140
150
160
Non-conformities that require Engineering Rework
Figure 2-2: The percentage of non-conformities that required engineering change requests.
It is calculated as an average of each 10 airplanes
B
600% baseline
Figure 2-3: A study comparing the number of releases of design work related to the EFA
function within different airplane programs.
18
of three main factors affecting the impact of engineering rework.
[15] The complexity of
airplane development, therefore, mandates that our view of the challenge cant be decoupled
from understanding the impact of complexity of the system on the product development
process. In this section, we review academic literature that analyzes the impact of complexity on product development. The academic literature in this area is is abundant whether
qualitatively or quantitatively.
First, this section will review the definition of engineering
complexity. Then we highlight a qualitative, conceptual approach to the issue as well as a
quantitative modeling analysis.
Product complexity can take on many definitions. However, Jarratt and others highlight
the prominence of understanding complexity as the linkages between elements of a product
and their interaction together as a complete system or as a sub-system of a larger system.
[15] Understanding elements of a product is a function of its description as well as the organizational hierarchy attached to such break-down. This is an area that is heavily dependent
on the organization within which the product is developed. Regardless of the organizational
development, a product can be broken down to the number of systems, sub-systems, or
unique parts constituting it as a measure of complexity.
In a field study conducted by Jongbae and Wilemon [16] 32 companies were surveyed
regarding complexity and its impact on new product development.
The work was moti-
vated by the lack of understanding of the effects of complexity on new product development.
The survey focused on participants from the new product development organizations within
technology companies located in the United States North East region. The study results
showed that 66% of participants indicated that complexity of products led to development
cycle delays. Another significant consequence to product complexity is developing organizational and strenuous relationships within the people of a given organization. This negative
consequence is very difficult to quantify, even though it was very prominent in the results of
the survey. The survey highlights that a companys capacity to manage complexity well is a
strong competitive advantage against competitors.
The work of Weilin and Young in 2012 [26] modeled engineering change management processes in new product development through looking at the effects of the activity uncertainty,
design solution uncertainty, and environmental uncertainty, on the new product development
19
rework. When exploring overlap and rework probability, Weilin found that increase in overlap will shorten lead time at high and low rework probabilities. However, it will increase the
amount of efforts expended in completing the project. This result does not take into account
the large increase of rework effect on quality of engineering work as well as probability of
even creating more rework.
Finally, The Rule of 10 suggested first by Boehm in 1981 established the base ground to
many academic papers discussing the impact complexity an rework. [4] [13] Boehm concludes
that the cost of catching rework is exponential as a project moves from development phase
to the other. This theory suggests that performing ten changes at a given phase is equal in
cost to performing only one change at a later stage. While this theory has been discussed in
depth by many other papers whether in support or in criticizing, [13] the gist of it remains
critical 35 years later. It is far more difficult and costly to carry an engineering change at
later stages of a product development process.
2.3
Hypothesis: Integration and Process Discipline
The hypothesis of this thesis is that unplanned engineering rework can be significantly reduced through the following two factors:
1. Improved integration between engineering teams: clear understanding and measurable
quality of the input information required to complete tasks successfully will eliminate
the most time-consuming unplanned reworks.
2. Process discipline: In the multiple stage development process, each stage must have
a clearly defined set of measurable thresholds that qualify it as complete and triggers
the following stage.
The hypothesis is built on findings explained in section 2.1 of the current state of the
development process. This thesis will discuss the magnitude of improvement possible with
the added improvements. The discussion of this thesis does not eliminate the organizational
aspects affecting the challenge of unplanned engineering rework. The hypothesis implies that
20
the introduction of measurable metrics will increase transparency and accountability both
of which will drive improvements of organizational factors related to the challenge.
Both factors are founded in that a process must have a clearly defined inputs and outputs of each step. However, implementation and adoption of both factors are significantly
different. The first factor points to process management at engineering teams level. A measured metric clearly identifying the quality of required input information empowers a given
team to start a task only when they have the resources needed to complete a task. In lean
manufacturing, this practice is referred to as "full kitting." Consequently, this will drive an
improved communication behavior between all teams. For example, if team B relies on data
from team A to complete their task, and team A knows they are held accountable for any
delays due to their deliverables quality, both teams will be significantly more inclined to
communicate and discuss the tasks.
The second factor empowers the leadership of the development process at large. Using
measured thresholds and metrics adds quality to the discussion of managing a development
schedule. This thesis argues that it would be more beneficial to deviate from planned timeline and schedule to improve quality of a deliverable before starting with the new stage of
development.
21
Chapter 3
Integration in Airplane Development
Process
This chapter addresses the first factor of the hypothesis, integration, through quantitative
study of unplanned rework in a small engineering team. There is no argument about the
complexity of the airplane development process simply based on the very large number of
engineering teams involved. The complexity itself that makes integration a significant challenge is unavoidable. However, this chapter takes a closer look at a single small engineering
team as an example.
Section 3.1 develops the academic background and foundation driving this study. Then in
the second section the approach and scope of the study is discussed. Section 3.3 discusses the
results of the study whereas section 3.4 puts forward the conclusions and recommendations.
Systems engineering as a discipline is dedicated to understanding the complexity of
systems which are defined as a set of interacting components
technical artifacts
with
well-defined behavior and a well-defined function or purpose, like the individual technical
artifact. [9]
This thesis relies on Spear's definition of systems
[25] as a pathway of connected activities
producing a measured output. The great emphasis on measured outputs and clear activities
is articulated in this case study through breaking down the process into simple stages: input,
technical analysis, and output.
22
Case Study Scope and Approach
3.1
Scope
3.1.1
The study focuses on an engineering team of 5-7 engineers hereafter referred to as Team TA.
This team is considered a supporting function. Team TA is not responsible for developing
or designing any particular systems or components of the airplane. However, it's specialized
in an attribute, referred to as TA, required by all other engineering teams. For example, if
the avionics engineering system requires an analysis of TA attributes of its avionics systems,
it will request such analysis from Team TA.
The study reviewed all deliverables by Team TA performed for only one airplane development program over a time period of one year and 4 months. A cut-off date was determined
and no deliverables completed during the study were included.
Furthermore, the study's
goal was to understand Team's TA performance regarding unplanned rework.
Therefore,
there was a focus on the step during which the unplanned rework was caused rather than
the root cause of the problem such as tools or experience.
Approach
3.1.2
The study was done in collaboration with the senior managers in Team TA. It followed the
following steps, figure 3-1:
1. Leadership buy-in: completion of this project required additional workload from Team
TA. Their senior leadership approval was critical engage with Team TA. The leadership
was interested in understanding and measuring the quality performance. Furthermore,
they were interested in setting-up a repeatable method to perform similar analysis with
other engineering teams.
2. Developed project baseline: A discussion of Team TA's own view of their performance
took place.
At that point, there were no metrics measuring adequately the perfor-
mance of the team. Therefore, it was essential to establish the Team's view of their
own performance regarding unplanned rework. Team TA also highlighted areas, they
believe, are the cause of all unplanned rework at this stage.
23
Leadership
Buy-in
Process
Mapping
Performance
Baseline
Review
Deliverables
Figure 3-1: The steps followed in studying the integration related case.
3. Mapped current process in detail: process mapping workshops were facilitated with
Team TA. Their process was mapped as practiced currently in details. It is very critical
for the study to eventually simplify the process into input-activity-output-customer
review; however, these workshops allowed Team TA to break down their processes to
their very details before grouping them again in the input-output categories.
4. Reviewed deliverables within the scope: Finally, all of the deliverables, defined in
the scope above, were reviewed individually. First, the unplanned rework deliverables
were identified. Second, the source error that caused each deliverable was determined.
Finally, the step at which the causing error occurred was established. If an unplanned
rework had several errors, only the error relating to the earliest step was considered.
The rational is that errors occurring in later stages are most probably caused by errors
made in earlier steps.
24
79% First Pass Yield
% of Rework Total
21%
4.9
2.7
Hours/rework*
Hours have been normalized for proprietary reasons
Figure 3-2: The results of the case study. The data here is normalized.
3.2
Case Study Results
Figure 3-2 shows the simplified version of the process. In this, the Inputs step is the step
where Team TA receives tasks requests from other teams. The requests are usually a set of
information defined by the customer (Engineering teams requesting analysis are referred to
as customers). Team TA currently may either request some additional information or move
forward using knowledge and experience based assumptions. The modeling stage is where
the engineering analysis takes place. After the analysis is completed, the engineers document
their analysis to communicate to the customer. The customers, upon receiving the analysis,
review the results and either accept or not accept.
The team had at first created a baseline assumption that unplanned rework is majorly due
to communication errors as well as documentation errors. There was an apparent sentiment
that unplanned rework is mostly caused by customers.
The result of the study is highlighted in figure 3-2.
Of all the deliverables completed
in the determined scope, 21% of all deliverables were unplanned reworked. New requests of
Team TA that were not planned and not an iteration on a planned task are not considered in
this percentage. The term Firstpass Yield refers to all tasks completed without unplanned
iterations. Therefore, in this case, Team TA achieved 79% Firstpass Yield. The figure also
25
shows the amount of hours spent on average per each unplanned rework.
The hours per
rework are normalized for confidentiality reasons.
Of all unplanned rework 21% were caused by errors at the data input stage. This means
that engineering work was commenced without the availability of all required information.
Further analysis found out that in 66.7% of incidents in this particular step, Team TA did
not request the data required. This was caused by multiple factors: 1) Team TA used own
knowledge and experience to fill in the gaps.
2) Specific data inputs were not identified
as required. 3) Procedural and human errors. In the remaining 21% unplanned rework, in
16.7% incidents the data was requested but was not available, and in 16.67% data provided
to Team TA was invalid. Average hours per rework for unplanned rework caused at the
data input stage is highest at 8.4 hrs/rework. This is anticipated as an error here means
engineering rework has to start from the very first step of identifying required data input.
Engineering and modeling errors accounted for 25% of unplanned rework. In this case, the
actual modeling and engineering analysis didn't meet the customer's requirements. Average
hours per rework caused at this step is 2.7 hours per rework.
The majority of unplanned rework was caused at the Documenting step. Errors in this
stage meant that the analysis and engineering work itself was sufficient; however, the documents, templates, and actual communication of the analysis was not sufficient. Unplanned
rework caused at this step has the lowest number of hours per rework.
The last step that may cause an unplanned rework is the customer review step.
In
these incidents, Team TA delivered exactly what the customer requested. However, upon
reviewing the deliverables, the customer realizes that they themselves didn't request fully the
analysis they needed. Therefore, they request a rework of the same deliverable explaining
in details the analysis they are hoping for. Unplanned rework due to customer review could
require varying amount of hours depending on the change in scope the customer is requesting.
Therefore, average hours per rework caused in this step is 4.9 hrs/rework which is second
largest relative to all other steps.
26
3.3
Conclusions and Recommendations
This case study highlights the importance of integration through clear understanding of the
inputs and the outputs of the each given task. Team TA's unplanned rework was 85% within
their own control.
Furthermore, 60% of unplanned rework was attributed to procedural
errors (data input, and documentation) rather than technical errors. Therefore, the following
procedural recommendations will have a significant impact on improving the quality of the
engineering teams.
1. Measure team's first pass yield at an aggregate and step level.
This study
contradicted the team's understanding of resources of unplanned rework. Team TA
had thought a majority of unplanned rework was due to Customer's review. However,
that accounts to only 15% of unplanned rework.
This study, therefore, serves as a
baseline measurement of the performance. This baseline could be used to set targets
for the performance of the team. In addition, these metrics and measurements must
be included in the performance evaluation of the team's management.
2. Develop an engineering full kitting tool. As defined earlier, full kitting in this
context refers to having all required information and data input to a given design or
engineering stage. The unplanned rework caused at the data input stage constituted
21% of all unplanned rework.
Furthermore, it had the highest average hours per
rework with 66.7% easily avoidable if Team TA requested the required data. Within
the context of larger teams and more complicated processes, rework due to errors at
the data input stage becomes more determinant to the quality of rework. Focusing
on the data input stage would also improve the communication between teams. This
study is a significant first step to implement full kitting. The next two steps:
(a) Develop an internal tool/template identifying the input data required under several circumstances.
(b) Implement required fields requesting data needed by Team TA in the project
management system used.
27
3. Perform analysis and recommendations 1 and 2 above to other Engineering
teams.
Team TA is a small team. The exercise of performance analysis, including
process mapping, allowed the team to better understand the value stream of their
activities and areas of improvement.
28
Chapter 4
Applying System Dynamics to
Airplane Development Process
This chapter discusses the effects of propagating rework across several airplane development
stages on the overall time-line and schedule of the development. The first section will discuss
the academic literature regarding the use of system dynamics in product development. The
second section will detail the scope and approach of this project. The results will be discussed
before a final results section concludes this chapter.
4.1
System Dynamics in Product Development
The study of System Dynamics use in project management and product development had
come to life by the work of Pugh-Roberts Associates and KG Cooper work in 1980.[7]
Cooper's work was the first to formulate the rework loop. Abdel Hamid analyzed the dynamics of software development in project management in his PhD thesis. [1] He built on the
formulation used by KG Cooper and introduced quality assurance in product development
context.Ford and Sterman have built on the rich history of product development. [12] The
paper "Dynamic Modeling for Product Development Processes" by Ford and Sterman is
the core of modeling performed in this thesis. It details the impact of concurrences between
multiple project phases as well as the use of quality assurance and rework cycles. Pena-Mora
and Park have included the effects of rework due to internal errors as well as rework due to
29
external factors in their 2003 paper. [20] Modeling approaches make the assumption that
it is only possible that one rework is required per task in a given cycle. Rahmandad and
Hu's 2010 work considers multiple rework per given task in a single loop.[21] Lyneis provides an in-depth study of the literature relating to the use of System Dynamics in product
development broken down by unique model structures.[19]
The following two subsections provides examples of successful use of system dynamics in
project management and product development. Then, it highlights the modeling foundation
upon which this thesis work is built.
4.1.1
Applications of System Dynamics in Product Development
This section will highlight two examples of applying system dynamics to address organizational challenges. The two examples present two very different approaches to the application
of System Dynamics.
Olivier de Weck develops a tailored model to study application of
META design to speed systems development.[8] The purpose of the study is to validate an
hypothesis. In the second example, KG Cooper utilizes the rework loop formulation to improve change management at some of the most complex projects at Flour Corporation. [14]
In this example, the model is used repeatedly for many projects.
The 2012 conference paper by Oli de Week studies the feasibility of speed-up in systems
development through the use of META design. META design is an mechanism that includes
three approaches: the use of layers of abstractions, the use of extensive and trusted component model library, and use of virtual Verification and Validation. However, the focus here is
the use of system dynamics. The model built shown in figure 4-1 represents the model used
in this project. This model uses simple separate stock and flow structures that represent
the multiple stages of a given development project.
These stock and flow structures are
connected through causal loops that represent the various factors at play in the development
process.
At such macro level of representations, this modeling technique uses aggregate
mathematical formulations.
In this case, the model was validated against data from the
Boeing 777 Electric Power System design project at UTC. Such validation is critical due to
the significant use of aggregate formulations to represent. The model represents a rework and
change loop that is integral to the model. In this particular case, the model has successfully
30
Ri~ Sa~Ri~
F~
1
CaRA
lVc
CYv
*
*~
H1 IA
T.-0
Ubiraput
/
0
'~
Poxty~l)A
0
Q
0wch
aa
Rutmd
Figure
~
Ia A %
~
t' 4-:)eScks ysemdyamc odl.8
served
its biCo.TesseOyaisweemdlda
purpse to validate he feasibilityosee-p
onaint
illonand$1
$0
rap he
eneisof
sinfn
th
s
trainin
of
thes=
tween $10illionuan $10 billion.The
thoerayi and
laa
Figure~~~~~~o 4-:deek' ysewynmcodl[8
t projets whch
each poject Fthrore
uly
~~ ~
La do Pa
tu
mangeentprctsacrsstedvreprflo
twen
Vldldmd
ftaarmod
Ctu-mt
0
Lvwm=
luft 0
Avetap
LAW
ca
z
rag
e
intessesdvlpetpoes
off
the prj. aaeswa
model.w
system dynamicwr modeled
asa[ondtont
tlen aSproess anatoos weredeised to quicklyr cteoiecth
mde
forng
each project. Furthermore, significant training of the project managers was undertaken to
fully reap the benefits of the use of the model.
31
netk
t
The rework cycle of projects
Staff on
project
Productivity
Progress
Work to be
done
Known
rework
Work really
ACAdone
ZZ
Undiscovered
4,1rework
Rework
discovery
Figure 4-2: The foundation of Cooper's models, the rework loop. [14]
4.1.2
System Dynamic Model Structure for Product Development
The rework loop is critical to understand issues of quality in airplane development. This
section will discuss the modeling and formulation fundamentals based on Ford and Sterman
work. [12] The case study discussed in this chapter is heavily based on these fundamentals.
Any product development project includes multiple phases that are connected through
deliverables and rework errors.
Each phase produces deliverables that form the input to
later phases. Furthermore, each phase may detect errors that require rework and send back
to earlier phases. This dynamic is presented in figure 4-4.[12]
Each single phase is based on the stock and flow structure presented in Figure 4-5. This
structure assumes that tasks are moved through several steps until approval.
The tasks
begin in a stock of uncompleted tasks. These tasks are then completed and awaiting to be
checked. Upon quality assurance of these tasks, these tasks either move towards approval or
move towards a stock for rework.[12]
There are two quality assurance variables of critical importance at this stage: probability
of rework existing, and probability of discovering such rework. The probability of rework
existing refers to the chance that an error that requires rework may be made at the completion
stage. However, not all errors are discovered all the time. The probability of discovering
rework indicate how many of the errors made are caught in quality assurance.[12]
Concurrence is critical to each development stage.
32
Internal concurrence refers to the
~.
-H
/
Worker
, *
\
staff
Vendor
design
information
Siaffing
.o
. requested
Hours expended
to date
Pr
W
time
-Sta-on
pr
Productivity
o
s
korbe
Work ialy
done
edone
completion
4
Turmover
-
*. Overtime
.
Quality
Scheduled / e X
completion
/
Out-of-Sequence
time
work
Expected
_
.-
Schedule
pressure
-0-
rn
-
Worker
experience
morale
/
-
-
-
-
-
-
-
-
A
Expected
hours at
completion
d
Known
UndiScovered
rework
_Engineering
Rework
revisions
CISCOveyA.
rework
o 0
Perceived
-
progress
Figure 4-3: The causal loops presented by Cooper which tells the story of project management at Flour. [14]
Product
e
n.
PrototypePof
Testing
Derfinition
Reliability/
Design
Quay
OoProdcts
of Development Phase
..
-
LEGEND
Return Errors for Correction
[J3
Development Phase
Figure 4-4: Multiple phases representation of any product development project as shown by
Ford's 1997 work. [12]
33
TtCh (3)
Tasks to be Changed
Changes
rate
(8)
IDiscover
IDIscCh
ApprT (9)
C h a n g e T a s k rote
ChT (6)
Tasks
A ppr.v.
asks Complete
TCnotCk (2)
I Checked
T
RelT (11)
es :
ra
Tasks Approved
TAppr (4)
s
rooks Released
TRel (5)
TnC (1)
not Completed
Completion rate
CT (13)
Figure 4-5: The basic stock and flow structure of a single product development phase as
shown in Ford's 1997. [12]
amount of tasks available to complete at a given time within the same phase. For example,
at the requirements development phase the high level product definition requirements must
be completed before detailed performance requirements can be completed.
In this model,
concurrence is defined by a percentage of work available to complete at a given time.[12]
Given that this structure represents a single phase, it is connected with other phases
through three mechanisms: external concurrence, rework discovered downstream, and rework discovered of upstream phases. External concurrence define how much work is available
for completion in phase B based on the percentage of completion of tasks in phase A where
phase A precedes phase B in this example. If a rework relating to this phase is discovered
in a downstream phase, it is returned to this stock to rework. Furthermore, there is the
possibility that a rework of upstream phases is discovered in this phase at the quality assurance step. Both downstream and upstream reworks are moved into a stock of tasks to be
coordinated. [12]
34
probability Discover Change
p(DiscCh)
probability Change Req'ired
p(t~h
Discover Cans
Avg Quality Assurance Duration
Fraction
AA
DiacChFrac (10
Tasks
to be Changed
Quality Assurance Resource constraint
Discove
hanges ret
RCOA
Qualtty Assurance
QA (7)
Nominal Release Time
NomReMme
Phas
Avg Cheng Duration
AMh
Re PN
Change Resource onstraint
RQ-'hT
-
Change Test rate
AP
easks Completel not Ch
Roekm
(12)
CkSl e
Raises
ova ask rats
Teas
pproved
Scope
Task
rate
""""'
asks Released
completion Resource constraint
AC
Avg Completion
DuretIOn
Tasks Perceived
Satisfactory
Tasks not Completed
Completion ra
Fraction Percerive Sotisfactory
FPS (17)
e
Phase Scope
PS
Tasks Available for Completion
CTAval (14)
ternal Process Concurrence
%Avail Internal Concurr (16)
Total Tasks Avaialable
TTA(1 5)
Figure 4-6: Quality factors affecting the basic single product development phase in Ford's
1997 work. This graph also shows the internal concurrences. [12]
35
Avg Coordination Duration
ACoordD
Coordinat Ion resourse constraint
RE)Cord
TtCoord (21)
CDQA (23)
Tasks to Coordinate
Coord (2/4)
Coordination due to Downstream Ouaiiy Assurance
Coordinate asks
probablit tof Discovering Change
p(DIscCh)
probability of InterPhase Change
p(InterPhaseCh)
probablility o ntraPhese Change
p(intraPhaseCh)
Tasks
hanged
Discover InterPhase Changes
DisclnterPhaseCh (22)
Discover IntrPhas hanges
DisclntrafhaseCh (8
Qualit
pro
Change Task
ApprT (9a)
ask rat
Release Task rate
-
Assu ance
rate
asks Complete ot Che
TCnotCk (2a)
Teasks
Tasks Perceived
pproved
Tasks Released
TROI (5a)
Satisfactory
Tasks not Completed
Fraction Percerive
Completion rate
Phase
Tasks Available
Scope
for Completion
nternal
Min
etisfactorg
Process Concurrence
Total Teasks AvoalaSble
TTA (15a)
External Process Concurrence
% Avag Exiernal Concurr (19)
Figure 4-7: The addition of upstream and downstream quality effects. This figure shows
how a single product development phase connects with other phases through stocks of tasks
to be coordinated. [12]
36
4.2
Modelling Approach and Scope
Section 4.1.2 is a brief discussion of the modeling foundation upon which the detailed modeling was based in the following case study. Due to confidentiality and copyright reasons, the
details of the model which had to be tailored to the specifics of the case are not presented
in this thesis.
4.2.1
Approach
Three clear steps were followed in the study of this case. The first step was to define the
scope and stakeholders.
New airplane development is an enormous scope.
A very high
level model encompassing all stages and functions will lose much of its credibility. On the
other hand, choosing an extremely detailed stage of the development process will prevent the
analysis of rework effect on multiple stages. The second step was group modeling. During
the group modeling stage, stakeholders were involved in modeling exercises and involved in
providing direct feedback on the details of the model. Last is an analysis and check of results
applicability. It is important to note that the lack of available comparable data did not allow
to calibrate the model against current process behavior. However, the model's behavior was
acknowledged by stakeholders to be qualitatively similar to the experience.
4.2.2
Scope
This exercise focuses on the design and development of airplane components in collaboration
with suppliers.
The four main stages involved in component development are highlighted
in figure 4-8. An assumption was made that detailed rework due to high-level requirements
is completely caught at the component level requirements. Though this assumption is very
significant and not in-line with actual incidents, it was agreed upon for purposes of simplification of the model. All other rework incidents of other stages are ultimately discovered at
the compliance stage.
37
M~ng41or~TicTh rr~lonM9
Requirements
Change
Component level
requirements
Contract
Design
Change
4
CDepingn stage inComponent
Build
Desabaign t
collabr
suppliers
Airplane Level
Requirements
Development
h
W-----
....
Performance
Compliance and
Testing
Concurrence
Rework
Figure 4-8: The stages represented in the model built for this case
4.2.3
The Model
Figure 4-8 represents a macro level representation of the structure of the system dynamic
model built for this case study. Three of the four stages were represented by a single structure
similar to the one presented in figure 4-7. One of the stages, component level requirements,
was represented by multiple sub-stages.
Average duration of a given task, Figure 4-6, was an input to the model and not variable
through the simulation. However, in the component level requirements stage, the functionality of simulating schedule pressure was added to this stage. The average duration inputs were
collectively calibrated to make the whole process run at a similar duration to the time-line
expected from experience.
Similar the work of Ford [12], parallel structures were built into the model for rework.
These parallel structures simulate the time it takes to complete a rework item and release it
back into the process.
38
4.3
Results
The analysis of the model looks into two characteristics that determine the quality at each
stage in addition to the effects of schedule pressure on the quality:
1. The probability of creating rework (Prob.Cr): this probability determines the percentage of all work done at a given stage that will ultimately require rework.
2. The probability of discovering rework (Prob.Di):
this characteristic determines the
percentage of all rework discovered at a given stage. This includes rework carried on
from previous stages. In discussing the results, 'quality' refers to both Prob.Cr and
Prob.Di together. Discussing a quality of a stage means the changes in both of the
metrics. Any change to a single metric will be identified individually.
3. Sch.P effects are only studied for the requirement development stage. Schedule pressure (Sch.P) refers to the pressures to complete tasks in a given time. It is always
calculated as the ratio between amount of time remaining and the amount of time
required to complete a given task. Schedule pressure in this model can only be applied
to the requirement development stage. First, it affects the quality of the requirement
development stage; an increase in schedule pressure increases Prob.Cr. Second, Sch.P
affects the time it takes to complete a task; if Sch.P increases the the time spent on a
given task decreases. Third, Sch.P affects the time available in a given week; if Sch.P
increases the amount of overtime increases.
The model was run to understand the changes of Prob.Cr and Prob.Di over three scenarios; changes in the requirement development stage only, changes in the design stage only,
changes in both stages at the same time.
4.3.1
Quality of the Requirement Development Stage
This scenario studies the effects of quality metrics (Prob. Di and Prob. Cr) in the requirements development stage on the overall time to completion as well as amount of work done
till completion.
Furthermore, the effects of schedule pressure in this stage are studied as
39
Time to Complete due to Quality at Requirements Stage
571%
600%
500%
Time to
completion
400%
334%
294%
relative to base 300%
case
200%
100%
115%
104%
127%
105%
10%
50%
90%
Probability of Discovering Rework
10% Prob of Rework
-4-30% Prob of Rework
-A-
50% Prob of Rework
Figure 4-9: The results of varying quality factors in components requirement stage while
quality is constant in other stages. This figure shows the effects on time of completion. The
time of completion is normalized against an ideal quality case of zero errors.
well. Both Prob.Di and Prob.Cr are changed while quality in the design stage is constant.
The results are shown in figures 4-9 through 4-12.
Figure 4-9 looks at the effect of quality on time to completion of the process. The results
are in comparison to the planned situation (ideal situation with zero probability of creating
rework). At 10% Prob.Cr, the effect of Prob.Di is limited to a approximately 30% increase
in time to completion. However, at 50% Prob.Cr only a 10% decrease in the Prob.Di will
add 68% to the time to completion.
Figure 4-10 shows a similar analysis with a focus on the amount of work done to completion relative to base case. This case shows that the percentage increase in amount of work
is more than that of time to completion.
Figures 4-11 and 4-12 repeat the simulations performed in figures 4-9 and 4-10 while
enabling the Sch.P effects. In both cases, the effects of schedule pressure increase the amount
of work and time to completion. This indicates that the negative effects of schedule pressure
are more than the positive effects. It is noticeable that Sch.P affects the amount of work
much more than the time to completion. This indicates that the schedule pressure leads to
40
Amount of Work due to Quality at Requirements Stage
948%
1000%
800%
Amount of work
494%
600%
relative to base 400%
case
200%
447%
273%
244%
148%
1 2SO
0%
--
10%
50%
90%
165%
Probability of Discovering Rework
-- 1.0% Prob of Rewo rk
-*-30% Prob of Rework
---
50% Prob of Rework
Figure 4-10: The results of varying quality factors in components requirement stage while
quality is constant in other stages. This figure shows the effects on amount of work needed.
The amount of work is normalized against an ideal quality case of zero errors.
an increase amount of work over lengthy period of time. However, the model structure does
not account for burn-out and turn-over of workforce. The addition of those two features to
the model may change this observation.
4.3.2
Quality of the Design Stage
This scenario studies the effects of quality metrics (Prob. Di and Prob. Cr) in the design
stage on the overall time to completion as well as the amount of work done till completion.
Both Prob. Di and Prob.
Cr are changed while quality in the requirement development
stage is constant. The results are shown in figures 4-13 and 4-14.
The system's response to the change in quality metrics in this stage is similar to the
response of the system to quality metrics in the requirement stage. However, changes in
quality metrics in this stage has a more limited impact than a previous stage.
This is
expected as rework created in this stage travels to the next stage where it is discovered.
Rework created in the requirement stage travels two stages before completely discovered.
41
-
Time to Complete due to Quality at Requirements Stage
Schedule Pressure
700%
578%
600%
500%
Time to
completion 400%
relative to base
300%
361%
301%
188%
208%
109%
104%
105%
127%
90%
50%
10%
case
200%
100%
V70
Probability of Discovering Rework
-10%
Prob of Rework
--
30% Prob of Rework
-r-
50% Prob of Rework
Figure 4-11: The results of considering schedule pressures while varying quality factors in
components requirement stage with quality is constant in other stages. This figure shows
the effects on time of completion. The time of completion is normalized against an ideal
quality case of zero errors.
4.3.3
Quality of Both Stages
In this scenario, both Prob.
Cr are changed simultaneously for both the
Di and Prob.
requirement development and design stages.
The results are shown in figure 4-15.
The
results show the compounded effect of poor quality in both stages. It increases the time to
completion significantly.
4.4
4.4.1
Conclusions and Recommendations
Conclusion
The results of the model scenarios show that quality in an early development stage, the
requirement development in this case, affects the completion time significantly more than
quality in other stages. Both quality factors, Prob.Cr and Prob.Di, have a significant impact
on the completion time. Furthermore, schedule pressure is only beneficial given small amount
42
-
Amount of Work due to Quality at Requirements Stage
Schedule Pressure
1400%
1153%
1200%
1000%
675%
Amount of work 800%
relative to base 600%
case
400%
200%
553%
377%
36%
1775%
192%
146%
179%
0%
10%
50%
90%
Probability of Discovering Rework
-10%
Prob of Rework
-M-30%
Prob of Rework
-*-50% Prob of Rework
Figure 4-12: The results of considering schedule pressures while varying quality factors in
components requirement stage with quality is constant in other stages. This figure shows
the effects on amount of work needed. The amount of work is normalized against an ideal
quality case of zero errors.
Time to Completion - Quality of Design Stage
329%
350%
300%
250%
Time to
completion 200%
relative to base
150%
case
100%
232%
204%
156%
163%
145%
121%
137%
137%
90%
50%
10%
50%
0%
Probability of Discovering Rework
-
10% Prob of Rework
-- 30% Prob of Rework
-Lr50% Prob of Rework
Figure 4-13: The results of varying quality factors the design stage while quality is constant
in other stages. This figure shows the effects on time of completion. The time of completion
is normalized against an ideal quality case of zero errors.
43
Amount of Work - Quality of Design Stage
273%
300%
250%
206%
198%
200%
Amount of work 150%
relative to base
100%
case
148%
164%
110%
113%
118%
90%
70%
50%
143%
50%
0%
Probability of Discovering Rework
10% Prob of Rework
30% Prob of Rework
--
-*-50% Prob of Rework
Figure 4-14: The results of varying quality factors in the design stage while quality is constant
in other stages. This figure shows the effects on amount of work needed. The amount of
work is normalized against an ideal quality case of zero errors.
Time to Complete Quality of Both Stages
362%
400%
350%
Time to
completion
relative to base
case
300%
250%
200%
150%
100%
50%
0%
277%
196%
199%
171%
158%
130%
137%
142%
-
4
~~
-
50%
70%
90%
Probability of Discovering Rework
-10%
Prob of Rework
-4-30% Prob of Rework
*50%
Prob of Rework
Figure 4-15: The results of varying quality factors in both the components requirement
stage and the design stage. This figure shows the effects on time of completion. The time of
completion is normalized against an ideal quality case of zero errors.
44
of rework only.
In the first scenario, Quality changes within the Requirements Development stage shows
a non-linear growth of the time required to complete project as both Prob.Di decreases
and Prob.Cr increases. At low Prob.Cr of 10%, the time to complete the full phase only
increases by 27% by the decrease of Prob.Di. However, at 30% Prob.Cr, time to completion
increases by 194% as Prob.Di decreases to 10%. Similar results appear when considering the
number of requirements work. There is a non-linear behavior with much larger percentages
of increase in-comparison to time.
A factor of 2 differentiate the impact of Prob.Cr and
Prob.Di. Changes in Prob.Cr impacts the overall time of completion by approximately twice
as much as a similar change of Prob.Di.
When schedule pressure effects are taken into account, there is an increase in completion
time only at poor quality levels. This is due to the fact that increase in errors overwhelm
the schedule pressure impact on time to perform each task. Ultimately, the large increase of
rework leads to a longer completion time.
In the last scenario, when quality was changed across both stages it was expected that
time to completion would increase significantly more than previous stages.
However, the
increase was comparable to that of changing quality in the Requirement Development stage.
This supports our findings of the importance of the requirements quality stage.
4.4.2
Recommendations
This model assumes a very small scope and an ideal scenario where change management
is perfected. Therefore it is expected that the impact of propagating poor quality to later
stages is much larger than shown in this limited model. However, in this case three factors
are deemed critically important: 1) The probability of creating rework. 2) The probability
of discovering rework.
3) The impact of schedule pressure. Boeing has significant effort,
many of which are very successful, to improve the first factor. Nonetheless, this model shows
the significant impact on later stages even at low Prob.Di.
Furthermore, improving the
probability of discovering rework is closely related to anticipating rework within the project
planning by increasing the margins for possible rework.
Another approach to addressing
Prob.Di is to integrate clear and strong stages of development iterations at each stage.
45
Schedule pressure adds very little value to completion of development stages. Completion of
any particular stage must be completely reliant on a thorough review of the deliverables.
46
Chapter 5
Conclusions
This study assessed the hypothesis that integration and process discipline are determinantal
to the quality of airplane development at Boeing. This thesis focused on the rework necessary
and completion time of any engineering development stage or activity. Two case studies completed in collaboration with Boeing gave support to the hypothesis. The recommendations
and opportunities for next steps are the following:
1. Process discipline is engineering quality: large engineering projects such as new
airplane developments require a significant magnitude of integration between multiple
teams. Clear definitions of inputs and outputs of each team, stage, and activity are
a first step to improve the integration of engineering teams.
Each stage must have
a clear inputs required as well as clear outputs planned. Therefore, the only criteria
of entering a new stage are the completion and acceptance of a given stage's inputs.
Moreover, beginning a new stage without the completion of its input can be possible
if iterations are already planned out.
2. Implement full kitting: a full kit is a term used in lean operations principles that
refers to having all the tools and materials required to complete a given activity before
starting that process. Applying this to engineering teams within the airplane development process creates an inherent quality assurance process. However, each team
must first go through the process of quantifying their performance. Only if quality is
measured it can improved. Through the analysis activities of their own performance,
47
the team must develop a clear understanding of information and data inputs required
of their activities. This leads to transparency and accountability in the overall process.
3. Account for iterations in the development processes: discovering rework at the
stage within which the rework was caused is critical to prevent significant increase in
rework required. Planning the development process such that it includes iterations of
development stages is critical. Furthermore, there is a great opportunity to integrate
robust design principles adapted in other industries such as the software industry.
48
Appendix A
Interview Questions and Sample
Answers
This section presents a sample of the questions being asked in the initial interviews that led
to the hypothesis as well as a sample of the answers. The names and answers have been
reviewed to redact any confidential data and information. The answers are based on notes
taken in the meetings and not are exactly verbatim.
A.1
Interview Guidelines
These interviews were conducted in the mission of fact finding and sense-making. Therefore,
open ended and high-level questions were presented to the interviewees.
Most interviews
were conducted by the author of the thesis with a Quality Engineer who supported the
efforts of this project.
Introduction: In the introduction of the interview we highlight that we are trying to
study engineering rework in new product development. Then, we ensure to clearly identify
the area of expertise and scope of interviewees. Interviewees were always responsible for a
specific system, component, team, etc regardless of their position. The following questions
are representative and not exclusive of all areas explored with each individual interviewee.
Questions:
1. Can you walk us through the process [of your scope]?
49
2. What are the triggers and inputs to your [of your scope]?
3. How do you measure quality of input information?
4. How do you measure completion of your [scope] tasks?
5. Where would rework happen?
6. What is the approval and/or quality assurance process within your [scope]?
7. Do you know to whom your process output is going?
8. How do you manage schedule pressure?
Samples of Interviews
A.2
These interviewees are leaders, senior engineers, and managers within various groups of the
airplane development organization.
Interviewee 1 The following are excerpts from the interview and not exact.
"
two groups didn't coordinate major portions of requirements ...
design of system was not completely mature before moving to the next phase
..
there were different base-line designs from which different groups started their
"
design ...
extremely complicated and over-engineered designs were produced
...
"
S" . .
Interviewee 2 The following are excerpts from the interview and not exact.
after missing a major milestone we started a review to find a [surprising number
of design changes]
..
team used extremely difficult change management process that slowed down nec"
essary change ...
50
...
"
team(s) were releasing their deliverables without validations
no, in this case suppliers were not to blame ...
"
9 "...
Interviewee 3 The following are excerpts from the interview and not exact.
"...
document (ABC] were critical for us to start our work. It always got to our team
too late, and never mature enough. No change stages were considered in the schedule.
...
"
We were supposed to get it late, and still produce correct engineering work in-time
*
"...
even when we missed a deadline, our schedule won't change
*
"...
we can't request our team data input early, because other teams want to delay it
even further
..
fa
51
"
...
Bibliography
[1] Tarek K. Abdel-Hamid. The dynamics of software development project management: an
integrative system dynamics perspective. PhD thesis, MIT Sloan School of Management,
1984.
[2] airbus.com. Press release: airbus launches the a330neo.
[3] Bloomberg.com. Boeing loses $1.76 billion 747 order after cutting output.
[4] Barry W. Boehm. Software engineering economics. Prentice-Hall advances in computing
science and technology series. Englewood Cliffs, N.J. : Prentice-Hall, c1981., 1981.
[5] Boeing.com. Boeing 777 facts.
[6] Boeing.com. Boeing launches 747-8i.
[7] Kenneth G. Cooper. Naval ship production: A claim settled and a framework built.
Interfaces, 10(6):20 - 36, 1980.
[8] Olivier L. de Weck. Feasibility of a 5x speedup in system development due to meta
design. In ASME 2012 InternationalDesign Engineering Technical Conferences, number
DETC2012-70791, Chicago, Illinois, August 12-15 2012. Accepted.
[9] Olivier L. De Weck, Daniel Roos, and Christopher L. Magee. Engineering systems
meeting human needs in a complex technological world. Cambridge, Mass. : MIT Press,
c2011., 2011.
[10] Claudia Eckert and P. Clarkson. Planning development processes for complex products.
Research in Engineering Design, 21(3):153 - 171, 2010.
[11] Forbes.com. What went wrong at boeing?
[12] David N. Ford and John Sterman. Dynamic modeling of product development processes.
1997.
[13] Ernst Fricke, Bernd Gebhard, Herbert Negele, and Eduard Igenbergs. Coping with
changes: Causes, findings, and strategies. Systems Engineering, 3(4):169, 2000.
[14] E Godlewski, G Lee, and K Cooper.
System dynamics transforms fluor project and
change management. INTERFACES, 42(1):17 - 32, n.d.
52
[15] T. Jarratt, C. Eckert, N. Caldwell, and P. Clarkson. Engineering change: an overview
and perspective on the literature. Research in Engineering Design, 22(2):103 - 124,
2011.
[16] Kim Jongbae and D. Wilemon. Complexity and the multiple impacts on new product development: results from a field study. International Journal of Innovation and
Technology Management, 9(6):1250043, 2012.
[17] Wall Street Journal. Under pressure, airbus redesigns a troubled plane.
[18] Jin Kato. Development of a process for continuous creation of lean value in product
development organizations. c2005., 2005.
[19] James M. Lyneis and David N. Ford. System dynamics applied to project management:
a survey, assessment, and directions for future research. System Dynamics Review
(Wiley), 23(2/3):157 - 189, 2007.
[20] Moonseo Park and Feniosky Pefia-Mora. Dynamic change management for construction:
introducing the change cycle into model-based project management. System Dynamics
Review, 19(3):213-242, 2003.
[21] Hazhir Rahmandad and Kun Hu. Modeling the rework cycle: capturing multiple defects
per task. System Dynamics Review (Wiley), 26(4):291 - 315, 2010.
[22] Nelson P. Repenning. Understanding fire fighting in new product development. Journal
of Product Innovation Management, 18(5):285 - 300, 2001.
[23] Reuters.com. Timeline: Boeing's 787 dreamliner woes.
[24] seattletimes.com. Boeing celebrates 787 delivery as program's costs top $32 billion.
[25] S. Spear. The High Velocity Edge. McGraw-Hill, 2009.
[26] Li Weilin and Moon Young. Modeling and managing engineering changes in a complex product development process. InternationalJournal of Advanced Manufacturing
Technology, 63(9):863 - 874, 2012.
53