Measuring KCs using Scorecards Practical Application
Ian James
Chief Engineer
Electronics & Software Technology
Aero Engine Controls
February 2009
A Rolls-Royce and Goodrich Corporation joint venture
Measuring KCs Using Scorecards
Contents of Lecture
ƒ Introduction
ƒ Customer Expectations
ƒ What is a Scorecard ?
ƒ How to Compile A DFSS Scorecard
ƒ A Little More About Goodrich Engine Controls
ƒ EEC Scorecard Example
ƒ Summary
1
Customer Expectations
ƒ The Customer Expects …
ƒ Supplier will design and manufacture a product that ….
ƒ Meets all requirements defined by design specifications
ƒ Demonstrates Six Sigma Levels of quality
ƒ The following list summarizes the bounds of customer
expectation for 6σ levels of quality
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
Safety
Expected Operation (performance)
Life / Reliability
Maintainability (Cost of Ownership)
Cost / Weight
Producible / Cost Effective (Rework)
Customer Expectations
ƒ Key Characteristics [Customer View]
ƒ Key Characteristics (KCs) may be provided by Customer
ƒ Critical To Quality (CTQs) characteristics
ƒ Critical Characteristics
ƒ Major Characteristics
ƒ Characteristics can be identified via …
ƒ Quality Function Deployment (QFD)
ƒ Failure Mode and Effects Analysis (FMEA)
ƒ Historical Quality Data
2
Customer Expectations
ƒ Key Characteristics [Supplier View]
ƒ Further Key Characteristics may be defined ‘internally’
ƒ Critical To Cost (CTC) characteristics
ƒ Key Control Characteristics (KCC)
ƒ Additional QFD and FME(C)A may be conducted jointly
by both parties to agree a definitive list of KCs prior to
project implementation
implementation.
Customer Expectations
ƒ Capability Metrics
ƒ Customer expects all Key Characteristics 6σ by E.I.S.
ƒ (Cp=2.0, Cpk=1.5, DPMO=3.4, Zst = 6.0)
ƒ Supplier predicts capability level for each KC at E.I.S.
ƒ Prediction based upon similar equipment and processes
ƒ Combination of Data Analysis, Simulation and Modelling
ƒ If predictions do not indicate 6σ levels by E.I.S. then a
jointly developed plan will be created to drive the levels
to 6σ.
3
Customer Expectations
ƒ Monitoring Capability for Key Characteristics
ƒ The DFSS Scorecard is a tool that can be used to assess
product & process performance to establish if customer
requirements are met at all stages of the design process.
ƒ The Scorecard can be used to
ƒ Locate areas for improvement
ƒ Recognise key missing issues & problem areas
ƒ Improve & optimise the design and manufacturing processes
ƒ The Scorecard is a live document that should be updated
and reviewed at each phase of the project.
Customer Expectations
ƒ Monitoring Capability for Key Characteristics
ƒ A DFSS Scorecard may be enforced by a customer to
facilitate the monitoring of predicted capability
ƒ The scorecard is a live document and will be updated
as a minimum for presentation at:
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
Proposal Submittal
Preliminary Function Review (PFR)
Preliminary Design Review (PDR)
Detailed Design Review (DDR)
Pre-Production Manufacturing and Quality Review (PM&QR)
Production Readiness Review (PRR)
4
What Is A Scorecard ?
ƒ The Scorecard lists all the Key Characteristics of a design
For each Key Characteristic …
Identify the
‘Specification Limits’ or
‘Targets’
Identify the Process
Behaviour
i.e. Mean, Variation, Defects
As a result…..
knowledge can be
gained about:
Comparison to determine whether
CUSTOMER EXPECTATIONS
ƒ Robustness of
product (at all levels)
are being met
ƒ Manufacturing
process performance
What Is A Scorecard ?
ƒ The DFSS Scorecard Consists of Two Parts
ƒ Product & Sub-assembly / Component level
level.
DFSS Scorecard
The DFSS Scorecard consists of
two parts:
Product Scorecard
Subassembly/ Component Scorecard
Assesses the overall robustness of the
design and its ability to meet customer
expectations – estimate the defects
experienced by customer
Assesses the quality of all components and
sub-assemblies (& subsequently the
processes) used in the design – estimates
internal/supplier defect rate
(Customer Perspective)
(Internal Perspective)
5
What Is A Scorecard ?
ƒ How To Compile a DFSS Scorecard
Determine the
Key
Characteristics
1
• Information generated
at all stages of this
process are recorded
on the DFSS Scorecard
• The same process is
used to compile both
parts of the scorecard
Perform
Measurements
Systems
Analysis on the
Key
Characteristics
For each Key Characteristic, identify
the target and specification limits
2
3
Collect ‘performance
d t ’ for
data’
f each
h Key
K
Characteristic
Analyse data to establish if the
performance of each Key
Characteristic is acceptable
4
5
How To Compile A DFSS Scorecard
Step 1: Determine the Key Characteristics
ƒ For the Product Scorecard
Tips - Ways to identify Key
Characteristics
- Customer Requirements
documents
ƒ Obtain Customer Requirements Document
- Brainstorm by project team
ƒ Use these requirements to define KCs
- Use list of Key
Characteristics on similar
products
ƒ Assign metrics and units
ƒ For the Component Scorecard
Note : The list of Key
Characteristics will grow as
the project progresses &
better understanding is
gained about product
performance
ƒ Identify all the subassemblies, functions and
Components that may impact the Product KCs
ƒ Accordingly, derive the Key Characteristics for
each of these subassemblies and components.
ƒ Assign metrics and units.
6
How To Compile A DFSS Scorecard
Example : Identifying Key Characteristics
The Design
g Team developing
p g a flight
g control actuator for use on the
A380 used the following method to identify the key characteristics
for their product. Since the team had no historic data to draw from,
they decided to hold a brainstorming session.
The Brainstorming
Session was
structured to focus
on the key
characteristics of
the overall product
and those related
to the top level
subassemblies
How To Compile A DFSS Scorecard
Example : Identifying Key Characteristics (Cont’d)
Key Output Characteristics Brainstormed
Key Input Characteristics
Brainstormed
(Customer Requirements Document
used as a starting point.)
At each level, the team
• Brainstormed all the key
output characteristics
• Brainstormed all the key input
characteristics
• Developed an input-output
matrix
t i and
d ranked
k d th
the
correlation between the inputs
and the outputs using 1, 3, 9
or none
• Used the matrix to prioritise
the key input parameters
Numbers used to ensure all key input characteristics
have been identified and prioritise them
7
How To Compile A DFSS Scorecard
Step 2: Perform measurement system analysis
ƒ BEFORE gathering performance data for each KC
ƒ Ensure that the performance of the measurement system
that will be used to collect the data is acceptable
ƒ Define All assumptions
Step 3: Determine Target & Specification Limits
ƒ May be straightforward ….
ƒ From
F
customer
t
requirements
i
t
ƒ Previous similar products
ƒ May require some additional analysis / estimation
ƒ Tolerance apportionment at the System Level
ƒ Justification by reasoned argument
How To Compile A DFSS Scorecard
Example : Define Metric and Specification Limits for Key Characteristic
For the A380 Flight Control Actuator, the mechanical balance of the rotor was
identified as a Key Characteristic during brainstorm activities.
Key Characteristic: Mechanical Balance of Rotor
Gage name:
Date of study :
Reported by :
Tolerance:
Misc:
Gauge R&R Study of Balancing Machine
1. Amount of Unbalance in the NonDrive End of the Rotor (Plane 1)
Ralph
August 20001
Rakhee Kar
4 gmm
Xbar Chart
Sample Mean
4
2. Amount of Unbalance in the Drive
End of the Rotor ((Plane 2))
1
2
3
4
5
6
By Part
7
8
9
4
10
3
3
3.0SL=2.902
X=2.710
-3.0SL=2.517
2
2
1
1
2
3
4
5
6
7
8
9
10
1
2
3
Part
1
2
3
4
5
4
5
6
7
8
9
10
Part
R Chart
Ch
0.8
6
C
Components
off Variation
V i i
7
8
9
10
%Total Var
100
%Study Var
0.7
3.0SL=0.4840
0.4
0.3
0.2
Percent
Sample Range
%Toler
METRIC
0.6
0.5
50
R=0.1880
0.1
-3.0SL=0.00E+00
0.0
0
1
2
3
4
5
6
7
8
9
10
Gage R&R
Part-to-Part
Part
The Amount of Unbalance
should be less than 4 gmm in
each plane
MEASUREMENT SYSTEM:
Balancing Machine
MEASUREMENT SYSTEM
ANALYSIS confirms that
measurement system is
acceptable
SPECIFICATION LIMIT
8
How To Compile A DFSS Scorecard
Step 4: Collect Data for each Key Characteristic
ƒ Data can be obtained from
f om man
many sources.
so ces
ƒ Prototype Test Data, Simulation, Analysis, SPC
ƒ Customer Complaints, Audit Results, Warranty issues
ƒ Historic information for similar products, components and processes
Step 5: Assess the performance of each Key Characteristic
ƒ
Check the data for Normality
ƒ Determine Mean and Standard Deviation
ƒ Compile the Scorecard using pre-designed spreadsheet
How To Compile A DFSS Scorecard
Example : For Each Key Characteristic Collate Data In a Spreadsheet
How variable is the
critical parameter ?
The solution assumes that
data is normally distributed
Sigma Metric
How
centered
is it ?
Expected
Defect
Rate
DFSS SCORE CARD
Customer Requirement
Level
Key
Characteristic
List of Critical to Quality
Requirements for the
product, subassemblies
& components
Unit
Data
Type
Expected
Performance
for the critical
parameter
Target
Performance
USL
LSL
Mean
Information that must
be entered using
current performance
data for the critical
parameter
Standard
Deviation
Z
Score
(ST)
Cp
Cpk
Expected
Long Term
Performance
(PPM)
These are the ‘Performance Indicators’
and show how well the critical
requirements are being met. (Values
automatically generated by the
spreadsheet.)
9
How To Compile A DFSS Scorecard
Example : Using Performance Indicators To Drive Design Improvements
Below is an extract of the DFSS Scorecard for Product X Generator.
DFSS SCORE CARD
Customer Requirement
Level
Target
Our Performance
Key
Characteristic
Unit
Data
Type
USL
Product
Vibration at
15150 RPM
(Load)
mm
Variable
-
0.434
Product
Vibration at
171871 RPM
(Off Load)
mm
Variable
-
0.434
LSL
SIX SIGMA Project
raised for Product X
Vibration
Mean
Standard
Deviation
Z
Score
(ST)
Cp
Cpk
Expected
Long Term
Performance
(PPM)
-
0.24
0.05
5.38
1.45
1.29
53.0
-
0.328
0.07
3.014
1.03
0.505
648978.1
Scorecard shows the vibration
performance of the generator is
unacceptable at 171871 RPM (Off Load)
How To Compile A DFSS Scorecard
Example : Using Performance Indicators To Drive Design Improvements
For Those of You who Miss the Maths…..
Vibration At 15,150 RPM
Vibration At 171,871 RPM
LSL = 0, USL = 0.434
LSL = 0, USL = 0.434
Mean, X = 0.328, Standard Deviation, σ = 0.07
Mean, X = 0.24, Standard Deviation, σ = 0.05
Cp = USL - LSL = 0.434 – 0
6σ
6 x 0.07
Cp = USL - LSL = 0.434 – 0
6σ
6 x 0.05
Cp = 0.434 = 1.033
0.42
Cp = 0.434 = 1.446
0.35
Cpk = Min
X – LSL , USL – X
3σ
3σ
Cpk = Min
X – LSL , USL – X
3σ
3σ
Cpk = Min
0.328 - 0 , 0.434 – .328
3 x .07
3 x .07
Cpk = Min
0.24 - 0 , 0.434 – .24
3 x .05
3 x .05
Cpk = Min
1.5619 , 0.5047
Cpk = Min
1.6 , 1.2933
Cpk = 0.505
Cpk = 1.293
10
How To Compile A DFSS Scorecard
Example : Using Data To Refine The List Of Key Characteristics
The data collected was used to investigate whether the amount of
rotor run out had a significant influence on the Vibration of the
Generator.
Regression plot for Vibration Level versus Runout
Y = 3.59E-02 + 0.736884X
R-Sq = 36.2 %
Regression Analysis shows
that there is no statistically
significant correlation
between Vibration and
Rotor Run out. Therefore,
Rotor Run out
out’ was
‘Rotor
eliminated from the list of
Key Characteristics.
0.6
0.5
Vibration
n Level
The concentricity of the
rotor/stator was identified as a
key input characteristic for the
Product X Generator
Generator. It was
thought have an influence on
the Vibration of the generator
– a key output characteristic.
As a result it was decided to
collect data for the amount of
rotor run out on all production
units.
0.4
03
0.3
0.2
0.1
Regression
0.0
95% CI
95% PI
-0.1
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
Rotor Runout
A Little More About EECs
ƒ Engine Control System
ƒRolls-Royce Trent 700 Engine
11
A Little More About EECs
Full Authority Digital Electronic Control (FADEC)
A Little More About EECs
VSVA
Main Engine Pump
VSVC
FMU
EEC, PCU
and OPU
12
A Little More About EECs
Computer Systems - Personal Computer [P.C]
A Little More About EECs
Computer Systems - Personal Computer [P.C]
Flexible…..User Friendly…..Multi-Application…..Expandable
Operating (System) Software
MODEM
Application
Software
Monitor
Scanner
Microprocessor
and Support
Circuitry
Printer
Mouse
Keyboard
Operating
Software
Device Drivers
Standard Interfaces
(e.g. USB, PCMCIA)
Manages the interface between the
computer hardware and its peripherals and
is also the visual user interface. e.g.
Windows, UNIX.
Device Drivers are used to configure new
peripherals into the system.
Sound Card
Application software
CD Burner
Harnesses the resources of computer
system to fulfil practical ends. e.g. Word
Processing, Accounting, Games, …..
Limited only by the power of the Computer
system, e.g. Speed, Memory (and the
imagination of the software house !)
13
A Little More About EECs
Computer Systems - Electronic Engine Controller
A Little More About EECs
Computer Systems - Electronic Engine Controller
Embedded…..Application Specific…..Limited Modification Capability
Key Characteristic #1
Speed Probe
KC
Thermistor
KC
Resolver
KC
Switch
KC
Application
Software
Microprocessor
and Support
Circuitry
Operating
Software
Speed Measurement Accuracy
0.01% of point over range X to Y
KC
Solenoid
KC
Torque
Motor
KC
Lamp
KC
Spare
Key Characteristic #2
Solenoid Drive Accuracy
y
+/- 0.5V over a range X to Y
Key Characteristic #3
Torque Motor Drive Accuracy
+/- 1mA over a range X to Y
14
A Little More About EECs
Production Acceptance Testing (P.A.T.)
ƒ Every Product will be subjected to a series of Tests
ƒ
ƒ
ƒ
For an E.E.C. the number of tests may exceed 5000
A computer-based piece of test equipment records all results
Pass / Fail for Production Pass-Off Purposes
ƒ A Range of products are likely contain Generic Circuit Blocks
ƒ e.g. Speed Measurement, Solenoid Drivers, Torque Motor
ƒ Analysis of results from existing product Test can be useful
ƒ Indicate area of design marginality
ƒ Provide performance data for new design KCs
ƒ Determine capability of test equipment
ƒ
EEC Test Results are ideal for measuring system capability
EEC Scorecard Example
Example - EEC Scorecard Analysis Process
ƒ Define KCs of new design
ƒ Evolve hardware requirements from KCs (e.g use of FMEA, QFD)
ƒ Select existing functional circuit blocks which meet requirements
ƒ From PAT schedule determine test step for each function selected
ƒ Review applicability of data for direct comparison with relevant KCs
ƒ Apply and record relevant assumptions to enable comparison
ƒ Extract PAT data for each relevant test step
ƒ Maximise data: Use ALL tests and ALL test equipment
ƒ Carry out Capability Analysis for each Key Characteristic
ƒ Record results in Scorecard
Feedback in the case where analysis shows
KCs cannot be satisfied:
Inappropriate Analysis or
Ineffective Design ?
15
EEC Scorecard Example
Example - EEC Scorecard Analysis Process
ƒ Key characteristic are, in fact, hardware requirements
ƒ Data available for Ambient, Hot and Cold PAT test
ƒ Testing carried out on a selection of (identical) test rigs
ƒ Typically 10 tests per test point
ƒ For each test point, max, min and average recorded in test results
ƒ Max, Min and Average to fall between limits for a pass
EEC Scorecard Example
Example - EEC Scorecard Analysis Process
ƒ Torque Motor Drive Interface
ƒ Existing Design – Product X
ƒ Torque motor drive output is tested at +40mA and -40mA
ƒ Accuracy limits +/- 1mA
ƒ Current is measured using internal current monitor
ƒ Key Characteristic
ƒ Torque Motor Driver Output accuracy - +/- 1mA
ƒ Capability Overview
ƒ Combined accuracy of driver and monitor
ƒ Sample of 332 Data Points
– @ +40mA: Mean = 0.346, StDev(LT) = 0.129 and Cpk = 2.84
– @ -40mA: Mean = 0.385, StDev(LT) = 0.151 and Cpk = 2.3
16
EEC Scorecard Example
Normal
Probability
Plot
Normal
Probability
Plot
@ +40mA
.999
.99
Probability
.95
.80
.50
.20
.05
.01
.001
5.98
5.99
6.00
6.01
6v Excite
Average: 6.00006
StDev: 0.0066425
N: 332
Anderson-Darling Normality Test
A-Squared: 0.458
P-Value: 0.263
EEC Scorecard Example
Capability Analysis
Process Capability
Analysis for Error for +4
@ +40mA
LSL
Process Data
USL
Target
LSL
Mean
Sample N
USL
ST
LT
1.00000
*
-1.00000
0.34608
332
StDev (ST)
0.076725
StDev (LT)
0.129089
Potential (ST) Capability
Cp
4.34
CPU
2.84
CPL
5.85
Cpk
2.84
*
Cpm
Overall (LT) Capability
-1.0
-0.5
0.0
Observed Performance
0.5
Expected ST Performance
1.0
Expected LT Performance
Pp
2.58
PPM < LSL
0.00
PPM < LSL
0.00
PPM < LSL
0.00
PPU
1.69
PPM > USL
0.00
PPM > USL
0.00
PPM > USL
0.20
PPL
3.48
PPM Total
0.00
PPM Total
0.00
PPM Total
0.20
Ppk
1.69
17
EEC Scorecard Example
Capability Analysis
Process Capability
Analysis for Error for -4
@ -40mA
LSL
Process Data
USL
USL
ST
LT
1.00000
Target
LSL
*
-1.00000
Mean
0.38525
Sample N
332
StDev (ST)
0.089271
StDev (LT)
0.151432
Potential (ST) Capability
p
Cp
3.73
CPU
2.30
CPL
5.17
Cpk
2.30
Cpm
*
Overall (LT) Capability
-1.0
-0.5
0.0
Observed Performance
0.5
Expected ST Performance
1.0
Expected LT Performance
Pp
2.20
PPM < LSL
0.00
PPM < LSL
0.00
PPM < LSL
0.00
PPU
1.35
PPM > USL
0.00
PPM > USL
0.00
PPM > USL
24.58
PPL
3.05
PPM Total
0.00
PPM Total
0.00
PPM Total
24.58
Ppk
1.35
Summary
ƒ Summary
ƒ Customers Expect …..
ƒ 6σ quality levels as well as functional & performance compliance
ƒ A means of tracking this data …. and a part in the process
ƒ Scorecards provide a monitor for KCs throughout design process
ƒ KCs can be defined by the customer, supplier or both
ƒ Typical 6σ metrics used to gauge performance (Cp, Cpk, Zst, DPMO)
ƒ Scorecard
ƒ Makes all design assumptions and targets explicit
ƒ Collation process provides the means to refine KCs
ƒ Locate areas for improvement in design / manufacturing process
ƒ Provides insight into prior products as well as new design
18
DFSS Definitions
Characteristic
ƒ Those dimensional
dimensional, visual,
visual functional and material features or
properties which describe and constitute the design of the item
and can be measured, observed or identified to determine
conformance to the design requirements.
Key Characteristic (KC)
ƒ The Select few, measurable features of a specific part, drawing,
specification or process whose variation can significantly impact
customer satisfaction, manufacturability, durability or
performance. These characteristics include, but are not limited to
the following:
ƒ CTQs, Critical Characteristics,
ƒ Major Characteristics, CTCs and KCCs
DFSS Definitions (Con’t)
Critical To Quality (CTQ) Characteristic
ƒ The select few
few, measurable key characteristics of a specific
part/drawing/specification where reduced variation will have a
positive impact on the customer.
Critical Characteristic
ƒ Those characteristics of an item which, if nonconforming, may
result in hazardous or unsafe conditions for personnel using
maintaining or depending of the unit-of-product; or which may
prevent or seriously affect the satisfactory operation or function of
the unit-of-product. The applicability of this definition should be
evaluated at a point 50% beyond the specification limits.
19
DFSS Definitions (Con’t)
Major Characteristic
ƒ Those characteristics of an item
item, other than critical
critical, which
which, if nonconforming, may result in operational or functional failure of the
item or which materially reduce the usability, physical or functional
interchangeability or durability of the unit-of-product for its intended
purpose. The applicability of this definition should be evaluated at a
point 50% beyond the specification limit(s).
Critical to Cost (CTC) Characteristic
ƒ Those characteristics which drive manufacturing losses upward.
They are usually characteristics that have repetitive MRB activity,
excessive rework loops, low process yields, high scrap, etc.
DFSS Definitions (Con’t)
Key Control Characteristic (KCC)
ƒ A specific characteristic for a specific part or process that will be
monitored by SPC or another appropriate monitoring method, in
order to assure conforming CTQs, KQCs (Key Quality
Characteristics) or CTC. It may be a process parameter or an
output characteristic.
Quality Function Deployment (QFD)
ƒ A communication tool to translate customer expectations /
requirements into specific design and manufacturing requirements.
20