Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
LeanTest key:
Test coverage analysis
powered by traceability
Christophe LOTZ
christophe.lotz@aster-technologies.com
ASTER Technologies
Conclusion
IEEE 11th International
Board Test Workshop
1
If there is no solution, there is no problem
Targets: Key objectives
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
Our targets are to provide tools that:
 Create an effective environment to continually
improve the delivered quality of manufacturing
processes.
 Assist in reducing costs of assembly, test,
rework, scrap and warranty.
 Help improve line utilization and reduce cycle
time.
 Allow manufacturers to better prioritize the
deployment of constrained resources.
 Allow manufacturers to benchmark their DPMO
rates to others in the industry.
… regardless of board complexity.
2
If there is no solution, there is no problem
Test coverage and traceability
Definitions
• Key Objectives
 Good products must be defect-free and cheap.
 How to detect or prevent all faults on the product
so that only good products are shipped?
• Defect Universe
• Test Coverage
Defect Detection
• Test efficiency
Case Studies
Defect Prevention
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
 Test coverage is a key metric as it will be the
quality warranty and the main driving factor for
LeanTest.
 This paper describes how traceability tools
should be used in order to improve test
coverage understanding.
3
If there is no solution, there is no problem
Defect Universe
 Identify the faults that can occur.
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Solder
•
•
•
•
•
X(unpowered)
Ray
Insufficient
Excess
Cold Solder
Marginal
Joints
Voids
•
•
•
•
•
• Extra Part
• Bridging
• Tombstone
• Misaligned
In-Circuit
Material
Polarity (PCAP)
Missing
Gross Shorts
Lifted Leads
Bent Leads
Conclusion
• Polarity
Placement
AOI
•
•
•
•
•
Dead Part
Bad Part
In-System
Programming
Functionally Bad
Short/Open on PCB
• Shorts
• Open
JTAG
• Inverted
memory
• Wrong Part • At-speed
tests
• At-speed
interconnect
• Fault Insertion
• Gate level
diagnosis
(unpowered)
4
If there is no solution, there is no problem
Defect Universe
 Typical manufacturing defects:
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
Missing
Wrong value
Misalignment
components
Open
Tombstone
Broken
Incorrect
Circuits
components
Polarity
Short
Excessive
Insufficient
circuits
solder
solder
• Test repeatability
• Real contribution - AOI
Conclusion
 We need to group defects into categories, to
understand what defects can be captured by a
particular test strategy.
Material
(Supply chain)
Placement
Solder
5
If there is no solution, there is no problem
Test coverage
Definitions
• Key Objectives
• Defect Universe
 The ability to detect defects can be expressed
with a number: coverage.
 Each defect category fits with its test coverage:
• Test Coverage
• Test efficiency
MPSF [1]
PPVSF
PCOLA/SOQ /FAM
Case Studies
Material
Value
Correct
• Faulty boards
• DPMO estimation
Live
Placement
Presence
• Test repeatability
Presence
Alignment
• Real contribution - AOI
Conclusion
Solder
Polarity
Orientation
Solder
Short
Open
Quality
Function
Function
Feature
At-Speed
Measure
If there is no solution, there is no problem
Test coverage by defect category
Definitions
• Key Objectives
• Defect Universe
For each category (Material, Placement, Solder) of
defects (D), we associate the corresponding
coverage (C).
 DM  CM +  DP  CP +  DS  CS +  DF  CF
• Test Coverage
• Test efficiency
Effectiveness =
Case Studies
 DM +  DP +  DS +  DF
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
The test efficiency is based on a coverage
balanced by the defects opportunities.
Conclusion
Coverage
We need a better coverage
where there are
more defect opportunities!
DPMO
7
If there is no solution, there is no problem
Test coverage by defect category
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
 Each test technique brings a certain ability to
detect the defects defined within ‘defect
universe’.
 No single solution is capable of detecting all the
defects.
M
P
S
F
M
P
S
F
M
P
S
F
M
P
S
F
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
 Good coverage = combination of tests.
8
If there is no solution, there is no problem
Case 1: Faulty boards at system level
Definitions
• Key Objectives
 Electronic plants, in charge of board integration,
often discover a significant amount of defective
boards at system test.
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
AOI
ICT
BST
FT
 How is it possible to get failures at system level
if we only buy good boards?
 The defect appears during packing and transportation
(vibration, extreme temperatures, moisture).
 The defect is a dynamic problem which is revealed by
the integration of the board in the complete system.
 The reality is usually more simple…
9
If there is no solution, there is no problem
Case 1: Faulty boards at system level
Definitions
• Key Objectives
 If the board is failing at system test, it is usually
because the escape rate (or split) is higher than
expected.
• Defect Universe
Good
• Test Coverage
• Test efficiency
Products
shipped
FPY
Case Studies
• Faulty boards
• DPMO estimation
Slip
Pass
Test
False
reject
• Test repeatability
• Real contribution - AOI
Conclusion
Fail
Bad
Good
Products
repaired
FOR
Bad
 There are only two possibilities:
 The combined coverage is lower than optimal.
 The DPMO figures are higher than expected.
10
If there is no solution, there is no problem
Case 1: Faulty boards at system level
 The auditing conclusions were:
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
 Wrong or inadequate coverage metrics are produced:
Example: confusion between accessibility and
testability; coverage by component only - without
incorporating solder joint figures ; Over optimistic
report (marketing driven report),
 Wrong DPMO figures due to limited traceability or
incorrect root cause analysis (Example: confusion
between fault message and root cause/defect).
• Real contribution - AOI
Conclusion
Coverage
estimation
Coverage
measurement
Selected
strategies
11
If there is no solution, there is no problem
Case 2: DPMO estimation
 Going beyond solving surface issues.
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Solder
Material
Insufficient solder
Short
Open
Missing components
Conclusion
Polarity
Broken leads
Tombstone
wrong value
Misalignment
Placement
12
If there is no solution, there is no problem
Case 2: DPMO estimation
 The weighted coverage (with DPMO) is a key factor
to estimate the production model.
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
 We need an accurate value for DPMO if we want
realistic production models.
13
If there is no solution, there is no problem
Case 2: DPMO estimation
 Production model in a test line
Definitions
• Key Objectives
• Defect Universe
AOI
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
ICT
• Real contribution - AOI
Conclusion
FT
14
If there is no solution, there is no problem
Case 2: DPMO estimation
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
 Basic analysis uses average numbers coming from
iNemi or the PPM-Monitoring.com web site. It does
not reflect the reality, but it is much better than
considering all defects as equally probable!
 The best approach is to use the traceability database
in order to extract a table including parameters such
as partnumber, shape, mounting technology, pitch,
number of pins, function/class, DPMO per category
Database
(MPSF).
Repair
Conclusion
Test, inspection
& other machines
Assembly
machines
15
If there is no solution, there is no problem
Case 2: DPMO estimation
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
 Define data collection methods around existing
IPC standards.
 IPC 9261 In-Process DPMO and Estimated Yield.
 IPC 7912 Calculation of DPMO and
Manufacturing Indices for PCBAs.
 Define data stratification and classification
methods.
 Combine the data into a single database:
 DPMO for Material (Part number).
 DPMO for Placement (Package type).
 DPMO for Soldering (Reflow & Wave).
It requires good cooperation between test and quality
services.
16
If there is no solution, there is no problem
Case 3: DPMO estimation
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
 Range and standard deviation for any DPMO
statistic.
 Compare actual yield to estimated yield:
 By test step.
 Full test line.
 Correlation of test coverage/strategy to DPMO
rates.
• Test repeatability
• Real contribution - AOI
Conclusion
17
If there is no solution, there is no problem
Case 2: DPMO estimation
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
 During test coverage analysis, TestWay uses
various algorithms to estimate the DPMO.
  Same Part Number,
  Same shape for placement DPMO,
  Same pitch and number of pins.
 With an accurate DPMO representation, it is
possible to:
 Estimate the yield and the escape rate. Two key
factors used to select the best Contract Manufacturer
or EMS - DPMO figures per EMS site.
 Identify the real overlap for test/inspection
optimization. DfT becomes one of the
principal contributors in cost reduction.
18
If there is no solution, there is no problem
Case 3: Test Repeatability
Definitions
• Key Objectives
 During first production run, we selected a set of
boards where a SPC analysis has been conducted in
GR&R context.
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
 Gage R&R
(Gage Repeatability
and Reproducibility) is the
amount of measurement
variation introduced by
a measurement system,
which consists of the measuring instrument itself and the
individuals using the instrument. A Gage R&R study is a
critical step in manufacturing Six Sigma projects, and it
quantifies :
 Repeatability – variation from the measurement instrument.
 Reproducibility – variation from the individuals using the instrument.
19
If there is no solution, there is no problem
Case 3: Test Repeatability
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
Quality and traceability analysis helps to
compute the classic Cp, CpK … and CmC.
CmC means “Calibration and Measurement
Capability”. CmC = Tolerance / k   (k = 6 for
critical components).
 A test which is not repeatable cannot claim to
qualify a component. So CmC is used to weight
the Correctness coverage.
 In addition, the Failure Mode and Effects
Analysis (FMEA) gives the criticity per
component which limit the oversized test
tolerance.
20
If there is no solution, there is no problem
Case 3: Test Repeatality
 Passive measurements
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies





Correct value: Value is tested at 100%.
Minor deviation: Value is tested at 95%.
Medium deviation : Value is tested at 50%.
Major deviation: Value is not tested.
Incorrect value: Component is not tested.
• Faulty boards
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
For more accuracy:
Compare CAD value
and tolerances against
minimum and maximum
tested values
95%
50%
0%
21
If there is no solution, there is no problem
Case 4: Real contribution of AOI/AXI
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
 AOI and AXI are inspection techniques which
are checking for deviations.
 When deviation is big enough, it should become
a defect.
 When a test line includes an inspection machine
and an electrical tester (ICT, BST), it is difficult
to agree on test contribution.
• Test repeatability
• Real contribution - AOI
95%
47%
53%
Conclusion
AXI
BST
FT
22
If there is no solution, there is no problem
Case 4: Real contribution of AOI/AXI
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
• DPMO estimation
• Test repeatability
 With a traceability system that collects
defects/repair information in real time, we are
able to record that a fault has been detected and
how it has been diagnosed (ie: Root cause
analysis).
 We compare the defects that have been
detected with ICT and FT against the defects
detected by AOI in order to adjust real coverage.
• Real contribution - AOI
Database
Conclusion
Test Coverage Analysis
AOI
ICT
FT
Diagnosis/
Repair
23
If there is no solution, there is no problem
Conclusion
Definitions
• Key Objectives
• Defect Universe
• Test Coverage
• Test efficiency
Case Studies
• Faulty boards
 Continual reassessment of capability metrics.
 Improved accuracy of quality estimations.
 Enhanced defect detection rate by increasing
the understanding of test coverage.
 Reduced escape rate (bad boards to the
customer).
• DPMO estimation
• Test repeatability
• Real contribution - AOI
Conclusion
Zero-defect road…
24
If there is no solution, there is no problem
25
If there is no solution, there is no problem