Successful Quality
Assurance Activities
Our Pillars for Success
March 3-4, 2008
Darrell Smith
Mission Assurance Manager
Northrop Grumman Corporation
0
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
So What Happened!
Quality is everyone’s job and essential for survival
1
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
You Have a JOB
 Quality is everyone’s job and essential for survival
2
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
My Objectives – Today Is:
 Share with you some concepts about Quality and Processes
 Share with you the QA role in our processes
 Provide some examples of success
3
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Quality Assurance Exposed





What does Quality Assurance mean to you?
What are the popular perceptions of Quality Assurance?
Why have Quality Assurance at all?
When does Quality Assurance become important?
How do we make Quality Assurance successful?
Could this be your motto?
Quality is important in everything we do.
Quality is everyone’s responsibility and is achieved
through continuous improvement.
We routinely seek ways to do things better.
4
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Is It MY Responsibility?
Quality is everyone's responsibility
Project Managers
Ultimately responsible for the
quality of the products and
services that they provide
customers in accordance with
contractual agreements and
consistent with corporate
policies
5
7/28/2016 2:15 AM
Every Employee
Responsible for the
quality of the products
they produce and the
services they provide
Copyright 2005 Northrop Grumman Corporation
Definition
 Quality Assurance Definitions
 A program for the systematic monitoring and evaluation of the
various aspects of a project, service, or facility to provide adequate
confidence that the product or service will satisfy given
requirements for quality
 Assurance that we deliver what we promise to our customers
 Described by a System of Controls
 Infrastructure that describes how to run the business
 Detailed in policies, handbooks, plans, procedures, etc.
 Usually structured around and compliant to a standard
 AS9100
 ISO
 CMMI
QA Processes Supports The Bottom Line
6
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Program Phases and Quality System Deployment
Management Vision, Commitment, and Review
Design Review
Configuration Management
Document Control
Supplier Quality
Design Management
Supplier Quality Management
Calibration, Inspection, and Documentation
Non-conforming Material Control
Stamps
Servicing
7
7/28/2016 2:15 AM
Design
Manufacturing
Testing
Production
R&D
Test Configuration Management
Documentation
Calibration
Production Configuration Management,
Documentation, Calibration,
Performance Measures, Training, Audits
Configuration Stability
Performance Measures and Product Improvement
Copyright 2005 Northrop Grumman Corporation
Basic Quality System Infrastructure












Management
Design review
Configuration management
Document control
Supplier quality
Calibration
Training
Stamps
Non-conforming material handling
Manufacturing
Inspection
Test




Audits
Production
Measurement
Servicing
If the process is right, the results will be right
8
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Pillars of Successful Quality Systems
 Management Support
 Attitudes
 Cooperation
 Adequate resources
 Coverage and emphasis
 Early involvement
 Teaming
 Detailed supplier involvement
 Gather information
 Choose partners wisely
 Develop relationships and trust
 Aggressively manage progress
 Free communication
 Information is shared freely, respected, and appropriately
considered
 Lessons learned are captured and shared
 The QA counterpart relationship
Communication & Teamwork are Keys that Open Many Doors
9
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Pillars of Successful Quality Systems (cont’d)
 Meticulous data and configuration management
 Revision control
 Accurate description of deliverables
 Careful regard for risk
 Proactively characterized
 Incidents are thoroughly investigated and understood
 Effective corrective action
 Identification
 Tracking
 Root cause analyses
 Closure
 Measurement
 Desire to improve
 Performance evaluation
Those using the best processes will win
10
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
It Could Happen to You!
“If the procedure is right, implementation should pose no problem…”
Dr. Kaoru Ishikawa
11
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Developer Walkthroughs
Milestone Reviews
Example Lifecycle of a SW Build
Start
Start of
of
Increment
Increment
GFC/C
Project
Increment Product
Baseline
Established
Increment
Requirements Baseline
Established
CINC/Site
Configurations Tested
IR
GFC/C
Increment
Increment
Requirements
Integration and Test
Planning
-x
Tests Conducted
• GFC/C subsystems
RWTs
RWTs
GFC/C
Products
Products
Increment Test
Baseline
Established
Baseline
Established
TPWTs
TPWTs
DWTs
DWTs
SW
SWCapability
CapabilityRefine
Refine
IntegrationPrep
Prep
Integration
• Object Class Definitions
CWTs
CWTs
• Integrated Software
Subsystems
TWT
SW
SW Integration
Integration
Increment
Development Baseline
Established
Object
Classes
Classes
RR
CWTs
CWTs
• Coded Object Classes
Object
ObjectClass
ClassImplementation
Implementation
Pacing Benchmarks
12
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Said Another Way..
Requirements /
Top Level Design
Information
Architecture
Requirement
s
Design
Walkthrough
Development
Software
Architecture
Internal
Defect
Reports
Development
Team
Code
Walkthrough
Pacing
Benchmarks
Integration
Team
Test
Walkthrough
Requirements Testing
13
7/28/2016 2:15 AM
Resolution
Trouble
Reports
Copyright 2005 Northrop Grumman Corporation
SW Development/Production Cycle Example
Initial
Review
(IR)
Definition
Phase
Software
Implementation
Phase
Test
Walkthrough
(TWT)
Design
Phase
Code
Walkthroughs
(CWTs )
Software
Integration &
Test Phase
Requirements
Test
Phase
Design
Walkthroughs
(DWTs )
Pacing
Benchmarks
(PBMs)
Release
Review
(RR)
Product Delivery
Peer
Code
Inspections
A Proven Cycle…
14
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
QA Role:
SW Development/Production Cycle
Definition
Phase
• Objectives,
capabilities, &
requirements of the
build/version are
developed/reviewed
• Requirements are
allocated from the
Requirements
Document
• QA Audits
• QA participates in
Requirement
Walkthroughs
• QA participates in
Build Team
Meetings
Initial
Review
(IRs)
• Establishes content/plans
w/regard to requirements
and software capability for
the version
• Build Lead gives approval
to proceed with
development
• Requirements are
baselined through the
Change Board
• QA audits
• QA impounds data from IR
• QA participates in Change
Board activities
• QA participates in Build
Team Meetings
Design
Phase
• Software Architectural Design
- transforms software
requirements into software
design at the software
architectural design level
• Software Detailed Design transforms software
requirements into a
documented software design
at the detailed design level
that can be directly
implemented into code
• QA audits
• QA participates in Change
Board activities
• QA reviews/approves all
documentation
• QA participates in Build Team
Meetings
Design
Walkthroughs
(DWTs)
• Purpose is to
identify defects
via internal defect
reports (IDRs) in
the design
• QA audits
• QA attends and
participates in
peer
reviews/DWTs
• QA impounds data
from peer reviews
• QA reviews IDRs
QA objectively evaluates processes and work products…
15
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
QA Role:
SW Development/Production Cycle
Software
Implementation
Phase
• Transforms the
detailed software
design into
software products
(e.g., code)
Peer
Code
Inspections
• Identifies defects in the software
via IDRs
• Ensure adherence to project
standards and processes
Code
Walkthroughs
(CWTs)
• Identifies
defects in the
software via
IDRs
• QA attends and
participates in
CWTs
• QA impounds
data from CWTs
• QA reviews IDRs
• QA audits
Software
Integration &
Test Phase
• Integration Test integrate the software
units/components into a
reliable software product
that operates correctly
• QA audits
• QA participates in
change Board activities
• QA reviews/approves all
documentation
• QA participates in Test
Procedure Walkthroughs
Pacing
Benchmarks
(PBMs)
• Provides demonstrationbased progress
• QA reviews/approves all
documentation
• QA attends PBMs as an
observer
• QA audits
• QA impounds data from
PBMs
• QA attends and participates in
peer reviews
• QA impounds data from peer
reviews
• QA reviews IDRs
• QA audits
16
QA identifies and documents noncompliance issues…
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
QA Role:
SW Development/Production Cycle
Requirements
Test
Phase
Test
Walkthrough
(TWT)
• Assesses readiness of the
Product/system for
Requirements Testing
• CM captures configuration
• CM locks configuration
• Change Board establishes
Test BL
• Release Notices (RNs)
• Code
• Documentation
• QA reviews all test
procedures prior to test
• QA participates in Change
Board
• QA gathers all data
available to prepare for
configuration verification
• QA audits
• Evaluates Product against
the baselined requirements
to determine whether
requirements have been
successfully met
• Conducted by an
independent test team
•
•
•
•
QA witness of test
QA verifies configuration
QA verifies test results/data
QA reviews all change
paper (STRs)
• QA participates in Change
Board
• QA audits
Product Delivery
Release
Review
(RR)
• Verifies that the Product satisfies
requirements, performs as specified, and
is ready for release
• CM captures configuration
• Change Board establishes Product BL
• RNs
• Code
• Documentation
• QA impounds data “objective evidence”
• QA reviews all change paper (STRs)
• QA reviews all documentation (SVDD and
test reports)
• QA audits
• QA participates in Change Board to
release product to customer
• QA supports verification review with
customer
QA provides feedback to project staff and managers…
17
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Successful Process
Pacing Bench Mark Schedule Example
Mar
Apr
3/17 3/24 3/31
4/7
May
4/14 4/21 4/28
5/5
Jun
5/12 5/19 5/26
6/2
6/9
Jul
6/16 6/23 6/30
7/7
Aug
7/14 7/21 7/28
8/4
8/11 8/18 8/25
Sep
9/1
9/8
9/15 9/22 9/30
2a 2b
DWTs
PBM 1
Design
Demonstration
Walkthroughs
Based c/o
Completed
Part 1
PBM 3
CWTs
• New Functionality
• Demonstration
Based c/o Part 3
Code
Walkthroughs
Completed
PBM 2
PBM 4
• New Functionality
• Demonstration
Based c/o Part 2
• New Functionality
• Demonstration
Based c/o Part 4
PBM 5
• New Functionality
• Demonstration
Based c/o Part 5
TWT
RR
Test
Release
Walkthrough
Review
Conducted Conducted
• Release Tests
• Release Tests
Conducted
• Dryruns
• Integration Tests • Requirements
Test
QA involvement throughout
18
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Project-x Defect Density Example
Defects = STRs since TWT
(after Dev)
Project-x Defect Density
defect density
yellow defect density
red defect density
2.00
DD = STR/KSLOC
1.80
Defects/KSLOC
1.60
1.40
1.20
1.00
0.80
0.60
0.40
0.20
Defect Density:
2/8/08
2/1/08
1/25/08
1/18/08
1/11/08
1/4/08
12/28/07
12/21/07
12/14/07
12/07/07
11/30/07
11/23/07
11/16/07
11/9/07
11/2/07
10/26/07
10/19/07
0.00
10/19/07 10/26/07 11/2/07 11/9/07 11/16/07 11/23/07
STRs Opened
243
245
251
253
253
253
KSLOC
defect density
790
0.31
790
0.31
790
0.32
790
0.32
790
0.32
790
0.32
1
2
1
2
1
2
1
2
1
2
1
2
yellow defect density
red defect density
Outstanding product quality in delivered SW
19
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Project-x Defect Density Example
Prod Ver x Defect Density
Defects/KSLOC
2.00
1.80
Prod Ver x Defect Density
1.60
DD yellow threshold
Defects = STRs
since TWT (after
Dev)
DD red threshold
1.40
DD = STR/KSLOC
1.20
1.00
0.80
0.60
0.40
0.20
Product Defect Density:
01/08
12/07
11/07
10/07
09/07
08/07
07/07
06/07
05/07
04/07
03/07
02/07
01/07
12/06
11/06
10/06
0.00
10/06 11/06 12/06 01/07 02/07 03/07 04/07 05/07 06/07 07/07 08/07 09/07 10/07 11/07 12/07
98
347
100
347
103
347
105
347
109
347
99
347
01/08
Total STRs Opened
KSLOC
94
347
111
347
111
347
133
347
133
347
135
347
139
347
141
347
142
347
142
347
Prod Ver x Defect Density
0.27 0.28 0.29 0.30 0.30 0.31 0.29 0.32
0.32
0.38
0.38
0.39
0.40
0.41
0.41
0.41
DD yellow threshold
1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
DD red threshold
2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00
2.00
2.00
2.00
2.00
2.00
2.00
2.00
2.00
Outstanding product quality in delivered SW
20
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Project-x First-Time Quality Example
FTQ = STRs since RR
Project-x First-Time Quality
yellow first-time quality
red first-time quality
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
FTQ = STR/KSLOC
2/8/08
1/25/08
1/11/08
12/28/07
12/14/07
11/30/07
11/16/07
11/2/07
>1 Defect/KSLOC for New &
Modified code
10/19/07
Defects/KSLOC
first-time quality
First Time Quality:
10/19/07 10/26/07
STRs Opened
143
145
KSLOC
790
790
first-time quality
0.18
0.18
yellow first-time quality
0.50
0.50
red first-time quality
1.00
1.00
11/2/07 11/9/07 11/16/07 11/23/07
147
790
0.19
0.50
1.00
149
790
0.19
0.50
1.00
149
790
0.19
0.50
1.00
149
790
0.19
0.50
1.00
Outstanding product quality in delivered SW
21
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Project-x First-Time Quality Example
Product Ver X First Time Quality
Defects/KSLOC
1.00
0.90
Prod Ver X FTQ
0.80
FTQ yellow threshold
FTQ = STRs
since RR
FTQ red threshold
0.70
FTQ =
STR/KSLOC
0.60
0.50
0.40
>1
Defect/KSLOC
for New &
Modified code
0.30
0.20
0.10
04/07
13
347
0.04
0.50
1.00
05/07 06/07
25
25
347
347
0.07 0.07
0.50 0.50
1.00 1.00
01/08
12/07
11/07
03/07
23
347
0.07
0.50
1.00
10/07
02/07
19
347
0.05
0.50
1.00
09/07
01/07
17
347
0.05
0.50
1.00
08/07
07/07
12/06
14
347
0.04
0.50
1.00
06/07
11/06
12
347
0.03
0.50
1.00
05/07
04/07
02/07
01/07
10/06
8
347
0.02
0.50
1.00
03/07
Prod First Time Quality:
Post-Release SW TRs
KSLOC
Prod Ver X FTQ
FTQ yellow threshold
FTQ red threshold
12/06
11/06
10/06
0.00
07/07 08/07
47
47
347
347
0.14 0.14
0.50 0.50
1.00 1.00
09/07 10/07
49
54
347
347
0.14 0.16
0.50 0.50
1.00 1.00
11/07
56
347
0.16
0.50
1.00
12/07
57
347
0.16
0.50
1.00
01/08
57
347
0.16
0.50
1.00
Outstanding product quality in delivered SW
22
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Product Ver X STR History Example
Cumulative Prod Ver x SoftwareTRs (Trouble Reports)
100
Month Open TRs
Month Closed TRs
Cum TRs Found
Cum TRs Closed
Total Deferred
90
80
Number of TRs
70
60
50
40
30
20
10
23
7/28/2016 2:15 AM
07/07
08/07
09/07
10/07
11/07
12/07
01/08
Month
Cum TRs Found
Cum TRs Closed
Month Open TRs
Month Closed TRs
Total Deferred
06/07
05/07
0
05/07
31
0
31
0
31
06/07
36
0
5
0
36
07/07
54
0
28
0
54
08/07
57
0
3
0
57
09/07
61
0
4
0
61
10/07
64
0
3
0
64
11/07
77
0
13
0
77
12/07
85
0
8
0
85
01/08
91
10
6
10
81
Copyright 2005 Northrop Grumman Corporation
Product Ver X STRs by Severity Example
100
Prod Ver x Trouble Reports (TRs) by
Severity - Jan 2008
90
Number of TRs
80
70
Self-assessed as
GREEN
Found
Removed
Open
60
50
40
30
20
10
0
1
2
Severity3
Prod Ver x TRs by Severity
Severity
Found
Removed
1
0
0
2
2
2
3
35
5
4
54
3
Total
91
10
24
7/28/2016 2:15 AM
4
Open
0
0
30
51
81
Total
SEVERITY Definitions:
1- Crash
2 - Major function perturbation
3 - Minor function perturbation
4 - Superfluous
Copyright 2005 Northrop Grumman Corporation
Product Ver X STRs by Phase Example
100
90
Number of TRs
80
70
Prod Ver x Trouble Reports(TRs) by
Phase - Jan 2008
Found
Self-assessed as
GREEN
Rem oved
Open
60
50
40
30
20
10
0
Pre-TWT
Dev Tes tPhasePos t-RR
Total
Prod Ver x TRs BY PHASE
Phase
Found
Removed
Pre-TWT
22
3
Dev Test
31
6
Post-RR
38
1
Total
91
10
25
7/28/2016 2:15 AM
Open
19
25
37
81
Copyright 2005 Northrop Grumman Corporation
Project-x SW Defect Containment Example
Phase Originated
Req
Phase Detected
IDRs
TRs
Internal
DevelopCode &
Design
Integrati Pre-TWT ment Post-RR
Unit Test
on
Test
Total
Requirements
1
Design
0
37
Code & Unit Test
0
0
36
36
Internal Integration
0
0
96
96
Pre-TWT
1
3
8
12
13
60
73
2
9
47
58
4
62
247
Development Test
Post-RR
Total
1
37
0
0
0
0
Code & Unit Test
- Most Complex
Code only
Internal
Integration –
Primary Error
Finding Process
Post-RR – Inc.
Field
Development
Test =
Req/CR/TR/Regr
ession
313
Evaluation of Defect Containment is done with DD and FTQ and comparison
Over builds. There is no stand-alone objective scoring for green-yellow-red status.
26
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
How Can We Experience Failure?
27
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
How Can We Experience Failure?
 Known Knowns (We know what we know)
 Ops Data/Demonstrated Performance/Test or Analysis Data
 Address by: QA involvement, sound program, engineering,
and operational management, Test as you operate
(understanding waiver implications)
 Known Unknowns (We know what we don't know)




Generic but unseen failure modes and hazards
Unexplained and irreproducible anomalies
Risk analysis uncertainties
Acknowledged test and analysis limitations; unverified
modeling and simulation based predictions
 Address by: conservative operations, technical standards and
safety factors, stress testing, attention to anomalies, trending,
follow QA processes
28
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
How Can We Experience Failure?
 Unknown Knowns (We don't know what we do know)




Mis-communicated test or analysis results
Hesitancy in coming forward with issues
Poor documentation combined with loss of corporate memory
Address by: clear org structure, open communications, peer
review, skill retention, knowledge capture during
“recertification” processes, training, QA involvement
 Unknown Unknowns (We don't know what we don't
know)





29
Bad assumptions, unclear requirements
Untested new or changed environments
Inadvertent operation outside of certification limits
Unknowable complex interactions
Address by: closer customer involvement, research and testing,
challenging past assumptions, independent assessment, follow
processes, QA audit processes
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Failure-Maybe:
Have Tools to Fill the Holes
NEW PROGRAM DEFECTS
DEVELOP PROCEDURES,
AUDIT COMPLIANCE
QA REVIEWS
REQUIREMENTS
INVOLVED IN
SUPPLIER SELECTION
S/W REQUIREMENTS
VERIFICATION, S/W
TEST VALIDATION
ASSURANCE OF PROPER AND
COMPLETE SUPPLIER
REQUIREMENTS FLOWDOWN
QA SITE REPRESENTATION,
SOURCE INSPECTION AT
SUPPLIERS
RECEIVING AND LINE
INSPECTIONS
CALIBRATED TEST
AND INSPECTION
EQUIPMENT
WITNESSES TESTS TO VALID
TEST PLANS
NONCONFORMING MATERIAL IDENTIFICATION, SEGREGATION, CONTROL, AND
DISPOSITION
PROPERLY DEVELOPED PRODUCT THAT WORKS, IS DEPLOYABLE AND SERVICEABLE
MISSION SUCCESS
30
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
Summary







Definition of Quality Assurance
QA elements of a typical quality system infrastructure
Program phases and quality system deployment
Pillars of successful quality system
Reviewed SW Life Cycle
Reviewed QAs Role in SW Life Cycle
Example Metrics
Be Proactive and Communicate
31
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
BACK UP
32
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
For YOU who Like to See a Flow Diagram
RWT
Development
Standalone
Testing
IR
Design
DWT
CWT
Assessment
Required
?
Coding
CASE Tool
Quality Analysis
Peer
Code
Inspection
Engineering
Change Board
(ECB)
Yes
CWT
No
Integration and
Development
Thread Testing
TPWT
TRs
RR
33
7/28/2016 2:15 AM
Formal
Release
Testing
TWT
Adequate
Test
Not
Assessment
Adequate
Adequate
?*
?
Additional
Testing
Copyright 2005 Northrop Grumman Corporation
QA Activities: Design and Development
Design
and
Development
Fabrication
and
Assembly
Subcontract
and Supplier
Management
Integration
and Test
Acceptance
and
Delivery
 QA verifies that project design and development activities
comply with contractual requirements, policy
requirements & project plans

Includes activities for project management, hardware
fabrication & assembly, software design & development,
CM/DM, procurement, etc.
 QA verifies that work products of each design and
development activity comply with established standards

34
Proces
s
Audits
Produc
t
Audits
Includes hardware assemblies, code, SDFs, TDFs,
plans, specs, drawings, procedures, records, tools,
baselines, libraries, etc.
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
QA Activities: Design and Development
Design
and
Development
Fabrication
and
Assembly
Subcontract
and Supplier
Management
Integration
and Test
Acceptance
and
Delivery
 QA participates in or monitors critical meetings & reviews and
conducts process audits
 Peer reviews/walkthroughs
 Technical reviews (Technical Reviews, Test Reviews, Code
turnovers)
 Management reviews (Staff meetings, Sr. Mgmt, PMRs, etc.)
 Product acceptance/sell-off meetings (e.g, FCA/PCA, etc.)
 Review Boards (CCB, MRB, etc.)
 QA documents non-compliances from audits and inspections
 Assures that non-compliances are communicated to the
appropriate personnel, documented, resolved, and tracked to
closure
 Assures that nonconforming products are not used or released
except as defined in project plans
35
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
HW QA Activities: Fab and Assembly
Design
and
Development
Fabrication
and
Assembly
Subcontract
and Supplier
Management
Integration
and Test
Acceptance
and
Delivery
 QA performs activities on prior chart as well as the following:
 Monitors Manufacturing Planning
 Reviews Manufacturing Shop Orders (MSOs) to verify that
 Appropriate inspection points are included
 Appropriate instructions or workmanship standards are
referenced
 Parts lists identify (per drawing) correct part numbers,
revisions and trace code
 Quality inspection planning stamp is applied to documents
that have been approved by inspection planning
 Participates in Material Review Board (MRB)
36
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
HW QA Activities: Fab and Assembly
Design
and
Development
Fabrication
and
Assembly
Subcontract
and Supplier
Management
Integration
and Test
Acceptance
and
Delivery
 Performs hardware fabrication and assembly inspections
 Verify Hardware component/parts/assembly conform to current
drawings
 Verify Visual and Identification
 Verify ESD Requirements
 Generate Inspection Records
 Monitors the use of Measurement Test and Support Equipment
(MT/SE), when applicable, to verify that
 Test procedures using MT/SE are approved and controlled
 MT/SE is calibrated in accordance with the manual
 When MT/SE is inoperative, inaccurate, or malfunctions, or when
its calibration period has expired, the equipment is segregated
and designated as “Invalid for Use”
 Suspect MT/SE used during testing are reviewed to determine
the extent of possible impact upon product conformance
37
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
QA Act: Subcontract and Supplier Mgmt
Design
and
Development
Fabrication
and
Assembly
Subcontract
and Supplier
Management
Integration
and Test
Acceptance
and
Delivery
 On projects with subcontractors, project quality requirements
are flowed to the subcontractor in two ways:
 QA requirements may be stated explicitly in the subcontract SOW
 Quality requirements may be defined in a PAR, SMART, or
equivalent document that must be referenced in the subcontract
SOW
 Where specific quality requirements are to be imposed on
suppliers (e.g. compliance with ISO 9001/CMMI), quality clauses
are included in the Purchase Order and/or SOW
38
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
QA Act: Subcontract and Supplier Mgmt
Design
and
Development
Fabrication
and
Assembly
Subcontract
and Supplier
Management
Integration
and Test
Acceptance
and
Delivery
 QA performs procurement document reviews (PDRs)
 Assures that quality requirements are provided to suppliers, as
required
 Assures that drawings or specifications provided to suppliers are
accurate, controlled and of the correct version
 QA may be required to perform source surveys of
subcontractors/suppliers to determine process capability prior to
procurement
 QA performs in-process surveillance inspections as defined in
subcontract or procurement document
 Assures that contractually-defined processes are followed
 Assures that contractually-specified product quality is met
39
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
QA Act: Subcontract and Supplier Mgmt
Design
and
Development
Fabrication
and
Assembly
Subcontract
and Supplier
Management
Integration
and Test
Acceptance
and
Delivery
 QA performs receiving inspections on procured items, GFE and
COTS to

Verify that what was procured or requested is what was
received

Verify that procured item meets its procurement specifications

Verify that GFE meets contract requirements

Verify that the condition of the items is acceptable (e.g., not
damaged)
 QA provides oversight of subcontractor QA activities as defined
in the subcontract
40
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
QA Activities: Integration and Test
Design
and
Development
Fabrication
and
Assembly
Subcontract
and Supplier
Management
Integration
and Test
Acceptance
and
Delivery
 QA monitors integration and test activities to assure that:
41

Test documentation (plans, procedures, results) are reviewed

Test support software is tested and qualified prior to start of
qualification/acceptance testing and MT/SE is properly calibrated

Only the latest released versions of the test procedures and test
support software are used

Test readiness reviews are held to identify and/or resolve open
issues

The customer’s quality personnel, when required, are apprised of
test scheduling and performance issues

The integration and test activities are monitored as defined in the
test planning documentation
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
QA Activities: Integration and Test
Design
and
Development
Fabrication
and
Assembly
Subcontract
and Supplier
Management
Integration
and Test
Acceptance
and
Delivery
 For Qualification/Acceptance Tests, QA assures that:
 Test anomalies or problems are documented and resolved per project
plans
 Test Data Files (TDFs) and other records of tests are initiated and
maintained as described in the project plans
 Changes made to the test procedures during testing are documented
and controlled
 The hardware/software configuration for the test is defined in the test
plan/procedure and is verified to be accurate prior to test initiation
 Tested hardware/software meets all of the performance parameters
specified in the test plan/procedure
 The final test report accurately reflects what occurred during the test
 All actions have been closed or are identified as being open in the test
report
 There are appropriate approvals of the test report
 Test failure reviews (lessons learned) are performed
42
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation
QA Activities: Acceptance and Delivery
Design
and
Development
Fabrication
and
Assembly
Subcontract
and Supplier
Management
Integration
and Test
Acceptance
and
Delivery
 For product deliveries, QA assures that:
 Items to be delivered/shipped:
 Have received a planned level of acceptance
 Waivers/deviations have been documented
 The hardware/software configuration is correct and properly
identified and marked
 Are, in fact, present
 Packaging and container marking is proper
 Delivery/shipping documentation is completed and coordinated in
accordance with contract requirements
 Customer Certification, if applicable, is included
43
7/28/2016 2:15 AM
Copyright 2005 Northrop Grumman Corporation