Risk-Based Commissioning &
Qualification Benchmarking
Educational Track #1
Agenda
Part 1
(8:30 – 10:00)
• ISPE Baseline® Guide Volume 5: Commissioning &
Qualification vs. ASTM E2500 – Mr. Robert Chew
• Overview of C&Q approaches – Abbott, Lilly, Pfizer
Break – Vendor areas open
Part 2 (10:30 – Noon)
• Panel Q&A
Wrap-up
(10:00 – 10:30)
ISPE Baseline Guide Volume 5:
Commissioning and Qualification,
vs. ASTM E2500
Robert E. Chew, PE
President, Commissioning Agents, Inc.
My Background in Subject
• Official trainer for ISPE Baseline
Commissioning and Qualification
Guide
• Member of team that wrote ASTM
E2500
Key Points to Compare
•
•
•
•
•
Terminology
Qualification practices
Focusing the effort
Roles and responsibilities
Managing changes during the project
Terminology
• C&Q Baseline guide
• Commissioning
• Qualification
• ASTM E2500
• Verification
Qualification Practices
• C&Q Baseline Guide
• “To meet the demands of the regulators”
• ASTM E2500
• Not mentioned
Focusing the Effort
• C&Q Baseline Guide
• Impact assessments
• Direct impact systems and critical
components
• V-model
• ASTM E2500
• Risk assessment -> critical aspects
Roles and Responsibilities
• C&Q Baseline Guide
• Quality pre and post approves protocols; quality control
role
• ASTM E2500
• Quality involved in process requirements, risk
assessments, critical aspects, and acceptance and
release
• Subject matter experts used throughout
Managing Change
• C&Q Baseline Guide
• QA pre-approved change control
implemented at start of IQ/OQ
• ASTM E2500
• QA pre-approved change control
implemented at start of manufacture of
product for distribution
Common Elements
•
•
•
•
•
•
Good engineering practices
Planning
Requirements definition
Engineering design reviews
Project change management
Documented inspections and testing
Risk-Based Commissioning &
Qualification Approach
Abbott Laboratories
Global Pharmaceutical Operations
Steve Connelly
Overview
Where Abbott Is Today
C&Q Approach
Commissioning & Qualification Activities
How We Manage Cost
Lessons Learned
Where Abbott is Today
Risk Based Maturity Model*
*Adapted from 2007 ISPE Tampa Conference Presentation
Commissioning & Qualification Approach
 C&Q Approach:
• Application of ISPE Baseline® Guide Volume 5:
Commissioning & Qualification (last 2 years)
• Little to no implementation of ASTM E2500-07*
• Level 3 (Risk-based Maturity Model)
 Implementation:
• Corporate policies support Science & Quality Risk
Management and includes Good Engineering Practice
(GEP) reference
• Training ( RBA, GEP, ECM. Commissioning, Risk Assessments)
• Tools and Templates
• Discretion of the Validation Review Board (VRB)
*ASTM E2500-07: A Standard Guide for the Specification, Design, & Verification of Pharmaceutical and Biopharmaceutical Manufacturing Systems and
Equipment
START
User
Requirements
Direct
Impact
Systems
VCR Submitted
w/Assessment
Report and
Rationale
Functional Design
Evaluate Risk/
Impact
Assessment
Does system
directly impact
product quality?
Yes
No
Science &
Quality Risk
Management
Qualification
Process
Flow
Design
Qualification
Does system
support a direct
impact system?
No
Impact
Systems
Commissioning
Plan
Indirect
Impact
Systems
Yes
VCR Submitted
w/Assessment
Report and
Rationale
No
VCR Submitted
w/Assessment
Report and
Rationale
Commissioning
Plan
Pre-Qualification
Engineering tests /
FAT as applicable
Engineering tests /
FAT as applicable
Process
Complete
VCR Closed
Yes
Installation
Qualification
No
System have
operational
functions?
Yes
Operational
Qualification
Support of
CPPs tested in
PV?
No
Performance
Qualification
Good Engineering Practices
 Formal acknowledgement of GEP in corporate policy
 Divisional GEP guideline consists of minimum requirements
 Additional divisional guidelines are in development for lifecycle management
– Calibration
– Maintenance
– Engineering change management
 Training to aid site support (implementation)
Requirements Based On Impact Category
Three methods:
–ISPE-Style System* Impact Assessment Questionnaire
–Risk Assessment with any standard method (FMEA, HAZOP, etc.)
–Technical Evaluation by SMEs
•Documented and VRB approved
•Determining impact  to identifying critical elements
Quality Risk Control
–Risk Assessment (i.e. FMEA, etc) used to enable:
•Qualification focus on critical functions and/or components
•Leverage limited commissioning testing (FAT/SAT) and
•Eliminate some redundant activities (i.e. alarms, etc)
*System = equipment, facility, utility
Commissioning
 When supporting Qualification, Commissioning must
include or address:
• Engineering Change Management (ECM)
– After point commissioning data intended to support
Qualification
• Testing
– Methods described in sufficient detail for repeatability
– Documentation confirms critical elements met criteria
– Acceptance criteria documented prior to execution
– Criteria based on User Requirements
– Documentation of observed results
 Commissioning & Qualification (C&Q) Plans typically
contain run rules for project
Qualification – Equipment

User requirements
• Product and Process User Requirement (PURS)
• General User Requirements (GURS)

Commissioning (in support of qualification)
• Test it once (or one set of multiple test repetitions, where
appropriate); document it correctly
• Primarily installation activities, operational testing
leveraged in some instances (VRB discretion);compile,
summarize test results for VRB approval

Qualification
• When approved, commissioning reports complete the
IQ/OQ phase of Qualification with no additional “protocol”
testing
Qualification – Software (APC/HMI/DCS)


Categorization (based on product quality impact)
• Managing control issues for critical elements
•
Supplier Assessments for direct impact systems only
•
VRB discretion with technical advisory from Subject
Matter Expert
Holistic Approach
• Integrated Qualification document (IOQ for
equipment/controls; as applicable)
•
APC specific qualification requirements
– Traceability matrix = Design Qualifications (DQ)
– Commissioning = Qualification except critical items
Qualification – Facilities

Indirect Impact Facilities - Commission
– Design and construction conforming to GEP
– Abbott Engineering Standards

Direct Impact Facilities – Commission/Qualify
– Meet quality specifications (i.e. Aseptic facility, etc)
– Design and construction conforming to GEP
– Abbott Engineering Standards
Qualification – HVAC

Categorization
•
Interdependent with facility quality attributes
– For example: Continuous monitoring for critical
environmental aspects; therefore facility and HVAC
can be Indirect Impact (HVAC controls would then be
qualified)
– For example: Temperature, RH, and room pressure
differentials typically critical elements which are
monitored and recorded continuously, alarming when
parameters exceed specifications; VRB discretion as
to qualify
Qualification – Utilities

Impact Categories - Typically critical or non-critical

Utility systems typically not big source of Validation
Change Request (VCR) sources

Only regulatory requirement concern is typically Potable
Water feed to purification
– Purpose of pretreatment usually to preserve life of
purification equipment rather than direct impact of water
quality
How We Manage Cost…
$ In 2007, reviewed C&Q costs for several capital projects
to:
– Establish a baseline
– Compare costs for the “traditional, validate everything” C&Q
approach versus the risk-based C&Q approach
$ Cost comparisons across projects difficult because
– Inconsistent reporting and project scope varies widely
$ Drive cost reporting model for capital projects needed to:
– Address scope differences, track additional parameters:
 Number of Qualification Protocols per project
 Number of Qualified Systems
 Total Number of Systems
– Three parameters documented in C&Q Plan
How We Manage Cost…
Commissioning & Qualification Cost Tracking Details
• Monthly Capital Project Status
Reports are used to track
costs and progress
• There is a wide variation in the
tracking systems used by the
engineers for the project costs
• Most project engineers /
project managers use
spreadsheets of their own
design to track costs
How We Manage Cost…
Opportunities Identified for Consistent Reporting
Engineering/Commissioning/Qualification costs categorized below
 Activities not linear, categorization to clarify “gray” areas of cost
reporting
ENGINEERING
FEP 1-3
User Requirements
Functional Specification (Draft)
Preliminary Design
Detailed Design
Construction
Loop Tuning
Equipment Installation/Alignment
COMMISSIONING
Risk Assessment
Failure Mode Effect Analysis
I/O checkout
Calibration
Functional Specification (Revisions)
Factory Acceptance Testing
Startup
Trouble Shooting
Sequence Testing
As-Builts (Non Qualified Systems)
QUALIFICATION (CERTIFICATION)
Validation Plan
Design Qualification
Equipment Installation Qualification
Equipment Operational Qualification
Computer Hardware Installation Qualification
Computer Software Installation Qualification
Controller Modification Qualification
HVAC Installation/Operational Qualification
Utility Installation Qualification
Process Qualification
Validated Drawings
Samples/Testing
Cleaning/Passivation
Inspections
Engineering Work Orders / Contracts
 Defines appropriate cost “bucket” for consistency across
projects
Lessons Learned
Risk-Based C&Q is an iterative process (one
tiny
step at a time)
 Corporate & divisional policies in place, tools developed,
although success for various sites, concern about corporate &
regulatory audits
 Sites want to maintain autonomy with guidance; common
elements, not same approach
(i.e. all engineers comply however variations due to specialty)
 Application:
Site readiness tools - identifying opportunities for closing gaps
Lessons Learned
Listen, Speak, Deliver. Communication is key.
 Quality may wonder:
What if engineering doesn’t do its job?;ensure compliance
 Engineering may wonder:
Quality = police & spell checkers; “right” the first tyme 
 Application:
Regular/frequent communication with VRB & project team
Lessons Learned
Sustainable Paradigm Shift
 Sites: Implementation, accountability & ownership a must for
sustainability
 Resources: People, time & money, site retaining technical
knowledge
 Application:
Training & support implementation as applicable
Thank you!
• Questions will be addressed during
the Panel Q&A session (after break)
Eli Lilly & Company
Commissioning and Qualification
Approach
Eli Lilly & Company Approach to
Commissioning & Qualification
• C&Q Delivery Strategy
• C&Q Metrics
• C&Q Lessons Learned
• Improved C&Q Delivery Strategy
Recent GFD C&Q Delivery Strategy
~ Baseline Guide approach (’02 – ‘07)
•
•
•
•
Developed procedures governing C&Q activities
Fully commissioned all assets ~ GEP
Reserved qualification for critical aspects of DI systems only
Leveraged commissioning documentation to support
qualification – commissioning subject to GDPs
• Involved Quality at C&Q strategy level
• In addition,
• Simulation has been used to commission automation aspects
• CSV testing has been integrated within C&Q test documents
Fundamental C&Q Delivery Strategy
~ for Global Facilities Delivery (GFD) Capital Projects
Profile
Approve
CQMP
System
Classification
DQ/DR
Component
Classification
Planning
Design
Construction
QA
Requirements
Initial Cals
Procurement
Loops
Construction/
Installation
Simulation/
Offline Testing
Commissioning
Package
Project
Commissioning
IQ OQ PQ
Qualification
Owner
IV
FT
SAT
FAT Startup
Approve Systemlevel Plans
RV
Validation
Operations/
Maintenance
C&Q Delivery Strategy ~ Results
This strategy has resulted in:
• Earlier resolution of issues
• Opportunity to understand systems
• Reduced Qualification time
• Reduced Qualification costs
• Fewer Qualification comments/discrepancies
C&Q Metrics
• Schedule Metrics
• Cost Metrics
• Discrepancy Metrics
Typical C&Q Schedule Metrics ~ Durations
• 10 months from facility mechanical completion to OQ completion
• Total Commissioning duration: 24 mos. (172 days/system)
• Total IQ/OQ duration: 6 months
• FT duration/system: 42 days
• IQ duration/system: 11 days
• OQ duration/system: 17 days
• Average # of days from MC to OQ completion: 227 days
Note: Metrics for new powder filling facility consisting of approx. 40 systems
C&Q Cost Metrics ~ Historical view
C&Q Cost
as % TIC
40%
25%
20%
10%
10%
5%
5%
No C&Q prog
1998- 2001
Part. C&Q prog
2001-2003
Full C&Q prog
2002- pres.
3% GOAL
C&Q Budget Metrics
~ capital projects completed in ‘07
Total Installed Cost
$319MM
$186MM
$180MM
$145MM
$14.5MM
$11.6MM
$7.2MM
C&Q Cost (% TIC)
6.0%
1.5% (comm. only)
5.5%
5.7% (incl. non-CQ activities)
4.3%
4.3%
2.2%
Average:
4.7% (C&Q projects only)
C&Q delivery strategy has resulted in a step change in C&Q savings!!
Lilly C&Q Work Breakdown Structure
Lilly budgets/tracks C&Q expenses via 4 primary
WBS categories:
•
•
•
•
Planning & Design
Commissioning
Qualification
Administration
New C&Q Cost Data spec recently approved to
provide guidance
Planning & Design - Subcategory
•
•
•
•
•
•
•
•
•
•
•
•
C&Q plans and strategy
Initial schedule development
Estimating
System classification
Component classification
System C&Q plans
Vendor assessments
User requirements
Design review, spec review, drawing review reviews/tag list
Engineering requirements
Design Qualification
Traceability Matrices
Commissioning - Subcategory
•
•
•
•
•
•
•
•
•
FAT development
FAT execution
Receipt Verification development
Receipt Verification execution
Installation Verification development
Installation Verification execution
SAT development
SAT execution
Construction turnover packages
•
•
•
•
•
•
•
•
Start-up protocol development
Start-up protocol execution
Functional testing development
Functional testing execution
Commissioning packages
C&Q plan summary reports
FAT travel
Supporting documents
Qualification - Subcategory
•
•
•
•
•
•
IQ development
IQ execution
OQ development
OQ execution
PQ development
PQ execution
Administration - Subcategory
•
•
•
•
•
•
Training
Project management
C&Q resource travel
Administrative assistance
Audits
CSV support
Typical C&Q Budget Metrics
~ Cost Breakdown (%)
Procedures 2%
OQ 2%
IQ 2%
CCs 2%
Training
1%
Impact Assessment
SAT
RI 1%
FAT
1%
1%
1%
Doc Revs 1%
Scheduling 0%
Doc mgmt 12%
DQ/EDR 3%
Project mgmt 11%
URs 3%
CSV Support 3%
FT 11%
Construction QA 4%
IV 7%
Planning 5%
SU 9%
Note: Only includes cost data for C&Q contractor
Travel 10%
RV 10%
C&Q Discrepancy Metrics ~ Philosophy
C&Q delivery strategy based on risk mitigation approach
• DR/DQ reduced risk during Construction/Build, Commissioning,
Qualification
• Construction quality assurance activities reduced Commissioning risks
• Simulation reduced risk during Commissioning (Startup, SAT, FT)
• Commissioning (RV, IV, FAT, SAT, FT) reduced risk during subsequent
Commissioning activities and Qualification
• As risk decreased, the number of discrepancies also decreased resulted in cleaner qualification documentation and better schedule
adherence during qualification
Typical C&Q Discrepancy Metrics
# of Discrepancies
700
600
500
400
300
200
100
0
RV
SIM FAT
IV
SU
SAT
IL
IC
FT
CP
IQ
OQ
C&Q Activities
 1230 discrepancies resulting from Construction QA activities
 2800 discrepancies during commissioning
 130 discrepancies during qualification
PQ
CQP
US
Pr
ot Aut
oc
om
ol
De atio
ve n/S
lo
pm oftw
M
ar
en
iss
e
t
Er
in
g/
r
M
In
ec ors
co
I
rre nco han
ic
ct
r
Do rec al
cu t D
a
m
en ta
Tr
ac tatio
kin
n
g
To
Ta ol
M
g
od
Pr
el ging
ot
oc
De Num
ol
c
b
M Exe ision er
iss
cu
in
tio Too
g
M nE l
od
rr
Un el N ors
u
av
a i mb
la
bl er
e
Pr
da
oc
Sy
ed
Vo ta
st
em
ur
id
Ch ed
M al/P
at
er lan ang
ial
D
e
of evi s
a
Co
t
ns ions
t
M
isc ruct
io
el
n
la
ne
ou
De s
sig
n
# of Discrepancies
Typical C&Q Discrepancy Metrics ~ Categories
900
800
700
600
500
400
300
200
100
0
Accepted
Corrective Action
C&Q Lessons Learned
1. Properly define User Requirements

Testable, verifiable, applicable

Not too prescriptive, no design details, no guesses

Include C&Q engineers in review/approval process
2. Clearly define data and documentation requirements
3. Change management

Don’t implement formal change control too soon

Don’t confuse change control with version control

Ensure commissioning change process is well understood

Ensure construction changes are communicated to team

Ensure automation changes are reflected in design docs
Current GFD C&Q Delivery Strategy
~ Enhanced Baseline Guide approach
• Conducted several six sigma initiatives to improve C&Q processes
• Developed additional procedures and best practices pertaining to C&Q
•
•
•
•
GDP SOP
Change Management SOP
Transfer of Care, Custody and Control SOP
Discrepancy Management Best Practice
• Developed new specifications governing the following
•
•
•
Vendor data and documentation
FAT/SAT
C&Q cost data
• Defined new project information management role
• Implementing new requirements definition process based on systems
engineering principles
• Implementing formal Construction QA program
(last 2 years)
Current GFD C&Q Delivery Strategy
~ Enhanced Baseline Guide approach
(cont.)
• Developed Automation Lifecycle Management engineering standard to
accommodate integration of CSV and C&Q resulting in:
•
•
•
Elimination of duplicate requirements, design reviews, testing
Elimination of “all or nothing” approach to software testing
Allowance for risk-based delivery strategies Leverage offline or simulation testing
to meet requirements
• Assessing risk in areas beyond product quality to streamline C&Q delivery
•
•
•
Vendor risk
Technology risk
Automation strategy
• Leveraging vendor testing to support commissioning vs. just using as
contractual gate/risk mitigation
The resulting C&Q delivery strategy will hopefully be realized as another
step change in C&Q savings! Potential C&Q costs as low as 3% of TIC
Questions?
Q&A
Risk Based Commissioning & Qualification Benchmarking
Presented by:
Nick Andreopoulos
Senior Manager/Team leader
Pfizer Global Engineering
Peapack, NJ
Chris Beltz
Senior Manager/Team Leader
Pfizer Quality Operations
Kalamazoo, MI
Agenda
•
•
•
•
Basis of C&Q Practices
C&Q Program Management Approach
C&Q Cost Management
Key Lessons
Current C&Q Practices
• Primarily based on C&Q Baseline® Guide 5
• Certain sites have streamlined C&Q process
• C&Q “Lite” version for small projects and minor equipment
changes
• Leveraging of Commissioning tests into IOQ
• Applying some ASTM concepts – Focusing IOQ on risk to
product quality/patient safety
• Functional Level Impact Assessment - Replacing
Component Level Impact Assessment
• Currently developing and piloting full ASTM E2500
based Verification approach at targeted sites
C&Q Program Management
• C&Q supported jointly by Quality,
Engineering and Automation/IT
• Quality requirements defined in Corporate
Quality Standards
• C&Q Approach defined in Global Practices
• Sites develop their own procedures and
validation master plans based on the global
documents
C&Q Cost Management
• C&Q costs that are associated with
equipment qualification are included
in Project Capital budgets. Process
Validation and Cleaning Validation are
not included in project budgets
• C&Q cost managed by Project team
Key C&Q Lessons Learned
• Upfront involvement by key stakeholders in the
Assessment steps to define the appropriate scope
for qualification
• IOQ scope should be focused on critical
aspects/functions. All other testing is GEP
• Testing documents should be based on SME
execution and leveraged, where possible, from
vendors. Level of documentation should be
commensurate with risk to product quality/patient
safety.
• Roles/Responsibilities need to be clearly defined
at the outset of projects
Risk-Based Commissioning &
Qualification Benchmarking
Educational Track #1
Part 2
Agenda
Part 1
(8:30 – 10:00)
• ISPE Baseline® Guide Volume 5: Commissioning &
Qualification vs. ASTM E2500 – Mr. Robert Chew
• Overview of C&Q approaches – Abbott, Lilly, Pfizer
Break – Vendor areas open
Part 2 (10:30 – Noon)
• Panel Q&A
Wrap-up
(10:00 – 10:30)
Thank You!!
•
•
•
•
•
•
•
•
•
•
Robert Chew, Commissioning Agents
Matt Warhover, Commissioning Agents
Steve Connelly, Abbott Laboratories
Dave Dolgin, Abbott Laboratories
Scott Hamm, Eli Lilly
Rick Gunyon, Eli Lilly
Nick Andreopoulos, Pfizer
Chris Beltz, Pfizer
Great Lakes Chapter Board and Directors
Ron Dunn, GLC Manager