NEH Safety Systems
Enzo Carrone
June 30th , 2009
NEH Safety Systems
NEH ARR 2009
1
1
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
PPS and BCS Status
Both systems are currently ready for safe
operation to the FEE
PPS successfully certified (IAT) on June 24-26;
BCS components certified on June 26.
Hutch systems
HPS will be ready by the end of July.
NEH Safety Systems
NEH ARR 2009
2
2
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Safety Systems Section
PPS and BCS Systems combined into a single
Section.
Full-time QC Group Leader (M. Saleski).
Manages Review Process.
Oversees Engineering Change Order Process.
Full-time Documentation Manager (S. Starner).
Documentation system compliant to DOE Order
414.1C and ISO 9001-2000.
NEH Safety Systems
NEH ARR 2009
3
3
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
NEH Safety Systems
NEH ARR 2009
4
4
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Safety Systems Section Documentation Structure
NEH Safety Systems
NEH ARR 2009
5
5
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Management of Change Plan
CD-SS-MAN-00-01-02
Design Review Plan
CD-SS-MAN-00-01-03
Document Management Plan
CD-SS-MAN-00-02-02
Engineering Change Order
CD-SS-MAN-00-02-07
Software Configuration Management
CD-SS-MAN-00-02-01
SLAC Guidelines for Operations
SLAC-I-010-00100-000
NEH Safety Systems
NEH ARR 2009
6
6
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
NEH Safety Systems
NEH ARR 2009
7
7
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Safety Systems QA
Subject to a new Engineering Change Order (ECO)
process
Better tracking of requirements, specifications, reviews (peer and
formal), and system documentation.
New project QA approach
Bench and pre-IAT field test of system software
Acceptance tests performed for complex SLAC-built assemblies
(such at stoppers)
Cold and energized checkout of installed hardware
Und Complex lessons learned with subcontractors
Formal IAT reviews with stakeholders
Overall, better management of schedule
Indirectly affects quality
NEH Safety Systems
NEH ARR 2009
8
8
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Documentation Control
A single on line repository is available for all
released documents.
Formal Documentation Control is in place.
Measures taken to regulate the Preparation,
Change, Withdrawal, Review, Approval, Release,
Distribution, Access, Availability, Storage,
Disposal of documents.
Document management Plan, Document Change
Control Procedure, design Review Plan, Drawing
Management Procedure, Engineering Change
Order Procedure.
NEH Safety Systems
NEH ARR 2009
9
9
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Configuration Control Elements
Prevention of Unintended Change
Physical Security of System
Labeling
Training
Control of Intended Change
Work Planning (adequate review of design)
Work Authorization (RSWCF)
Verification of Work (RSWCF)
Periodic Confirmation of System Integrity
Routine testing and inspections
NEH Safety Systems
NEH ARR 2009
10 10
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Configuration Management Documentation
Document Management Plan
Document Change Control Procedure
Document Change Order
Design Review Plan
Software Configuration Management
Engineering Change Order Procedure
Engineering Change Order
Drawing Management Procedure
NEH Safety Systems
NEH ARR 2009
11
11
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Software Configuration Management Support
Software versions are checked during annual
certification.
Written procedures exist for extracting PPS
code from CVS and for uploading it to
PLCs.
A documented training program tracks
personnel PLC qualifications in the Section.
NEH Safety Systems
NEH ARR 2009
12 12
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Software Configuration Management Procedure
PPS Software is stored
in a dedicated PPS
repository.
Released software
always has “N.0.0”
version tag.
Documented software
bench testing is
performed prior to
deployment.
NEH Safety Systems
NEH ARR 2009
13 13
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Software Configuration Control Issues
Program Security:
All communication to the safety-critical PLCs is
through TCP/IP to ‘buffer’ Allen-Bradley PLC, then via
DeviceNet (serial data communication).
Safety-critical program ‘smart card’ cannot be written
to while in the PLC
Communication from the safety-critical PLCs is
through DeviceNet to ‘buffer’ Allen-Bradley PLC and
output to control system via TCP/IP
Network Access Security:
Hardwire Enable from MCC required
Only specific IP addresses are allowed to issue PPS
commands
Physical Access Security:
PLCs and DeviceNet are inside locked racks.
NEH Safety Systems
NEH ARR 2009
14 14
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Software Version Management
Version Management:
Software versions are checked during annual certification
Written procedures Exist for extracting PPS code from CVS and
uploading it to PLCs
A documented training program tracks personnel PLC
qualifications in the Section
NEH Safety Systems
NEH ARR 2009
15 15
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Implementation, Operations, and Maintenance
Initiate RSWCF
Implement Change
Initial Acceptance Test
Development
and Review
Cycle
Problems
Success
Close RSWCF
12 Months
Safety Assurance Test
6 Months
Interlock Checks
Problems
Success
Problems
Success
Routine Testing
Per Guideline 27
System in Operation
Problems
Assessment of Failure
Correct the Procedure
Is the Failure Reportable?
Procedure
Error
Undesired
Functionality
Discovered
Initiate RSWCF;
Determine Tests
Assess Failure with RSO
Administrative
Mitigation
Failed
Hardware
Engineering
Change
Repair Hardware
Re-perform Test
Problems
Success
Need for New
Functional
Requirements
NEH Safety Systems
NEH ARR 2009
Close RSWCF
16 16
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Proposed PLC PPS ‘Dev and Rev’
Need for New PPS
System
Safety Functions
Requirements
Specification
Validation Scope and
Methodology
Determination
Software Functions
Determination
Rework Proposal
Hardware Functions
Determination
Preliminary Design
Review
(Project and RSO/RSC)
Success
Withdraw Software
from Version-Control
Repository
Software Design and
Development
Safety Validation
Planning
Rework
Software
Hardware Design and
Development
Software
Bench Testing
Bench Testing
Specified?
Success
Deposit Software in
Version-Control
Repository
Assign New Version
Number
Rework
Procedure
Validation
Procedure Review
Rework Software
Rework Hardware
(Project and RSO/RSC)
Success
Success
Lifecycle Special Functions Key
NEH Safety Systems
NEH ARR 2009
System Technical
Design Review
System Review
or Assessment
System in
Operation
System Testing
or Validation
Additional
Cycle
Implementation,
Operations, and
Maintenance
Cycle
17 17
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Established SSS ECO
Start
Memo to Engineering
Group Leader initiating
change request
NO
Notification to
ADSO/RSO of
job, risk
assessment,
and review plan
Proposal
Accepted
Drop
Request
Yes
RSO/RSC Review
Assign Engineer
ECO Initiated
Scope Defined
Formal
Requirements
Produced
Action Item
Management
System
Engineer/Design
Work
Implement Action
Items
Weekly PeerReview ‘as needed,’
including Risk
Assessment
NO
Emphasis on review and
development documentation;
Enables a project
management approach;
Collects development and
review docs for auditability.
Action Item Closeout Memo Issued
RSWCF
(and WAF if applicable)
Ready for
Formal
Review?
Work is Performed
Yes
Assessment of work
performed during
weekly meetings
Conduct Formal
Review
Yes
Approve
NEH Safety Systems
NEH ARR 2009
NO
Evaluation by
Controls
Department
Management
18 18
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
FEE/NEH PPS Checkouts
Cold Checkout
Verifies Hardware and Cable Plant is installed
and wired correctly
Verifies No Ground Shorts in the Cable Plant
EEIP Hardware and Racks
Hot Checkout
Verifies Functionality of Hardware, Devices,
and PLCs
Hi-Pot Cable Plant
NEH Safety Systems
NEH ARR 2009
19 19
Enzo Carrone
19
Enzo.Carrone@slac.stanford.edu
Review Status
Final Design Review #2 (April 22, 2009)
Initial Acceptance Test Review (internal
and external)
Starting this week.
NEH Safety Systems
NEH ARR 2009
20 20
Enzo Carrone
20
Enzo.Carrone@slac.stanford.edu
PPS Review Status for FEE & NEH
Major Review to Date
LCLS SCR: May 31st 2007 [Bong, Stefan, Horton, Hastings]
LCLS PDR: Feb 29th 2008 [Lessard, Tompkins, Schmerge,
Anthony]
LCLS FDR: Oct 22nd 2008 [Anthony, Mueller(LBNL), Lessard]
RSC: Oct 31st 2008
LCLS FDR2: Apr 22nd 2009 [Perry Anthony, Zoe Van
Hoover, Luc Lessard, Paul Miller, Robert Mueller, Mike Woods]
NEH Safety Systems
NEH ARR 2009
21 21
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
PPS FEE Schedule
NEH Safety Systems
NEH ARR 2009
22 22
Enzo Carrone
22
Enzo.Carrone@slac.stanford.edu
FEE/NEH Schedule
NEH Safety Systems
NEH ARR 2009
23 23
Enzo Carrone
Enzo.Carrone@slac.stanford.edu
Conclusions
PPS/BCS has a Quality Assurance model in place;
LCLS management is aware of the Safety Systems
process and supports it throughout the project lifecycle
(from conception to certification);
LCLS, Operations, Radiation Physics are heavily
involved in all review phases from early on (from
requirements/specifications to user manual).
NEH Safety Systems
NEH ARR 2009
24 24
Enzo Carrone
Enzo.Carrone@slac.stanford.edu