Avoiding Plant
Operating Errors
AT&T NP&E
April 10, 2012
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
POE Avoidance
• Overall trend for POEs is declining. However, we have had numerous vendor
caused POEs in 2011, which remains a concern with AT&T Senior Executives
• We expect our vendor partners to prevent POEs and achieve Zero POEs for the
year; protect AT&T’s network
• POEs affect our customers and may affect your bottom line
• POEs can be prevented using effective controls like documents and process
review before installation, detailed process compliance , second set of eyes
etc.
• In addition, we have made available a lessons learned document to help
prevent POEs. We hold our vendor partners accountable to use these lessons
and prevent future POEs
• We need our vendor partners to maintain sharp focus on POE elimination
2
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement.
WHY it’s more important than ever to avoid
Plant Operating Errors:
1. Consumers have become more reliant on our broadband and
mobility network.
2. The environment is more competitive than ever before.
3. Our network is handling more data on an hourly basis than ever,
making each outage more impactful.
4. Businesses choose AT&T because of our reliable network. Each
outage diminishes our value in the eyes of these businesses.
5. Today’s environment has us touching more parts of the network
simultaneously than ever before. Heightened awareness is critical.
3
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement.
Next Steps:
1. Cover this presentation with all employees and subcontractor
employees that are responsible for AT&T’s
Installation/Engineering/Quality work (CO's, POPs/Nodes, MTSOs,
Mobility Data Centers )
2. TALK about successes and failures in future team meetings.
• EVERY meeting should have some topic around this subject.
3. Ensure all employees in your organization complete the 2 hours
web based ASK YOURSELF course (#51247894) by June 1
4. Post the attached “POE Avoidance” placard as appropriate in the
work environment
4
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement.
Plant Operating Error
Avoidance Resources
5
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement.
Ask Yourself Principles
1. Do I have the proper ID and appropriate building access
permissions for the environment I am about to enter?
2. Do I know why I’m doing this work?
3. Have I identified and notified everybody - customers and
internal groups - who will be directly affected by this work?
4. Can I prevent or control service interruption?
5. Is this the right time to do this work?
6. Am I trained and qualified to do this work?
7. Are the work orders, MOPs, and supporting documentation
current and error-free?
8. Do I have everything I need to quickly and safely restore
service if something goes wrong?
9. Have I walked through the procedure?
10. Have I made sure the procedure includes proper closure
including obtaining clearance and release for the
appropriate work center?
6
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
7
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
MOP Best Practices Summary
Minimizing Impacts and POE for Tech Insertion and Change
Development of Methods of Procedure (MOP) for technology insertion and change


Driven from Deployment Guidelines and Gold Standards (Engineering rules and parameters).
Uses a standard template, including prerequisites, pre-checks, execution, post-checks, validation, and service assurance.
MOP Certification

Vetting of MOPs in a Production Support Lab (PSL) or approved peer review of paper documentation where no PSL exists, prior to
field use to minimize MOP defects
MOP Risk Assessment (RA) and review



Quantifies MOP impact risk via numerical scoring.
Enforces MOP review and approval to appropriate teams and management levels.
Enforces an End-to-End view on network and service impacts, and inter-element dependencies.
Execution of the MOP by the appropriate Work Center (e.g. NRC, PCO, etc.)
Validation of successful technology insertion or change and post-change service assurance
8
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement.
Procedural Error Reduction Best Practices
Fully implement the “Ask Yourself” program, including formal training.
Use MOP Development, Certification, Risk Assessment, Approval, and Review best practice; formal MOP Best
Practices Training (# 60398555).
Take an End-to-End view on procedure impacts, including impacts on services, and understand inter-element
dependencies.
Complete Service Validation and Assurance Testing.
Utilize 3CP testing processes during planned activity.
Communications, Command, and Control Process
Coordinated testing with the NRCs (e.g., MNRC)
Fully implement training and certification as appropriate, including subcontractor employees training.
Ensure Accountability: Connect Procedural Errors to yearly Goal commitments and track improvement status with
action plans. Implement appropriate disciplinary/corrective action plans.
Review monthly Procedural Errors (OTD misses, Defects, Outages etc.) on a regular basis.
9
© 2012 AT&T Intellectual Property. All rights reserved. AT&T and the AT&T logo are trademarks of AT&T Intellectual Property.
AT&T Proprietary (Internal Use Only). Not for use or disclosure outside the AT&T companies except under written agreement.