Jesper Nielsen
WHEN DISASTER STRIKES – INTR0
“Procedures and safeguards developed after 29 July 2006, when CERN suffered a total
black-out for the first time in its history”
A few words about CERN
The CERN Control Centre
Technical Infrastructure operation
CERN's electrical distribution
29/7/06: 18kV breaker explosion – blackout!
Where we were at – What really happened
Safeguards and procedures implemented post 2006
Future, new crisis management team for events where existing
procedures are not applicable
WAO – 5th August 2012
•
•
•
•
•
•
•
•
Jesper Nielsen BE-OP-TI
OUTLINE
Founded in 1954 by 12 countries
Today: 20 member states
About 10’000 users from all over the world
Annual budget: approximately 1 Billion US $
2400 permanent staff – 1000 fellows/students
WAO – 5th August 2012
•
•
•
•
•
Jesper Nielsen BE-OP-TI
CERN
Jesper Nielsen BE-OP-TI
WAO – 5th August 2012
ALL CONTROL ROOMS SEPERATED
PS
PSB
SPS
LHC
WAO – 5th August 2012
TI
Jesper Nielsen BE-OP-TI
ALL CONTROL ROOMS IN ONE
CRYO
Jesper Nielsen BE-OP-TI
WAO – 5th August 2012
CERN CONTROL CENTER
400.000V French network
WAO – 5th August 2012
3 DIFFERENT SOURCES OF ELECTRICITY
Diesels
130.000V Swiss network
Jesper Nielsen BE-OP-TI
Auto Transfer System
Jesper Nielsen BE-OP-TI
WAO – 5th August 2012
THE SOURCE OF THE PROBLEM!
Jesper Nielsen BE-OP-TI
WAO – 5th August 2012
18KV SHORT CIRCUIT
Jesper Nielsen BE-OP-TI
WAO – 5th August 2012
DIESELS
Imagine a control room without light, phones or PCs!
Specialists called via private phones collected during the
event
WAO – 5th August 2012
WHAT HAPPENED!
Jesper Nielsen BE-OP-TI
Electricians sent to places most likely to have failed
WAO – 5th August 2012
Jesper Nielsen BE-OP-TI
LESSONS LEARNED – EL NETWORK – POWER FOR CCC
WAO – 5th August 2012
Jesper Nielsen BE-OP-TI
LESSONS LEARNED – EL NETWORK – POWER FOR CCC
Jesper Nielsen BE-OP-TI
WAO – 5th August 2012
LESSONS LEARNED – PHONES
French landline
Jesper Nielsen BE-OP-TI
WAO – 5th August 2012
LESSONS LEARNED – PHONES
WAO – 5th August 2012
Simple is often the solution!
Jesper Nielsen BE-OP-TI
LESSONS LEARNED - GSM – OTHER PROVIDERS
A.#Vérifier#l'état#de#la#CCC,#demander#quelqu'un#de#vérifier#la#CCR
Vérifier:
La#configuration#d'alimentation#de#la#SE0
OK
NOK
WAO – 5th August 2012
LESSONS LEARNED – CHECKLIST
ESD1/0E sous tension
EOD1/0E sous tension par le biais des trois onduleurs (
ESS%/0E )
EOD103/0E EOD104/0E EOD105/0E OK
Vérifier l'alimentation des armoires électrique en
CCC (Diris ou voyant sur les portes) :
EOD3/CR : alimentation en secouru des ordinateurs et des
écrans en CCC
EOD2/CR et EOD1/CR : Alimentation des différents racks en
CCR et Star Point
EBD6/CR : éclairage, prise électrique mural et des 16 écrans
plats
EBD5/CR : alimentation des différents équipements sur les
îlots n'étant pas alimenté par le réseau secouru, alimentation
de EBD4/CR
EOD9/CR: Présence 3* 400 V / 50HZ
EBD3/CR : alimentation éclairage, prise divers,WC du 1er
étage
EBD2/CR : alimentation générale de la cuisine
EBD4/CR : alimentation EOD9/CR, EBD1/CR, EBD2/CR,
EBD3/CR, EBD6/CR
EBD1/CR : alimentation éclairage et prise mural
général RDC, contrôle d'accès
Actions
Appeler le piquet électrique pour l'informer de la situation en cas de problèmes
Jesper Nielsen BE-OP-TI
Localisation des armoires :
1 sensor on fault: Alarm in CCC + Firemen
2 sensors on fault: Alarm in CCC + Firemen + Evacuation
WAO – 5th August 2012
Jesper Nielsen BE-OP-TI
LESSONS LEARNED – OXYGEN DEFICIENCY HAZARD
A team of 10 persons is called in when a crisis occur
• Take decisions
• Make statements to press
• Handle priorities
WAO – 5th August 2012
LESSONS LEARNED – CRISIS TEAM
Jesper Nielsen BE-OP-TI
Crisis rooms in two locations
• France
- Cern Control Center
• Switzerland
- Firemen
I've introduced CERN, the CCC and TI operation
Explained how CERN is powered
Showed what happened on July 29 2006
Listed a number of important improvements made
• are
Communications
now reliable
We
better prepared
should disaster strike again!
• Electricity supply being renovated and made redundant
• Priority lists and checklists established to be prepared
• A crisis management team being set up to handle very big
events
WAO – 5th August 2012
•
•
•
•
Jesper Nielsen BE-OP-TI
Thanks for YOUR attention!