SNS Project Status
Ernest L. Williams Jr.
27 April 2005
EPICS Collaboration Meeting @ SLAC
ICS – Software Engineering Group
1
Outline

Installation Status
» Global Systems
–
–
–
–
»
»
»
»
Network
Timing
MPS
CCR
Accelerator Systems
Cryogenic Controls
PPS
Target Controls

Operational Improvements

Issues

Summary
ICS – Software Engineering Group
2
The Spallation Neutron Source

The SNS construction project will conclude in 2006

At 1.4 MW it will be ~8x ISIS, the world’s leading pulsed spallation source

An upgrade to 3MW has been approved for CD0 planning.
ICS – Software Engineering Group
3
Communication Infrastructure is nearly complete

CD4 Fiber Backbone is
complete

Tubes installed for
future addition of
“blown” fiber

“Global” cables
nearing completion
Fiber Backbone
» Ethernet
» Timing
» MPS

All accelerator cable
SROs entered

CISCO Switch firmware
bug found (and fixed)
ICS – Software Engineering Group
Cable Infrastructure
4
Timing System Installation is Keeping Up

All IOCs (including NADS)

Multiple Fixed Rep Rates for RF
Systems

CLO Test Timing Network
» Infrastructure for Diagnostics,
LLRF, XFD, and Power supply
groups

Line Synch monitor
» Delivered from BNL
» Setting up test network for XFD
Chopper testing

“General Time” added to EPICS
» Provides best time from multiple
sources
» Assures monotonicity
» Accepted into “EPICS Core”
ICS – Software Engineering Group
5

SNS Event Link Monitor
» Event statistics
» Actual Events vs.
Scheduled Events
Number of MPS Inputs has nearly Doubled
(170 → 313)
Linac Only
MPS Inputs
Commissioning - no correctors After CD4 - correctors added
MEBT_BS CCL_BS
L_Dmp MEBT_BS CCL_BS
L_Dmp
IBCM
1
BLM / ND
66
129
66
129
Beam
Stop / Dump
Intercepting
diagnostics
2
4
6
2
4
6
11
27
6
11
27
6
Power Supplies
11
15
52
11
61
90
Vacuum
3
8
10
3
8
10
RF Systems
11
26
104
11
26
104
MPS inter connects
DBCM
9
4
6
9
4
6
6
Totals
47
150
313
48
202
351
170

All linac MPS hardware installed – includes 27 new chassis

Copper phased out, using fiber everywhere

After CD4, corrector PS status will be added
ICS – Software Engineering Group
6
Operations have moved to the CCR
(Central Control Room in CLO)









Move took place in February as
scheduled
Two console arcs are complete
(ahead of schedule)
PPS and ODH systems moved
(That was the worrisome part)
Software Development Area in use
All work planned for FY05 is
complete
Servers are moved to “CER”
(Central Equipment Room)
TPS also now in the CCR
Prototype Console moved to
Target Control Room
Upgraded all multi-head OPIs
»
»
»

Faster processor
More RAM
Faster graphics adapter (Matrox
G450)
X-windows and Java Apps have a
much better performance
ICS – Software Engineering Group
7
PPS Equipment
Linac Controls Installation is
Keeping Up with the Project Schedule

We have kept up (barely!!) with RF testing, RF
Conditioning, PS testing, cryomodule cool-down,
etc.

Medium-Beta SCL controls mostly complete
»
»
»

Cryomodule MB1 controls testing remains
Some “warm section” vacuum controls testing remains
Some MPS hardware installation remains
High-Beta SCL controls approaching completion
»
»
RF, Cryo, & Vacuum controls checkout are well along
Magnet power supply interface checkout has been
integrated with overall power supply checkout

HEBT controls installation in progress

Automated procedures have helped keep schedule
ICS – Software Engineering Group
8
Linac LLRF Installation is Keeping Up
(This is the biggest task in terms of numbers and complexity)



Number of installed LLRF IOCs: 46 (up from 12) for 80 systems
(out of a total of 96 systems)
Configuration automated via scripts, avoiding inconsistencies
between SCL systems
From last ASAC: "Add Adaptive feed forward to firmware if
required"
» Was required for higher beam current
» Not foreseen in current FCM, so successfully added to IOC software
–

recently refined to eliminate parameters that were hard to tune
Automation of RF ramp-up, warmup, loop closure
» Allowed hands-off recovery from trips on the NC Linac
» Needs adaptation for SCL

Waveform updates synchronized to timing system
» Allows physics apps to compare data from several systems for the
same beam pulse

Refining overview screens, giving better notice of trip reasons,
active overrides, etc.
ICS – Software Engineering Group
9
Cryogenic Control System nearly complete!!

All CHL Controls equipment installed tested
and

All IOC and PLC programming
complete
» CHL Gas Management, Purifier and
Warm Compressor
» CHL 2.1K and 4.5K Cold Boxes
» Medium and High Beta Cryomodules
» HEBT Service Building IOC and PLC


Cool Down
2K Cold Box Turn-on was automated
Full RF Power
First Liquid
Cryomodules
will complete in April
Warm
Up
ICS – Software Engineering Group
10
The Cryogenic Control System layout
2 Servers
4 OPIs
11 IOCs
17 PLCs
ICS – Software Engineering Group
11
Ring Controls is ahead of schedule:
BNL effort completed in March

Handover Plan - Complete and in use (cooling water and ramping software scope
moved to ORNL)

FSDs - Complete, in use and being updated as needed to keep them as-built

Design - Essentially complete. All drawings and specs needed are in use for
fabrication and installation and being updated as needed to keep them as-built

Hardware Fabrication - WBS 1.9 work complete except for installing 1 PLC for cooling
water skid. WBS 1.5 work complete except for 2 motion control racks, 2 PLCs for RF,
and Vacuum racks

Procurement - All hardware procured except 1 PLC for water skid and some
miscellaneous parts

Cabling - Schedule for Ring and RTBT cables depends on budget allocations
»
»
HEBT Communications cables pulled, being terminated
HEBT Process control cabling in process

Software - Vac, BLM, Mag PSC/PSI, HPRF, Beam Dump TCs ready for handover. All
software ready for field testing by Jun 05

Test Plans - Drafted for all systems except RF - Will be updated as needed during
installation
ICS – Software Engineering Group
12
PPS System Installation is keeping up
(in spite of many configuration changes)

75% of the PPS controlled devices in the Linac have been
connected

The temporary Linac shielding has been removed and the
Linac has been returned to one segment for phase 1.3

PPS remote I/O racks for the HEBT are installed

Conduit & cable for PPS devices installed in HEBT tunnel

Installation of conduit & cable for PPS devices in Ring
tunnel is 100% complete

The PPS control room racks have been relocated to the
CLO CCR as a part of phase 1.3 (SCL commissioning)
ICS – Software Engineering Group
13
The Target & Instrument PPS are also
the responsibility of the Controls Team

Cable & conduit design for target building
complete, installation underway using AIMSI
crafts (beamline 2 and transfer service bay) and
fixed price subcontractor (high bay)

Target PPS equipment design is 85% complete

Target PPS equipment fabrication is 70%
complete

Equipment installation ramping up as electricians
become available

Instrument PPS design for beamline 2 underway.
Prototype user panel being fabricated
ICS – Software Engineering Group
14
The number of IOCs has risen dramatically

223 IOCs currently on Accelerator Network:
» 109 VME / VXI --- vxWorks-based
» 2 Linux-based “Soft IOC” Servers
– Running 24 Soft IOCs (18 + 6)
» 90 Intel-Pentium --- Win32-based IOCs (the “NADS”)
– Wirescanners, Faraday Cups, BPMs, BCMs

Upgraded control system to EPICS R3.14.7
» Improved reliability in channel access.
– Results in fewer white screens in the control room.

Upgraded OS for VME/VXI-based IOCs
» vxWorks 5.5.1 (SNS CVS Release SNS07)
– Better network driver and network stack configuration
– Auto-retry on boot-up if boot server is unavailable.
– Better memory management
ICS – Software Engineering Group
15
Power Supply Group is Routinely Using
Automated Testing and Calibration

Method uses EPICS and so
assures “end-to-end”
testing including control
system

Resulted in large schedule
reduction – one test instead
of two

PS Calibration, current/field
mapping and cycling
profiles come from Oracle
RDB

9 SCL IOCs installed, 8
operational

1 HEBT IOC installed and
operational
ICS – Software Engineering Group
16
Turn-on and Turn-off of HPRF Systems
has been automated
State Diagram
monitors the process
for each HVCM
Overall summary is
displayed
ICS – Software Engineering Group
17
The New Archive Engine is stable and a
many-featured Java Archive Viewer is now in use

Archives now well organized with central configuration file and scripts for
»
»
»
»

generation of directory structure and skeleton files
start/stop/restart/monitoring of sampling engines
update of status web page
email notification of engine status and disk usage
Currently 11 distinct archives each running one or more sampling engines
(total: 25).
»
(Conventional Facilities, Magnets, RF, Cryo, ...)

Data rate kept to ~2GB/day with occasional peaks at 4GB/day.
Limiting performance factor is currently neither CPU nor network but disk I/O.

New Java Viewer is stable and many features are being exploited

»
»
»
»
»
»
»
Uses XAL Framework
Web interface available
Plots actual data in various formats
Offers absolute and relative request times
Performs computations, exports data to spreadsheets
Investigating new uses via command line and web
Scroll, zoom, legends, titles, multiple plot styles, arithmetic and boolean operations
ICS – Software Engineering Group
18
The new Archive Viewer (Chevtsov)
has many useful features
•Relative start/end times "-5 d(ays)" → "now"
•Multiple value axes
•Tool-tip info for individual data points
•XAL Framework
Support for calculations based on archived data
Export to spreadsheets, progress bar
Green squares in plot launch tool-tips for meta
data, in this case daily restart of sampling
engine.
ICS – Software Engineering Group
19
Database Collaboration with APS has grown
(SNS, APS, SLAC, TRIUMF, BESSY, FNAL, Diamond, PSI)
•
IRMIS: Integrated Relational Model for
Installed Systems (v1.0 release in April)
•
Agreed “Core Schema” describes
control system with three hierarchies:
•
•
•
•
Signal
Housing
Power
Uses the SNS Physics XAL “Framework”
•
Pelaia et al
•
RDB synchronized using “crawlers”
•
106 SNS IOCs successfully “crawled”
•
•
EPICS versions 3.13.9, 3.14.6, 3.14.7
228,270 PVs found
–
–
–
–
–
•
SCL
112141
DTL
33607
CCL
27443
MEBT 10918
HEBT
8074
“Crawlers” in development for all EPICS
clients, IOCs, edm, Archiver, Alarms, etc.
ICS – Software Engineering Group
20
Issues

Increased installation pace.

RF Processing and installation checkout means
Control Systems Group must provide operational
support.

Hardware/Software Configuration Management is
highly correlated with reliability/availability. Work
is in progress using a RDB, CVS, and good
release control scheme.
ICS – Software Engineering Group
21
Issues: How to manage increasing number of IOCS?

223 IOCs currently on Accelerator Network:
» 109 VME / VXI --- vxWorks-based
» 2 Linux-based “Soft IOC” Servers
– Running 24 Soft IOCs (18 + 6)
» 90 Intel-Pentium --- Win32-based IOCs (the “NADS”)
– Wirescanners, Faraday Cups, BPMs, BCMs

Upgraded control system to EPICS R3.14.7
» Improved reliability in channel access.
– Results in fewer white screens in the control room.

Upgraded OS for VME/VXI-based IOCs
» vxWorks 5.5.1 (SNS CVS Release SNS07)
– Better network driver and network stack configuration
– Auto-retry on boot-up if boot server is unavailable.
– Better memory management
ICS – Software Engineering Group
22
Issues: How to manage increasing number of IOCS?
ICS – Software Engineering Group
23
Issues: How to manage increasing number of IOCS?
ICS – Software Engineering Group
24
Summary

Installation is keeping pace
» Required use of more automated configuration and
testing, database use
» PPS keeps pace notwithstanding changes
» Cryo controls nearing completion
» Will be ready for SCL commissioning
» Ready to pick up on Ring controls

Operational Improvements as well
» Automated HPRF turn-on
» Automated 2K Cold Box turn-on
» Improved Archive viewing tools
ICS – Software Engineering Group
25