Status of the Linac Controls
Upgrade
Tom Himel
June 8, 2009
Status of the Linac Controls Upgrade
LCLS FAC review
1
1
Tom Himel
thimel@slac.stanford.edu
Outline
Review of reasons for upgrade and plan
Progress
Problems
Summary
Status of the Linac Controls Upgrade
LCLS FAC review
2
2
Tom Himel
thimel@slac.stanford.edu
Reasons for the Upgrade
Through retirements and layoffs, have lost most
people who know how to maintain the SLC
control system.
Have decided to upgrade the SLC control
system to EPICS with a series of AIP projects.
This will reduce chances of long downtimes due to
some problem where expertise is lacking causing a
long MTTR.
Hardware is old. Upgrade is overdue.
SLC control system has many years of applications
being added and fine tuned. Main difficulty of the
upgrade is to not lose too much this functionality.
Status of the Linac Controls Upgrade
LCLS FAC review
3
3
Tom Himel
thimel@slac.stanford.edu
Phases
Phase 1 makes LCLS independent of the Alpha and
multibus micros
Replaces about 15 micros with VME crates, CPUs, and CAMAC
interface.
1 micro in each of last 10 sectors, few in BSY, few system micros
(e.g. MPG)
Main work is software
Phase II replaces the CAMAC with more modern
hardware
Main expense is hardware
Phase III-N propagates above upgrades to other regions
of the accelerator complex
Little planning here yet
There are different types of functionality and hardware modules
in other regions so not just simple replication
Status of the Linac Controls Upgrade
LCLS FAC review
4
4
Tom Himel
thimel@slac.stanford.edu
“Signed off” List of Tasks
The June 08 upgrade review
recommended we make a detailed list of
functionality and get it “signed off” by
operations, accelerator physicists, and
maintenance groups.
We have made such a list and gone over it
with above groups at a series of meetings.
With modifications made at those meeting,
it was approved, but no formal signatures.
Status of the Linac Controls Upgrade
LCLS FAC review
5
5
Tom Himel
thimel@slac.stanford.edu
“Signed off” List of Tasks
Total
Himel M&S cost labor
Short description
priority (K$)
(man-wk) % done resource
Set up test hardware in bldg 34
1.0
1
100 sheng
SAM driver
1.1
4
100 sheng
PDU driver
1.1
8
sheng
Vacuum pressure display
1.1
6
sonya
Linac temperature and water display 1.1
6
sonya
CamDmp
1.2
12
20 judy
Make EPICS DB from SCP DB
1.2
4
50 judy
Magnet control
1.2
8
30 kristi
Set up test hardware in LI17 and LI18 1.2
2
20 kristi
PIOP driver
1.2
16
20 sass
sub-booster interface driver
1.2
4
0 sass
timing control
1.2
20
sheng
DAC driver
1.2
4
100 sheng
Status of the Linac Controls Upgrade
LCLS FAC review
6
6
Tom Himel
thimel@slac.stanford.edu
AIPs for Upgrade
Linac dipole correctors to EPICS – 300k - ~approved
Linac new quadrupole PS + EPICS control – 800k ~approved
Linac BPMs to EPICS – 700k - approved
Linac micro -> IOC hardware conversion - ~495k –
approved long ago
Linac/BSY unique devices to EPICS – 200k – awaiting
better cost estimate
Linac software conversion – 2.03M - ~approved
Linac gate generator and master trigger generator –
139k - ~approved
RF controller upgrade R&D - 1008 k - ~approved
Status of the Linac Controls Upgrade
LCLS FAC review
7
7
Tom Himel
thimel@slac.stanford.edu
We DO have documents and Reviews
6/4/09 LI17/18 test station design review
5/5/09 Camcom design review
1/23/09 PIOP initialize and status check
design review
11/25/08 SAM and IDIM device support
design review
Status of the Linac Controls Upgrade
LCLS FAC review
8
8
Tom Himel
thimel@slac.stanford.edu
Hardware Progress
Have 2 crate test setup in lab
Expect to have IOCs to control LI17 and
18 by July 1, 2009 for testing purposes
Have chosen to delay installation of IOCs
for LCLS sectors until more software
development and testing is done so can
be sure we are getting the right stuff.
Status of the Linac Controls Upgrade
LCLS FAC review
9
9
Tom Himel
thimel@slac.stanford.edu
Progress
VME CAMAC board has had several bugs
exterminated. In state of no known bugs.
Still need to test output to multiple branches and
heavy loading
SAM, IDIM, and DAC device support are all
done.
PIOP image can be downloaded and booted and
fast status check works.
Indian problems nearly killed Bob.
Analog and digital status DBs will be mass
produced from data in old control system.
Status of the Linac Controls Upgrade
LCLS FAC review
10 10
Tom Himel
thimel@slac.stanford.edu
Progress
Got more hardware AIP money than originally
planned
Don’t have to develop device support for CAMAC
power supply controllers nor handle bulk+boost
supplies.
Have started R&D needed for phase II
replacement of RF hardware
Functional requirements partly written (some details
still need to be filled in)
Have a microTCA crate, ADC, DAC to test as a
possible implementation. Starting detailed ADC
testing. Has been delayed by lack of software person.
New hire will go here.
Status of the Linac Controls Upgrade
LCLS FAC review
11
11
Tom Himel
thimel@slac.stanford.edu
BSY
The BSY area has several micros each
which control some LCLS devices and
some devices for ESA, PEP, SLC.
As the last two are obsolete but ESA is
considering future runs, did not want to break
ESA by converting micros to IOCs and then
supporting only the LCLS devices
Few enough channels for LCLS that have
decided to just change all of them to EPICS
style hardware, eliminating the use of CAMAC
for LCLS in the BSY.
Status of the Linac Controls Upgrade
LCLS FAC review
12 12
Tom Himel
thimel@slac.stanford.edu
Slow getting people on board
As predicted by the June 08 review team, it has
been difficult to get people on board the upgrade
project
At that time, plan was to finish the project by Nov 08.
In fact, all work stopped after that review
People were promised first for Nov 08, then Jan 09.
For software, now have 2 ~80% people and 4 ~40%
people. More are on the way.
Not complaining. LCLS had to be finished and there are
still many important improvements needed.
That review team was really sharp. Especially the
chair!!
With the approval of FACET and plans for LCLS
upgrades, controls is now hiring many people.
Status of the Linac Controls Upgrade
LCLS FAC review
13 13
Tom Himel
thimel@slac.stanford.edu
Schedule
There still isn’t a detailed integrated schedule
for this project
Task list (with durations) is being used for
planning. Not many dependencies
Need to get unique device hardware installed
during the Jan 10 downtime.
No other external deadlines.
(Un)Availability of manpower has been the
main schedule determinant
Good to leave obvious reason for the FAC to
beat me up
Status of the Linac Controls Upgrade
LCLS FAC review
14 14
Tom Himel
thimel@slac.stanford.edu
FACET
FACET is a newly approved $13M project using S019 and the damping rings for accelerator physics
experiments
Planned to be done Oct 2010.
Bad timing for controls (busy and in middle of
upgrade)
New stuff will be done in EPICS, old, left in legacy
control system
Will compete for manpower with advantage of a tight
deadline
Fortunately we are now hiring people.
Status of the Linac Controls Upgrade
LCLS FAC review
15 15
Tom Himel
thimel@slac.stanford.edu
Old Worries with status
1.
2.
As implementation of phase I will take longer than
originally planned and as expertise in the SLC control
system is needed until the rest of the linac, damping
rings, NLCTA, the gun lab, and the test lab are
converted, people experienced with the SLC control
system must be kept or new ones trained. - This is
being done
If FACET proceeds on its schedule of completion in Oct
2009, it clearly will not use the new control system and
that much more expertise in the SLC control system will
need to be maintained. If FACET proceeds on a slower
schedule, management will need to decide soon
whether a controls upgrade is done as part of the
project, or with operating or AIP funds before the
project, or if it will use the SLC control system. - chosen
Status of the Linac Controls Upgrade
LCLS FAC review
16 16
Tom Himel
thimel@slac.stanford.edu
Old Worries with status
3.
4.
This is a large, important project. Even so, many LCLS
needs discovered during the commissioning process
will be higher priority. It is better to do this project right
and, if necessary, slowly, than to rush it to meet a
schedule deadline and lose too much functionality
when it is turned on. Delays have happened, project is
being done slowly (and hopefully right).
There are 2 steering feedback in the region being
converted. If they need to run at 120 Hz, then the
EPICS version of 120 Hz feedback must be done
before the switchover. (Planned for Jan 2010.) - Still
planned for Jan 2010, so should be OK.
Status of the Linac Controls Upgrade
LCLS FAC review
17 17
Tom Himel
thimel@slac.stanford.edu
Should we go straight to phase II?
As phase I is going slowly and we are converting
considerable hardware away from CAMAC (BPMs and
magnet control), I (and others) periodically ask, should we
skip phase I and just replace all the CAMAC as soon as we
can. Pros and Cons
Would save the work of developing CAMAC device drivers and
support
Much of DB and most displays will remain unchanged when we move
from Phase I to II. Some will change and this is extra work.
Phase II will cost several $M and needs RF R&D to be completed
before it can start. Hence would be stuck with VMS and micros for 1-4
years longer
With phase I software done, other areas could be converted fairly
cheaply (BPMs and PSC-II and “unique” devices would need to get
new CAMAC drivers or new hardware).
Phase II will require significant downtime and commissioning to
implement. This will also lead to delays.
Status of the Linac Controls Upgrade
LCLS FAC review
18 18
Tom Himel
thimel@slac.stanford.edu
Summary
Upgrade is started, but slowly
Phase I is mostly planned, but a real
schedule needs to be developed
The AIP projects which are tentatively
approved should allow completion of
Phase I.
Status of the Linac Controls Upgrade
LCLS FAC review
19 19
Tom Himel
thimel@slac.stanford.edu