AMOS/LUSI Controls and Data Acquisition
LCLS eXperimental End Stations (XES)
Robert C. Sass
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
1
Robert C. Sass
XES Slow Controls
No major problems forseen
AMOS has ~ 100 movers
Ultra high vacuum, power supplies etc.
Some requiring special consideration are:
Beam inhibit next pulse from the XES via MPS
Use chopper to rate limit
Synch gas jet & Laser to beam pulse
Synch mechanical chopper to the beam
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
2
Robert C. Sass
Experimental Data
Generation, storage, retrieval and analysis of experimental data is the
“product” of the LCLS.
LSST, if funded, will produce ~30 TB of data per night.
The AMOS experiment may eventually take data @120Hz from:
6 spectrometers @~15 KB.
5 CCDs @1 MB each
That’s ~700 MB/second or 2.4 TB/hour or ~58 TB/24 hour running
period.
Larger CCD densities are already planned.
LCLS experiments are relatively short term and so will follow the
technology more quickly than LSST.
These numbers have nowhere to go but up.
With the possible exception of the LHC or until the ILC
comes along, LCLS will be the world driver for Data
Acquisition, Storage, Retrieval and Analysis.
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
3
Robert C. Sass
How Will We Handle This?
Build on previous SLAC experience.
Initially use commercial digitizer and COTS CPU for AMOS spectrometer data.
Only one vendor found that makes an 8GHz digitizer.
Use scalable technology developed for LSST for CCD data.
Capitalize on years of BaBar experience with hierarchical storage and
management of HEP data.
Many aspects of traditional HEP computing and data management are applicable to
XES/LUSI
Large parts of the BaBar data management system have been developed at SLAC
Significant expertise for this type of system exists at SLAC.
BaBar Stores:
~1 TB/day raw data.
>>1 TB/day derived data products.
~1.5 PB total Babar data. Old data purged.
Extend/integrate several SLAC-developed and EPICS Java-based technologies
for data retrieval and analysis.
Java Analysis Studio (JAS)
Accelerator Integrated Data Access (AIDA)
EPICS Archive Viewer
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
4
Robert C. Sass
eXperimental End Station System Overview
Channel
Access
MPG/LCLS
Event
Generator
MCC Main Control Room
Linac & LCLS
Control GUI
Anywhere
120 Hz Archive
Engine
Event
Receiver
SCCS
Linac/LCLS
Injector
120Hz IOC
Machine Protection (MPS)
Linac
120 Hz Beam Line Data
LCLS
Analysis /
Viewing
Undulator XTOD
X-Ray
Beam
EPICS Archive
SLAC WAN
XES Hutch
Experimenter
XES
private
subnet
EELOG
DAQ
Config
LCLS Data
Retrieval
Fast
Instrument
Archive
Other
Databases
Slow
Instrument
Archive
DAQ
Archive
SLAC WAN
Hutch IOCs &
Instruments
Visual Data Monitor
MPS
Ethernet
Fast DAQ
Data
EELOG
Detector
Instrument(s)
XES Controller
Digitized
Data
EVR Hardware Triggers
Beam Line
Processor
Channel Access
Gateway
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
120 Hz
Detector
Control
Beam Line & Timestamp Data (dedicated Ethernet)
MPS
Experiment
Specific IOCs &
Instruments
5
Data
Acquisition
Subsystem(s)
Config/Status
X-Ray Transport to
Diagnostics & Next Hutch
Robert C. Sass
XES
Data
Cache
XES Detail – Data Acquisition
1. Channel Access (Ethernet)
2. Beam Line 120 Hz Data
4. MPS (Reflective Memory Fiber)
5. SLAC WAN (Ethernet)
3. EVR (Fiber)
6. DAQ Data to SCCS
XES Hutch
XES
private
subnet
EELOG
2 Detector Types
2 DAQ Types
Spectrometer
SBC DAQ
XES Controller
PC
5
1 Gb E
D
A
Non-RTOS
TOF/Momentum
Instrument
1 Gb E
cPCI ADC
G
1 Gb E
Visual Data Monitor
PC
RTOS
Spectrometer Monitor
EPICS Config PVs
Non-RTOS
1 Gb E
1 Gb E
1 Gb E
CCD Pixel
Detector
Beam Line
Processor
SBC
4
MPS PMC
EVR VME
2
Beam Line
Data PMC
1
RTOS
PPC
Front End Board
1 Gb E
1 Gb
E
Channel Access
Gateway
A. EPICS & Local Control (Hutch Subnet)
D. ADC Control & Digitized Data
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
RTOS
E
Fast Serial
FPGA
G
1 Gb E
3
B
CE Module DAQ
XES
Data
Cache
F
CCD Monitor
EPICS Config PVs
C
1 Gb E
10 Gb Enet
Experiment Chamber
B. Distributed EVR Hardware Triggers
C. Beam Line & Timestamp Data (dedicated Enet)
E. Detector Control & Digitized Data
F. DAQ Data to Cache
6
Robert C. Sass
G. Visual Monitor Data
6
Fabric clock
CE Module block diagram
Right side
MGT clock
Right side
PPC-405
(450 MHZ)
Right side
Memory
(512 Mbytes)
Micron RLDRAM II
FPGA
200 DSPs
Lots of gates
Xilinx XC4VFX60
Right side
Configuration
memory
128 Mbytes)
Samsung
K9F5608
Right side
Multi-Gigabit Transceivers
(MGT)
8 lanes
Left side
MFD
Reset options
JTAG
Left side
100-baseT
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
Reset
7
Robert C. Sass
Dual CE Module
PHYs (0-16)
May mix and
match lanes
to each CE
footprint:
~ 50 cm2
power:
~ 7 watts total +
~ 3/4 Watt/port
CE
CE
Cost ~1K
reset
JTAG reset
options
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
8
10 GE
10 GE 100B-T
Robert C. Sass
100B-T
CE Module Characteristics
Memory Subsystem
512 Mbytes of RAM
Sustains 8 Gbytes/sec
“Plug-In” DMA interface (PIC)
Designed as a set of IP cores
Designed to work in conjunction with MGT and protocol cores
Bootstrap loader (with up to 16 boot options and images)
Interface to configuration memory
Open Source R/T kernel (RTEMS)
10 GE Ethernet interface
100 base-T Ethernet interface
Full network stack
Utility software to manage I/O
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
9
Robert C. Sass
Archive Data Management System Overview
1. EPICS Channel Access
LCLS Operations
Anywhere on
Channel Access Networks
Slow Archive
Engines
Global Archive
Configuration
Management
9
Archive
Configuration
Interface
1
Anywhere on
SCRAMNet Network
2. DAQ Archive Data
3. Slow Archive Data
Fast Archive
Engine
4. Fast Archive Data
5. DAQ Archive Configuration
6. Slow Archive Configuration
4
3
7. Fast Archive Configuration
6
SCCS
Fast
Archives
Global Archive
Dictionary
Slow
Archive
Configs
Fast
Archive
Config
DAQ
Archive
Configs
Slow
Archives
8. Offsite/Cache Archive
Transfer
9. Global Archive
Dictionary Management
DAQ
Archives
Modeling &
Other Databases
8
7
Experiment
Operations
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
5
2
Archive
Configurations
Interface
Data
Acquisition
Module(s)
10
XES Hutch
XES DAQ
Data Cache
Robert C. Sass
Data Volume Makes DAQ Archive More Complicated
CE
10 GigE
SBC
1 GigE
Ethernet Network Switch
Local Cache Farm
Ethernet Network Switch
Links to Computer Center
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
10 GigE
11
Robert C. Sass
Archive Retrieval/Analysis System Overview
1. Global Archive Dictionary Read
Anywhere
2. Data Retrieval Request
3. Locate server & storage
1
4. Tape Data Request
Tape Data
LCLS Analysis /
Viewing
5. Database Data
6. Slow Archive Data
2
9
7. Fast Archive Data
8. DAQ Archive Data
LCLS Data
Retrieval
4
3
SCCS
Relational
Database
Server
Slow
Archives
Server
Fast
Archive
Server
DAQ
Archives
Server
5
6
Modeling &
Other Databases
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
Slow
Archives
7
Fast
Archives
8
Global Archive
Dictionaries
Data Retrieval
Temporary Cache
DAQ
Archives
12
Robert C. Sass
9. Results
Summary
Minor slow controls issues.
Data Acquisition and Archive are the major LCLS
challenges.
With CE for CCD and a commercial digitizer & SBC for
spectrometers we can scale the DAQ needs of LCLS.
With BaBar experience we can scale the data archive needs.
In the near future we must:
Specify and design an integrated data retrieval and analysis package
that can be used anywhere and is user extensible.
Specify and design a framework and tools for experimenters to use
in designing and building their instruments.
The CE provides a standard CCD interface
Existing EPICS tools could be extended to construct the experimentspecific EPICS database and screens.
April 16-17 2007
Facility Advisory Committee
XES Controls/DAQ
13
Robert C. Sass