LUSI
WBS 1.6 Controls and Data Systems
Overview – Plenary Session
G. Haller – Sub-System Manager
July 15, 2008
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 1
Gunther Haller
haller@slac.stanford.edu
Content
Scope
Cost & Schedule
WBS Organization
Cost
Schedule
Control & Data System Overview
Summary
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 2
Gunther Haller
haller@slac.stanford.edu
Experiment Controls and Data Acquisition
Near Experimental Hall
1
2
AMO
SXR
Part of LCLS
Proposed
External
Funds
Beam Transport
LCLS
LUSI
HEDS
SXR
X-ray Transport
Far Experimental Hall
3
4
XPP
XCS
5
6
CXI
HEDS
Proposed
External Funds
Separate WBS 1.6 to combine all LUSI control &
data needs due to commonality in requirements,
design, implementation, installation and
integration
XPP, CXI, XCS, Diagnostics & Common Optics
Common control and data systems design for
LUSI and rest of photon beam-line instruments
(AMOS, SXR)
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 3
Gunther Haller
haller@slac.stanford.edu
Scope – WBS 1.6 Control & Data Systems
Included in W.B.S. 1.6
All controls & DAQ, labor and M&S, for XPP, CXI, XCS instrument
components and diagnostics/common optics included in baseline
Includes controllers, racks, cables, switches, installation
Data-storage and processing for FEH
Initial offline (more effort will be on operating budget)
Input-signals to LCLS machine protection system link-node modules
Provided by LCLS X-Ray End Station controls (CAM is G. Haller)
Personnel protection system
Machine protection system (LCLS modules, fibers)
Laser safety system
Accelerator timing
Femto-second laser timing
Network architecture & security
Data-storage and processing for NEH
User safeguards
Laser controls
CXI 2D detector controls
Interfaces described in
1.1-517 ICD between XES and LUSI (released document)
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 4
Gunther Haller
haller@slac.stanford.edu
1.6 WBS to Level 4
1.6
Control & Data
Systems
1.6.1
Integration &
Management
1.6.2
Common
Controls
1.6.3
XPP
Example
XPP
1.6.4
CXI
1.6.5
XCS
1.6.6
Offline
Computing
1.6.3.1
Requirements,
Design, Setup
1.6.3.2
Standard Hutch
Controls
1.6.3.4
Specific
Controls
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 5
Gunther Haller
haller@slac.stanford.edu
W.B.S Structure
W.B.S. 1.6.2 Common Controls
Beam feedback
FEH data storage
Data processing
Racks & cables
W.B.S. 1.6.3 XPP, 1.6.4 CXI, 1.6.5 XCS
Requirements, design, setup
Standard hutch controls
Hutch cables, racks, installation
Workstations
Beamline processor
Channel access gateway
Machine protection system Interface
Specific controls
Controls & data for each section of instrument
Including vacuum, diagnostics, optics
W.B.S. 1.6.6 Offline Computing
Data-format, API, data-catalog, meta-data management, processing
framework, workflow, pipeline
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 6
Gunther Haller
haller@slac.stanford.edu
Control & Data Systems
Instruments
Controls & Data
Systems
Processing
2-D
Detector
Local
Storage
Software
RTEMS operating system
EPICS
Realtime DAQ programming (C/C++)
Offline
Hardware
Cabling, racks, trays
Controllers for
Pumps, gauges, motors, etc
Pumps
Configuration
& Run Control
Signal processing
Data acquisition and storage
Workstations, data servers, storage,
networking
Stages,
Motors
Intensity
Monitor
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 7
3 more “interesting” detectors &
electronics to control and readout
Controllers
CXI detector/ASIC
XPP/XCS detector/ASIC
8-GHz waveform sampler
Gunther Haller
haller@slac.stanford.edu
Cost Methodology: ESD’s
Basis for agreement on what components need to be
controlled and how
Detailed Engineering Specification Documents (ESD’s)
Two ESD’s for each instrument
Controls ESD
Describing devices to be controlled
E.g. motion, vacuum
EPICS processing to be performed
E.g scanning
Data Acquisition (DAQ) ESD
Describing devices to be read into DAQ
2-D detectors, waveform sampling, some 120-Hz cameras, etc
Online processing to be performed
Plus one ESD for diagnostics
Covers control and DAQ
Status: All ESD’s are approved and released
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 8
Gunther Haller
haller@slac.stanford.edu
Cost Methodology: ICD’s
Basis for agreement on who is responsible for what
and where the interface is:
Detailed Interface Control Documents (ICD’s) for each
instrument
Common Optics and Diagnostics components are covered in
the individual instrument ICD’s
Status: All ICD’s are approved and released
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 9
Gunther Haller
haller@slac.stanford.edu
ESD’s and ICD’s
All documents at
http://confluence.slac.stanford.edu/display/PCDS/LUSI+Document
+Page
XPP
SP-391-001-21 XPP Controls ESD
SP-391-001-22 XPP Controls & DAQ ICD
SP-391-001-23 XPP DAQ ESD
CXI
SP-391-001-13 CXI Controls ESD
SP-391-001-14 CXI Controls & DAQ ICD
SP-391-001-18 CXI DAQ ESD
XCS
SP-391-001-24 XCS Controls ESD
SP-391-001-25 XCS Controls & DAQ ICD
SP-391-001-26 XCS DAQ ESD
Diagnostics
SP-391-001-19 LUSI Common Diagnostics & Optics ESD
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 10
Gunther Haller
haller@slac.stanford.edu
Project Budget
Detailed bottoms-up cost estimate
Labor: number of hours listed for each task
All M&S itemized to the component level
Almost 100% supported by vendor quotes or recent purchase
orders
WBS 1.6
Resource Type
Control
Accounts
Work
Packages
WBS 1.1
6
12
$5,461,314
WBS 1.2
14
49
$5,942,486
WBS 1.3
11
45
$9,486,460
WBS 1.4
16
45
$7,715,265
WBS 1.5
10
39
$6,383,995
WBS 1.6 (G. Haller)
20
289
$7,135,691
WBS
Total BAC
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 11
Values
Value
Labor
$3,409,458
Non-Labor
$3,726,233
Total BAC
$7,135,691
$42,125,211
Gunther Haller
haller@slac.stanford.edu
Schedule
All tasks and materials (order, award, receive dates) in P3
1.6 is internally linked with predecessors and successors
“Available” mile-stones for each deliverable identified and
entered
Linked to instrument “Need” mile-stones
Have 90 days or more float
Resources leveled
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 12
Gunther Haller
haller@slac.stanford.edu
Milestones
XPP
XPP Controls PDR
CD-3A – XPP Instrument Start Construction
XPP Controls FDR
XPP Controls available
CD-4A – XPP Start Operation
Dec 08
Jul 09
Sept 09
Mar 10
Dec 10
CXI Controls PDR
CD-3B – CXI – Instrument Start Construction
CXI Controls FDR
CXI Controls available
CD-4B – CXI – Start Operation
Sep 09
Apr 10
Jun 10
Nov 10
Sep 11
XCS Controls PDR
CD-3C – XCS – Instrument Start Construction
XCS Controls FDR
XCS Controls available
CD-4C – XCS – Start Operation
Nov 09
Oct 10
Feb 11
Jul 11
Aug 12
CXI
XCS
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 13
Gunther Haller
haller@slac.stanford.edu
Data Sub-System
Difference to conventional X-Ray experiments
High peak rate & large volume comparable to high-energy physics
experiments such as BaBar @ SLAC
Data Rate/Volume of CXI Experiment
(comparable to other LUSI experiments)
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 14
LCLS Pulse Rep Rate (Hz)
120
Detector Size (Megapixel)
1.2
Intensity Depth (bit)
14
Success Rate (%)
30%
Ave. Data Rate (Gigabit/s)
0.6
Peak Data Rate (Gigabit/s)
1.9
Daily Duty Cycle (%)
50%
Accu. for 1 station (TB/day)
3.1
Gunther Haller
haller@slac.stanford.edu
Data Sub-System (con’t)
Challenge is to perform data-correction and image processing
while keeping up with continuous incoming data-streams
Tradeoff between tasks implemented online versus offline
Important to produce science output without piling up more
and more raw images
LUSI benefits that SLAC Particle Physics and Astro-Physics
group is involved which has substantial experience acquiring,
processing, and archiving large data volumes at high rates
Use common dataflow/processing/storage & offline interface
DAQ for instrument components in the real-time detector data
chain (BNL & Cornell 2-D detectors, future SXR detector,
waveform sampler, etc)
Minimizes development, production, commissioning, and
maintenance effort
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 15
Gunther Haller
haller@slac.stanford.edu
Data System Architecture
Instrument
specific
Photon Control Data Systems (PCDS)
Beam Line
Data
L1: Acquisition
Digitizers, Cameras,
2D Detectors
To SCCS
Offline
Timing
L0: Control
L2: Processing
L3: Data Cache
Level 0: Control
Level 2: Processing
Level 1: Acquisition
Level 3: Online Archiving
Run & configuration control
Run & telemetry monitoring
Image acquisition, calibration
Event-building with beam-line data
Correction using calibration constants
Data reduction (vetoing, compression)
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 16
Pattern recognition, sort, classify,
alignment, reconstruction
NEH/FEH local data-cache
Local cache can buffer up to 4-days
worth of data
Offline will transport data to tape staging
area in SCCS Computer Center
Gunther Haller
haller@slac.stanford.edu
Example 2D-Detector Control and DAQ
Chain
Beamline Instrument Detectors
Fiber
Cornell CXI 2D detector-ASIC
SLAC FPGA
front-end
board
Brookhaven XPP/XCS 2D
detector-ASIC
Main LUSI instrument custom
integrated circuits from Cornell
& Brookhaven are already
connected at SLAC to SLAC
LCLS high-performance DAQ
system
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 17
ATCA crate with SLAC
DAQ boards, e.g. the
SLAC Reconfigurable
Cluster Element Module
ATCA
Advanced Telecommunication Computing
Architecture
Based on backplane serial communication
fabric, 10-G E
2 SLAC custom boards (also used in other SLAC
experiments)
8 x 2.5 Gbit/sec links to detector modules
Dataflow and processing
Managed 24-port 10-G Ethernet switching
Essentially 480 Gbit/sec switch capacity
Naturally scalable
Gunther Haller
haller@slac.stanford.edu
Offline Data Management/Scientific Computing
Offline Data Management
Make LUSI data accessible for science
Appropriate format(s)
Appropriate storage/retrieval hardware and software
Offline will store the data in HDF5 files
Compatible with NeXus standard for X-ray, neutron and muon data
Written and released Interface Control Document
1.6-526 Online/Offline ICD
Scientific Computing for LUSI Science
Very dependent on the detailed nature of the science
Unprecedented size (for photon science) of data sets to be
analyzed
Unprecedented computational needs (for photon science)
Opportunities and needs being evaluated
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 18
Gunther Haller
haller@slac.stanford.edu
Core LUSI Team & Test Setup
Team from SLAC PPA experiment control and data
systems research engineering group
Particle & Astrophysics Experiments
Photon Beam-Line and Experiments
Have racks with many
components which will be
used by LUSI in the lab
Incrementally add
controllers and software
according to schedule
Schedule driven by budget
profile, not by availability of
manpower
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 19
Gunther Haller
haller@slac.stanford.edu
Summary
Technical and cost/schedule risks are low
Well documented agreements with instruments
Re-use of LCLS software, hardware where appropriate
Peak data rate/volume requirements are comparable to HEP
experiments, requiring matching data acquisition and
management system
Data subsystem concept & architecture are well developed
Use standard interface to all detectors
Use data management systems that have high bandwidth and are
scalable
Leverage significant expertise at SLAC in data acquisition and
management
Cost bottoms-up with detailed quotes for each component
Schedule fully linked and resource leveled
Ready to be approved for cost and schedule baseline
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 20
Gunther Haller
haller@slac.stanford.edu
END OF PRESENTATION
LUSI DOE Review Aug 19, 2008
WBS 1.6 Controls Plenary p. 21
Gunther Haller
haller@slac.stanford.edu