LUSI Physics Requirements
LCLS/LUSI Controls-CDR
04-10-2007
Yiping Feng
yfeng@slac.stanford.edu
Outline
LUSI Instruments & 1st Experiments
Coherent X-Ray Imaging (CXI)
X-Ray Pump-Probe (XPP)
X-Ray Photon Correlation Spectroscopy (PCS)
Requirements
Controls
Data Acquisition/Management
Timing
Applications
2
Location of Instruments
Coherent X-ray
Imaging
Instrument *
Near Experimental Hall
200 m
XPCS
transport
Tunnel
HEDS (outside
Funding)
X-Ray Photon
Correlation
Spectroscopy
Instrument
X-ray
Pump-probe
Instrument
SXR
Emission
Far Experimental Hall
LCLS AMO
Instrument
SXR
Imaging
Control and Data Systems for CXI,
XPP, PCS Instruments, PCS
Transport
3
Phased Construction
Phase-I – Mar. 2009
CXI and XPP co-located in Hutch-3 in NEH
Phase-ii – Mar. 2010
XPP in Hutch-3 in NEH
CXI in Hutch-5 in FEH
PCS in Hutch-4 in FEH
4
CXI Instrument
Molecule
injection
potential particle
orientation beam
To mass
spectrometer
Pixelated detector
(Cornell detector)
Intelligent
beam-stop
Optical & x-ray
diagnostics
Structures of Single Molecule
at near atomic resolution
Readout &
reconstruction
5
2D
Detector
Wavefront
sensor
1 micron
KB system
0.1 micron
KB system
Sample
chamber
LCLS
Beam
6
XPP Instrument
Time Resolved Scattering
at < ps time resolution
7
Laser System
(Fundamental)
Small Angle
Scattering
Wavelength
Conversion
X-ray
Diffractometer
Offset
Monochromator
8
PCS Instrument
Speckle Pattern
(Iron-Aluminum Alloy)
Dynamics of disordered systems
at < ns time & near atomic resolutions
9
PCS Diffractometer
10
PCS Transport
PCS vacuum
transport
PCS
monochromator
11
X-ray Optics
Parameter
Value
Energy Range
6 – 24 keV
Horizontal Offset
600 mm
Scattering Angle
90 - 500
 Accuracy
0.02 arcsec
χ Accuracy
4 arcsec
Scattering Angles (2 theta)
1.2.2.1 - Double Crystal Offset Monochromator
Narrows x-ray spectrum for resonant scattering experiments
Multiplexes LCLS beam (mono. beam, diagnostic beam)
1.5 Å
0.15 Å
Silicon
111
27.6°
9.1°
Silicon
220
45.8°
14.9°
Diamond
111
42.5°
13.9°
Diamond
220
-
22.8°
12
X-ray Optics
Flexure Stages
Piezoelectric Stages
1.2.2.1 - Double Crystal Offset Monochromator (cont.)
 motion
0.02 arcsecond resolution and repeatability (100 nrad)
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
X-ray Optics
Mono
190 m
Lens
4m
1.2.2.2 – Beryllium lens focusing optic
~ 2 µm FWHM focal spot size
~ 40% throughput
~10 6 gain
Positioning resolution and repeatability to 1 µm
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
X-ray Optics
Boron Carbide
Slit Blade
Tungsten Alloy
1.2.2.3 – Precision Slit System
Positional resolution and repeatability – 1 µm
High damage threshold
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
X-ray Optics
1.2.2.4 - Attenuators
Variable, up to 10 6 reduction
High damage threshold (Be or B4C)
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Pulse Split/Delay System
Thin Si(400) acts as beam splitter at 8 keV
10 µm Si crystal reflects >80% within BW~2x10-5
Outside this bandwidth, transmits 75%
Translation
stage to
change
path length
Tune upper and lower paths to slightly
different energies within LCLS BW of 10-3
Thin Si crystal
(beam splitter)
Thick Si crystal reflectors
on translation stage
Single translation stage changes relative path
lengths
54 mm translation along indicated direction ~
500 ps change in delay
1 µm resolution ~ 10 fs resolution
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
6-Circle Diffractometer
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Particle Injector
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Expected Fluctuations of the LCLS
Intensity fluctuations exceeding 30%
Expected spatial jitter ~25% of beam diameter
Wavelength fluctuations expected to be ~ 0.2% of center
wavelength (≈ LCLS intrinsic bandwidth)
Pulse duration expected to vary ~15%
X-ray Pulse/LCLS RF timing will fluctuate by ~ 1 ps
- Diagnostics are required to measure these parameters since they
cannot be controlled
- This information must be available to accelerator operations and
experiments
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Temporal Jitter
Master
Clock
Electron Gun
Accelerating
Elements
Coax RF
Distribution
Network
Experimental
Pump Laser
Sources of Short Term Jitter
• Coax RF distribution Network
• e-beam phase to RF phase
• End Station Laser phase to RF phase
Limited to ~ 1 ps !
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Electro-optic Sampling
Stabilized Fiber Optic RF Distribution (10 fs)
LBNL
Pump-probe
Gun Laser
Electro-optic
Sampling
Laser
Sector 20
LTU
NEH
Laser
Temporal resolution is now limited by:
1) Our ability to phase lock the lasers to the RF
2) Intra-bunch SASE jitter
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
SPPS Laser/X-ray Timing
100 consecutive shots
Controls Review
Apr. 10, 2007
Single shot, Lorentzian fit
Yiping Feng
yfeng@slac.stanford.edu
Data Sorting at SPPS
• 10 Hz
• Point Detector
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
XPP Data Sorting at LCLS
X-ray Detector
LCLS Beam Parameters
Intensity
Time of Arrival
Wavelength
180 MB/s
~ 1 MB/s
Real Time
Processing Unit
t1
Controls Review
Apr. 10, 2007
t2
t3
t4 …………………………………………………….………………tN
• 120 Hz
• 1 Megapixel Area Detector
• 2-dimensional binning or data filtering?
Yiping Feng
yfeng@slac.stanford.edu
Laser System Overview
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Ultrafast Laser System
Ti:Sapphire Oscillator & Power Amplifiers
Compressor, OPA, Harmonic Generation, Delay Stage
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Ultrafast Laser System
Cavity Length
Stabilization Mirror
1.2.3.1 – Ti:Sapphire Oscillator
119 MHz rep. rate, <30 fs ~ 2.5 nJ/pulse
Frequency stabilized to LCLS RF < 300 fs rms phase jitter
Demonstrated at SPPS
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Ultrafast Laser System
1.2.3.2 – Power Amplifiers
Regenerative amplifier
~ 2.5 mJ (< 1% rms stability), 120 Hz, <35 fs
Multipass amplifier
~ 20 mJ (<1.5% rms stability), 120 Hz, <35 fs
Second Compressor
External Pockels Cell
Arbitrary laser pulse train structure
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Ultrafast Laser System
1.2.3.3 – Temporal Pulse Shaper
Create complex excitation pulse envelopes
Multi-pulses
Compression optimization
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Ultrafast Laser System
1.2.3.4 – Optics & Optomechanics
Ultrafast Laser Optics (all specified wavelengths)
Mirrors
Polarization optics
Focusing optics
Non-linear crystals (harmonic generation)
Motorized optical mounts
~3 meter retro-reflecting delay stage
Steering mirrors & stabilization system (LCLS gun laser)
Rotation & translation stages
1.2.3.5 - Vacuum transport system
Relay imaging system
Vacuum environment (< 1 torr)
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Ultrafast Laser System
1.2.3.6 – Diagnostics
Pulse energy meters
Spectrometer
Spatial Profile
Temporal pulse characterization
Autocorrelator
FROG
Contrast characterization
3rd Order Correlator
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Ultrafast Laser System
1.2.3.7 – Optical table system
Earthquake restraints
Connections
1.2.3.8 – Laser timing
Phase locking electronics (PLL)
Testing
1.2.3.9 – Optical exp. hardware (in situ characterization)
High resolution spectrometer
Balanced detector
Lock-in amplifiers & chopping wheel
1.2.3.10 – Laser containment system
1.2.3.11 – Optical Parametric Amplifier (OPA)
Wavelength tuning
240-11,000 nm
Controls Review
Apr. 10, 2007
Yiping Feng
yfeng@slac.stanford.edu
Expected Fluctuations of the LCLS
Intensity fluctuations exceeding 30%
Expected spatial jitter ~25% of beam diameter
Wavelength fluctuations expected to be ~ 0.2% of
center wavelength (≈ LCLS intrinsic bandwidth)
Pulse duration expected to vary ~15%
34
Diagnostics
Item
Purpose
Specification
Position
Monitor
Image x-ray beam to align
x-ray optics
25 mm x 25 mm field of view (FOV)
- (50 µm res.)
5 mm x 5 mm field of view
- (10 µm res.)
Intensity
Monitor
Alignment of x-ray optics
- single pulse measurement
- rel. accuracy <10 -2
Transmissive
Intensity
Monitor
Normalization signal for
experiments
- Transmissive (<5% abs.)
- single pulse measurement
- rel. accuracy <10 -3
Wavefront
Sensor
Characterize spatial profile
of focussed x-ray beam
Single Shot, 120 Hz,
operation at 1.5 nm
and 0.15 nm (separate sensors)
All diagnostics will be standardized and modular.
35
Position Monitor
Alignment of x-ray optics
Images fluorescence of a
scintillating material
Attenuation of beam may be
required to avoid saturation
Large FOV operating mode
25 mm x 25 mm
50 micron resolving power
Small FOV operation
5 mm x 5 mm
10 micron resolving power
Retractable
36
X-ray Diode Intensity Monitor
Alignment of reflecting x-ray optics
(Bragg reflectors, mirrors, gratings)
Strategically placed in close
proximity to optic
Retractable
Experience at SPPS using
commercial Si tech.
37
Hard X-ray Intensity Monitor
Compton backscatter from Be foil (up to 10 8 photons)
Transmitting ( >98% w/ 100 micron Be)
Position information available with use of diode array
5 micron accuracy for commercial fluorescence monitor
Positional calibration performed with precision motion (< 2 µm)
38
Hartman Wavefront Sensor
Image obtained from Imagine Optics, Ltd
Measurement made far from focal plane
Single shot operation
120 Hz with CCD modification
1.5 nm and 0.15 nm operation with customization
39
Components of Instruments
Optics
KB mirrors for focusing
Refractive lens for focusing
Monochromator
Collimator
Slits
Attenuators
Split-delay
Pulse picker
Compressor
Sample environment
Particle injector
Cryostat
Cryo-em stage
Precision stages
Beam Diagnostics
Intensity monitors
Beam positioning monitor
Wavefront sensor
Measurement instrument
Diffractometer
e- and ion TOF
Mass spectrometer
EO timing measurement
Laser systems
Pump laser and diagnostics
EO laser
Molecular alignment laser
Vacuum systems
Turbo pumps
Ion pumps
2D Detectors
Cornell detector for CXI
BNL detector for XPP
BNL detector for PCS
40
1St Experiments
Coherent X-Ray Imaging
Diffraction using coherent x-ray beam
Obtaining structures by oversampling
Computational intensive
Large amount of data
By far the most computational demanding both in real-time or offline
X-Ray Pump-Probe
Use laser pulses as well as x-ray pulses
Timing delay measured for dynamics analysis
Real-time binning possible
Small data volume if real-time process performed
Post process straightforward
X-Ray Photon Correlation Spectroscopy
Double exposure by split-delay optics
Timing delay set and used to probe dynamics
Real-time data compression possible
Smaller data volume if real-time process performed
Post process straightforward
41
Data Rates and Volume
Year
2009-
2012-
2015-
Rep Rate (Hz)
120
120
120
Detector Size
(Megapixel)
0.58
1.16
5.8
Intensity Depth (bit)
14
14
14
Success Rate (%)
10%
30%
50%
Ave. Data Rate (Gigabit/s)
0.1
0.58
4.9
Peak Data Rate (Gigabit/s)
0.97
1.94
9.8
Daily Duty Cycle (%)
25%
50%
75%
Accu. for 1 station (TB/day)
0.26
3.14
39
Accu. for 3 stations (TB/day)
0.80
9.4
118
Yearly Uptime (%)
25%
50%
75%
Accu. (Petabyte/year)
0.072
1.7
32
Duration/Lifetime (year)
3
3
3
Total Accu. (Petabyte)
0.22
5.2
97
42
Specifications
Per pulse data collection
Experimental
Diagnostic – EO signal, e- and g beam
parameters
Raw data rate and volume
2 Gb/sec or higher
On-line storage capacity - 20 TB/day
Pump Laser operation
Vacuum controls
MPS systems
Laser PPS systems
EO system
Timing/Triggering
EO timing measurement < 1 ps
Detector trigger < 1 ms
Real time analysis
Frame correction, quality control
To the extent possible - binning,
sparsification, FFT
Quick view
Quasi real-time feedback, 5 frame/s
Alignment
Data Management
Unified data model
Archiving capacity – 5 PB/year
Analysis staging storage capacity – 20 TB
Offline Analysis
> 1000 node cluster
43
Architectural Requirements
LUSI
Control
& Data System
Timing & Triggering
Feedback
LCLS
Control System
Pulse-by-pulse
info exchange
Control Subsystem
for Operation
& Controls
Controls
for RF/Undulator
High peak rate/
large volume
Data Archiving/Retrieval
Offline Analysis/Rendering
Data Subsystem
for Acquisition
& Management
SLAC
Sci. Computing
& Computing Srvs.
Data Farm
(PB Tape Drive)
Computer Cluster
(2000 processor Node)
44
Functionality for Controls
LUSI
Control Subsystem
Endstation
Operation
Optics
Controls
Experimental
Instrument Control
Timing
& Triggering
Diagnostics
Controls
Laser Control & Laser PPS
Feedback
MPS
Vacuum
Controls
45
Functionality for Data Acquisition & Mgmt
LUSI
Data Subsystem
Measurements/
Applications
Data Acquisition/
Detector Control
On-line Storage
Quick View/Rendering
Data Archiving/Retrieval
Offline Analysis
Volume Rendering/
Visualization
Interface to
High-Level Applications
2D-Detector
46
Applications
User programs
Endstation operation
Calibration
Alignment
Interface to SW for diffraction/scattering experiments
SPEC
Interface to instrumentation/analysis SW
MatLab
LabView
User tools
STRIP tool
ALARM Handler
47
48
Computational Requirements
Real-time analysis to the extent possible
XPP  binning
CXI  averaging
PCS  compression
Offline analysis sufficiently fast to gain experimental insight
CXI  computational alignment, reconstruction
PCS  correlation analysis
Simulations to facilitate real measurements
CXI  Coulomb explosion
XPP  Molecular dynamics, etc.
PCS  Sample damage calculations
Infrastructure for integrated data management
On-demand data analysis
Archiving/Retrieval
Access/Security
Data integrity/replication or mirroring
Distribution to sister labs or centers
49
Hardware Requirements
Real-time computing
Power: 10 Tera-FLOPS
1000 processor cluster
Memory: 10 -100 GByte RAM
Bandwidth: 100 Gbit/s
Integrated w/ detector or immediate downstream of detector output
Data Storage/Management
10 – 100 Gbit/s links
10 – 100 on-line capacity: RAID disks, or flash memory
10 – 100 staging capacity: RAID disks, or flash memory
5 Petabyte yearly capacity
ESNET connection for transferring to sister institutes
Offline Analysis
Total FLOPs: 1017
If analysis done in minutes: 2000 – 40000 processor cluster
Large volume set rendering: 109
50
Coherent Imaging of Single Molecules
• Diffraction from a single molecule:
noisy
diffraction
pattern of
unknown
orientation
single LCLS pulse
unknown
orientation
• Combine 105 to 107 measurements into 3D dataset:
Classify/sort
Average
Alignment
The highest achievable resolution is limited by the ability
to group patterns of similar orientation
Gösta Huldt, Abraham Szöke, Janos Hajdu
(J.Struct Biol, 2003 02-ERD-047)
Reconstruct by
Oversampling
phase retrieval
Miao, Hodgson, Sayre, PNAS
98 (2001)
51
Computational Diffraction Imaging
Chapman et al.
JOSAA 23, 1179
(2006)
Memory
(GB)
Time
(Hours)
2563
0.6
0.2
5123
4.7
1.5
10243
38
14
20483
304
??
3D
size
Optimization
•109 nonlinear equations
•109
unknowns
Shrinkwrap/HIO
algorithm, S.
Marchesini et al,
PRB 68, 140101(R)
(2003)
2000 iterations, 2FFTs each
32 CPU G5 Infiniband cluster
Crandall, et al. Advanced Comp.
Group, Apple Computer (2004).
52
Computational Alignment
Experimental Data (ALS)
Difference of pyramid diffraction patterns 10º
apart, Gösta Huldt, U. Uppsala
q
kin
“The number currently used to obtain high-resolution
structures of specimens prepared as 2D crystals, is
estimated to require at least 1017 floating-point
operations”
R. M. Glaeser, J. Struct. Bio. 128, (1999)
kout

“Computational Alignment” requires large
computational power that can only be provided
by performing offline analysis
 Save first, and Analyze later
53
Mapping the Interatomic Potential of Photoexcited Bismuth:
Ultrafast Optical and X-ray Studies
54
X-Ray Pump-Probe
55
XPCS
56
XPCS
57