US ATLAS Operations Program
SLUO ATLAS Workshop, July 16, 2009
Mike Tuts
Columbia University
Topics
 What is the US ATLAS Operations
Program (OP)?
u
u
u
Management
Activities
Funding – Core Program vs Operations Program
 Who is US ATLAS?
u
u
Who we are
What we are involved in
 How does the US ATLAS OP interact
with overall ATLAS?
 How do you get involved?
US ATLAS Operations Program, SLAC 7/16/09 Tuts
2
Some History
 The US ATLAS construction project has been
completed
u
u
$165M
US contributed to all major systems
s
s
s
s
s
u
Inner detector (SCT, Pixel, TRT)
Calorimeters (LAr, TileCal)
Muon system (MDTs, End Caps, CSC, Alignment)
Trigger and Data Acquisition (ROIB, HLT, DAQ)
Core software, Computing
About 20% contribution to overall ATLAS
 As we moved to the operations phase, our
responsibilities evolved from our hardware
responsibilities
US ATLAS Operations Program, SLAC 7/16/09 Tuts
3
US ATLAS Operations Program
 Host laboratory is Brookhaven National Lab
 Three principal components:
u
WBS 2 - Software & Computing (S&C))
s
s
u
WBS 3 - Maintenance & Operations (M&O)
s
u
Computing infrastructure: Tier 1, Tier 2
Core software, computing support to facilitate physics
Pre-operations, commissioning, long term M&O
WBS 4 - Upgrade R&D
s
To handle future luminosity increases, prepare for upgrade
construction project
[WBS = Work Breakdown Structure]
 The Operations program supports technical personnel,
equipment, shared computing
u
US ATLAS Operations Program, SLAC 7/16/09 Tuts
4
US ATLAS Organization Chart
US ATLAS Operations Program, SLAC 7/16/09 Tuts
5
Funding Targets
(AY M$)
FY09 FY10 FY11 FY12
Software & Computing
Maintenance &
Operation
Upgrade R&D
Management Reserve
DOE guidance
NSF guidance
Carryover/Unobligated
Total
18.3
15.5
20.4
10.9
21.2
11.6
21.9
11.5
3.9
4.7
26.7
9.0
6.8
42.4
3.2
2.0
27.4
9.0
3.2
1.4
28.3
9.0
3.2
1.6
29.1
9.0
36.4
37.3
38.1
US ATLAS Operations Program, SLAC 7/16/09 Tuts
6
What is not supported by the US ATLAS
Operations Program?
 For personnel the Ops program does not
support physicists salaries (grad students,
postdocs, scientists, faculty)
u
Our test is whether the person is on a “physics”
career path
 It does not support physicist travel or COLA
while at CERN
 It does not support local computing
resources (“Tier 3”)
 All of the above we judge to be the
responsibility of the core (“base”) program
US ATLAS Operations Program, SLAC 7/16/09 Tuts
7
US ATLAS Institutions
 ATLAS is an international collaboration
u
Individual institutes are voted in by the ATLAS Collaboration Board (CB)
 Currently 44 US institutes (from 22 states)
u
u
Albany, ANL, Arizona, UT Arlington, Berkeley LBL and UC, Boston,
Brandeis, BNL, Chicago, Columbia, UT Dallas, Duke, Fresno State#,
Hampton, Harvard, Indiana, U Iowa#, Iowa State, UC Irvine, Louisiana
Tech* , Louisville#, Massachusetts, MIT, Michigan, MSU, New Mexico,
NIU^, NYU, Ohio State, Oklahoma, Oklahoma State, Oregon,
Pennsylvania, Pittsburgh, UC Santa Cruz, SLAC, SMU, South Carolina*,
SUNY Stony Brook, Tufts, Illinois Urbana, Washington, Wisconsin, Yale
Corresponding to 39 voting institutions
* = affiliated with BNL; # = affiliated with SLAC; ^= affiliated with ANL
 As of Sept 30, 2008 (used to determine our “dues” to ATLAS) [7/15/09]
u
38/169 voting institutions (22%)
u
395/1817 “current M&O authors” = ~PhDs (22%) – for cat A/B (these are
the shared operating costs for the detector) [428/1889 – 22.7%]
u
592/2800 M&O authors + in process of qualifying + students (21%)
[634/2893 – 21.9%]
u
497.75/2347.25 Operations tasks share (students count .75) (21%)
[551.75/2536.25 – 21.8%]
s
US ATLAS Operations Program, SLAC 7/16/09 Tuts
8
US ATLAS Demographics
 FY10 projection based on our personnel
database (snapshot July 2009)
FY
2009
Imperfect,
but
personnel
Head
CONSTRuseful:
M&O ~110
R&D FTE
COMP technical
PHYS
u
Row Labels
Admin
Computer
Professional
Designer
Electrical Engineer
Faculty
Grad Student
Mechanical
Engineer
Post Doc
Senior Research
Scientist
Technician
Term Scientist
Grand Total
Tech Subtotal
Count
3
(FTE)
0
(FTE)
1.3
(FTE)
0
(FTE)
0
(FTE)
0
73
2
36
140
150
0
0.3
0.1
0.1
0.1
13.3
0.8
10.9
15.2
35.0
0.2
0
10.5
6.5
3.4
46.3
0
1.7
14.1
19.8
0.8
0
0.1
45.6
86.1
62.2
1.0
23.3
81.4
144.3
27.5
0
9.4
38.6
104.9
10
96
0
0.2
5.3
22.1
1.5
3.1
0
19.7
0
50.6
6.8
95.7
2.8
75.5
74
26
28
643
150
0.2
0
0.1
1.0
0.4
14.9
7.3
6.3
132.3
38.8
8.4
6.0
2.5
43.7
19.9
11.0
1.4
7.3
121.2
49.4
21.6
0
8.1
212.7
0.9
55.9
14.7
24.2
510.8
109.3
21.6
2.8
11.7
294.8
42.5
9
US ATLAS Operations Program, SLAC 7/16/09 Tuts
TOT AL FTE FTE @ CERN
1.3
0
ATLAS Layout
MUON SYSTEM
MAGNETS
8 Barrel Toroids
Central Solenoid
End Cap Toroids
Monitored Drift Tubes (MDT)
Cathode Strip Chambers (CSC)
Resistive Plate Chambers (RPC)
Thin Gap Chambers (TGC)
INNER DETECTOR (ID)
Pixels
Silicon Strip (SCT)
Transition Radiation Tracker
(TRT)
CALORIMETERS
EM - Liquid Argon – Lead
HAD - Scintillator Tile
Diameter 25m
Length 46m
Weight 7,000 tons
CAP/DAP Review Jan 14-16, 2008
10
3.1 Silicon Trackers
US Institutes (3.4FTE): LBNL,
UCSC, OSU, Oklahoma ,
SLAC, UT Dallas, Columbia,
Louisiana Tech, Iowa, Iowa
State, UNM, Wisconsin
 Areas of responsibility
(derived from construction
project) – under Alex Grillo
(UCSC)
u
u
u
Cut-away view of inner detector
Pixel: disks; front end readout;
hybrids; end caps; opto boards;
support/ structures; services
SCT: front end readout; modules;
electronics – design oversight,
grounding, shielding
ROD: readout driver cards,
firmware, software
Barrel Pixels during construction
US ATLAS Operations Program, SLAC 7/16/09 Tuts
11
3.2 Transition Radiation Tracker (TRT)
US Institutes (1.4FTE):
Duke, Hampton, Indiana,
Penn, Yale
 Areas of responsibility
(derived from
construction project) –
under Harold Ogren
(Indiana)
u
Barrel modules;
electronics
US ATLAS Operations Program, SLAC 7/16/09 Tuts
3D
represent
ation of
inner
detectors
TRT (at
outer
radius) and
Inner silicon
detector in
assembly
hall in
cosmic ray
test
12
3.3 Liquid Argon Calorimeter (LAr)
US Institutes (14FTE):
Columbia, SMU,
Pittsburgh, Stony Brook,
BNL, Arizona
 Areas of responsibility
(derived from construction
project) – under Ryszard
Stroynowski (SMU)
u
u
Mechanical: Barrel cryostat;
feedthroughs; cryogenics; Fcal
Electronics: front end boards;
links; cables; crates; L1 trigger;
low voltage power supplies
Liquid
Argon
barrel and
end caps
Barrel
calorimeters
in detector
hall,
surrounded
by barrel
toroids
US ATLAS Operations Program, SLAC 7/16/09 Tuts
13
3.4 Tile Calorimeter
US Institutes (3.6FTE):
ANL, Chicago, NIU, MSU,
UTA, UIUC
 Areas of responsibility
(derived from
construction project) –
under Larry Price (ANL)
u
Tile Cal
Barrel and
extended
barrel
Extended barrel; modules;
electronics design; PMTs;
intercryostat scintillator
Tile Cal extended barrel in hall during assembly
14
US ATLAS Operations Program, SLAC 7/16/09 Tuts
3.5 Muon System
US Institutes (7.8FTE): Arizona,
BU, BNL, Brandeis, Duke,
Harvard, Illinois, MIT,
Michigan, SLAC, South
Carolina, Stony Brook, Tufts,
UCI, UMass, Washington
 Areas of Responsibility –
under Frank Taylor (MIT)
u
u
u
u
u
u
3D cut-away view of muon system
End Cap Monitored drift
tubes (MDT)
All chamber electronics
Cathode Strip Chambers
(CSC)
Alignment
Maintain configuration
database
Calibration center operation
US ATLAS Operations Program, SLAC 7/16/09 Tuts
Muon endcap big
wheel
during
assembly in
hall,
surrounded
by TGC
chambers
15
3.6 TDAQ
US Institutes (10.9FTE)
dual-CPU nodes
CERN
~10 ~1900 ~100 ~ 500
UCI, MSU, ANL,
computer
Secondcentre Event rate Local
Event
Event LVL2
Wisconsin, NYU, Data
~ 200 Hz Storage
level
Filter
Builder
farm
trigger
(SFOs)
(SFIs)
(EF)
BNL, Oregon, SLAC, storage
UTA, Chicago, UIUC,
DataFlow
Network
stores
Manager
switches
LVL2
Penn, Yale
DFM
Network switches output
LVL2
 Areas of
Supervisor
Schematic
SDX1
Responsibility –
of TDAQ
under Andy
system
Lankford (UCI)/
Reiner Hauser
Data of events accepted
(MSU)
1600
u
u
Hardware & Software
commissioning &
maintenance
Detector & TDAQ
support
Operations support
Gigabit Ethernet
Event data requests
Delete commands
Regions Of Interest
u
Requested event data
pROS
~150
PCs
USA15
RoI
Builder
Read-Out
Subsystems
(ROSs)
by first-level trigger
ReadOut VMEDedicated links
Links Read-
Timing Trigger Control (TTC)
US ATLAS Operations Program, SLAC 7/16/09 Tuts
Out
Drivers
(RODs)
Firstlevel
trigger
ATLAS
Detector
( UX15 )
16
2. Software & Computing

Analysis to be done on worldwide
US Institutes (48 FTE):
computing grid of 26k CPU’s (WLCG)
BNL, ANL, LBNL, Iowa State, SLAC,
u
WLCG is the Worldwide LHC
Chicago, Indiana, BU, Harvard, UTA,
Computing Grid
Oklahoma, Michigan, MSU, Arizona,
u
Open Science Grid (OSG) is the US
Duke, Louisiana Tech, Oregon,
grid and part of WLCG
UMass
 In the US we have
 Computing management under Jim
u
Tier 1 Computing center (T1) @ BNL
Shank (BU)/Srini Rajagopalan (BNL)
u
Five Tier 2’s (T2) ($600k/yr from OP)
s
s
s
s
s
u
Local Tier 3’s (T3) at universities
s

NET2 - BU/Harvard
MWT2 – Chicago/Indiana
GLT2 – Michigan/MSU
SWT2 – UTA/Oklahoma
WT2 - SLAC
Stimulus funding may prove to be a source
of funding – many proposals submitted
We manage the T1 & T2 (& some T3)
in such as a way to provide a
seamless computing fabric
ATLAS Tier 1 computing sites
US ATLAS Operations Program, SLAC 7/16/09 Tuts
17
T1 & T2 Resources
 US T1 and
T2 facility
anticipated
pledges
u
US ATLAS Operations Program, SLAC 7/16/09 Tuts
From Feb
09 review –
changing
with LHC
schedule
18
4. Upgrade R&D
 The Operations Program supports the R&D
necessary needed for construction of an
upgraded ATLAS
u
Phase 1 (~2014/15 install) envisions ~3x1034 and
few hundred fb-1
s
s
u
Inner B layer (IBL) silicon replacement
Muon and TDAQ
Phase 2 (2018/19??) luminosity goal of 1035 and
enough integrated luminosity that inner detector
requires full replacement, Calorimeters, Muon,
TDAQ
 Upgrade activities will be presented at this
meeting by Abe Seiden and others
US ATLAS Operations Program, SLAC 7/16/09 Tuts
19
Analysis Support
 While the Operations Program does not manage
physics, it does provide support for analysis
activities – under Jim Cochran (Iowa State)
 US ATLAS goal is to support entry into overall
ATLAS physics activities
u
3 Analysis Support Centers: BNL, ANL, LBNL (and SLAC)
s
u
Analysis Support Group
s
u
Host “Jamborees” on various analysis topics, tutorials
Team of “virtual” experts that can answer your questions
Physics & Detector Performance Forums
s
s
A more informal venue in which to discuss analyses, issues, in
US time zone
Venue for practice talks for US conferences
US ATLAS Operations Program, SLAC 7/16/09 Tuts
20
US ATLAS and DOE & NSF
 US ATLAS Operations Program is jointly funded and
managed by DOE & NSF
u
u
u
u
US LHC OP is reviewed annually by DOE/NSF with outside
consultants
Joint Oversight Group (M. Procario (DOE) + M. Goldberg (NSF))
meet twice yearly
BNL Program Office handles budgets; we get input from
Executive Committee on resource priorities
DOE funding distributed via BNL subcontracts & NSF funding
distributed via Columbia subcontracts
 Agencies seek advice from US ATLAS OP management
on core program needs and priorities
u
u
DOE “help” funds (has been $600k-$850k)
We review core program needs individually with each of the 44
institutes in US ATLAS
US ATLAS Operations Program, SLAC 7/16/09 Tuts
21
US ATLAS and ATLAS
 The US ATLAS Operations Program
management interacts closely with overall
ATLAS management
u
u
u
u
Coordinating resource priorities with overall
ATLAS priorities
Serving as “National Contact Physicists” for the
US
Supporting the “per capita” operating costs for
overall ATLAS (called cat A and B costs,
~$10k/year/per PhD collaborator)
Advising and serving as a point of contact for
ATLAS management regarding new US groups
US ATLAS Operations Program, SLAC 7/16/09 Tuts
22
US ATLAS - Conclusions
 US activities on ATLAS in Computing & Software,
Maintenance & Operations, Upgrade R&D are managed and
funded (DOE &NSF) centrally through the US ATLAS
Operations Program
u
u
Operations Program supports technical personnel & equipment
Operations Program pays for per capita costs for US PhD physicists
 ATLAS is an international collaboration – after being
accepted by ATLAS you are automatically in US ATLAS
u
New smaller institutions have typically joined affiliated through an existing
US group until they prove themselves and build up group strength
 We are anxiously awaiting first beam and the start of the
physics program – we hope you will join us!
u
If you are considering joining ATLAS we can help to interface you to the
overall ATLAS management
 At this workshop you will hear more details of some of the
ongoing activities
US ATLAS Operations Program, SLAC 7/16/09 Tuts
23