NUMI
MINOS
Far Detector Installation,
Commissioning, & Operation
Jeff Nelson
South Carolina Meeting
March, 2003
NUMI
Outline
MINOS
• FD Assembly
> Planes, staffing, & status
> Shield update
• FD Commissioning
>
>
>
>
Front ends, LI, DAQ, DCS, Magnet
Documentation update
Data quality & operations update
Electronic echoes (Roy Lee’s talk)
• Planning for shifts & operations
• CDMS 2 - progress worth noting
USC
3/03
Page 2
NUMI
J. Meier
1/30/03
MINOS
USC
3/03
Page 3
NUMI
Plane Assembly
MINOS
Far Detector Installation by Week
• 426 planes installed
• Vacation, departures, just-in-time
delivery mode for much of SM2
Jul-01
Sep-01
Nov-01
Jan-02
Mar-02
May-02
Jul-02
Sep-02
Nov-02
Jan-03
Mar-03
May-03
Jul-03
Sep-03
50
0
• Some numbers
FD Plane Installation
> Steel at Soudan: 90%
9
> Modules at Soudan: 94%
7
5
4
3
2
Apr-03
May-03
Mar-03
Jan-03
Feb-03
Dec-02
Oct-02
Nov-02
Sep-02
Jul-02
Aug-02
Jun-02
Apr-02
May-02
Mar-02
Jan-02
Nov-01
Feb-02
Planes/Week (Monthly Averages)
Dec-01
0
USC
3/03
Page 4
Working Schedule Planes/wk
1
Aug-01
> 85th week of installation
6
Jul-01
> 60 planes to go
8
Planes per week
> Over 1100 loads down shaft
FNAL Working Schedule
Projected Schedule
Planes Installed
L3 milestones
Oct-01
> Directly related to FTEs worked
350
300
250
200
150
100
Sep-01
> Averaging 5.5 PPW for SM2
(was 6.4 in SM1) due to
Planes Installed
> 86% completed, 4.75 kton
500
450
400
NUMI
MINOS
Schedule Projection
• We are planning our schedule based on the plane
construction completing June 2nd
> Current projection is May 28th
> Future based on some additional departures…
USC
3/03
Page 5
NUMI
MINOS
Staffing & Future Projections
• Crew in flux
> 7 departures since last summer (of the 22 original
builders/movers)
> We have hired 3 replacements
> 3 more temps will start in March
• Expect to lose more to the mines
> 50 more hires at just one mine this month
> Many of the new workers are from construction trades
• May leave when the road restrictions lift this spring
• Some may leave sooner as construction starts
> Could be as many as 5 to 7 more departures
• There will also be a substantial demobilization by materials
handling crew after the last steel/scintillator is
underground
• This leaves a +2/-1 week uncertainty in projections
USC
3/03
Page 6
NUMI
MINOS
More Schedule
• Roughly 2 months to complete the coil, shield, &
final detector shake down
• After the last large objects are removed in very
early June we will go to a days-only schedule
> Maybe 9-hour days summer and maybe 8 hours days
fall/winter
> Significantly reduced crew staffing next summer
USC
3/03
Page 7
NUMI
MINOS
Scinticicles
• Scintillator shipped in early February had some problems
> Some of the skins delaminated
> Some strips show difference in readout
> These were the first modules shipped in very cold weather
• 19 modules from that shipment mapped again
> 4 modules with problems (15 strips)
• 2 connector, 4 manifold, 7 fibers unglued (2 more unclassified)
• Ones with strange initial maps appear more vulnerable
> Some may be installed before we noticed
> Have to compare LI to maps to know
• These are the first major differences
in scintillator response observed at
Soudan
> Now sample mapping and transmission
testing CIT modules from Dec and
late Feb shipments
USC
3/03
Page 8
NUMI
MINOS
Some strips in the worst module
USC
3/03
Page 9
NUMI
MINOS
Commissioning
• 417 planes reading out
> Last commissioning on Friday
> Mostly routine at this point
USC
3/03
Page 10
NUMI
MINOS
Shield Update
• Since December we
commissioned the revised
Section 1 shield
> Shorter cables and
repositioned modules
• Added lower walls to
sections 1 & 2 (next page)
• Future work on hold until
funding situation is
clarified
• When/if OK’d
> Can install section 3 now
> Can reconfigure section 2
as double layer
• See Caius’s talk for the
justification
USC
3/03
Page 11
NUMI
MINOS
Shield Lower Wall Installation
USC
3/03
Page 12
NUMI
MINOS
FD DAQ/Rack Hardware
• Crate commissioning
> Crate 15 (the last crate)
• All the marginal bits ended up here
• Tass finally debugged
> Some damage in things commissioned but not used for a
few months
• Swapped with spares (aka ND DAQ)
• Not sure the effect
• Significant modifications to the crate procedures (pull
cards for extensive work; sign off before power)
• Branches (PVICs)
> Vendor has a hardware fix to the node/length problem
> Swap through spares for upgrades after an initial test of
a four node branch at Soudan
USC
> Months and a number of expert visits
3/03
before we convert back to 4 branches
Page 13
NUMI
MINOS
DAQ Software at Soudan
Areas of Emphasis
• ROP configuration overrides (Tim&Tass)
> Override of dynode threshold settings for a scan
> Override of sparsification mode to allow unsparsified
running
> Override of VARC trigger to allow untriggered running
> Masks so that high rate special runs can be limited to a
sub unit of the detector readout
• LI configuration improvements (Lisa)
> LI runs spanning many runs
> Tuning the light for individual LEDs
• RunControl enhancements (Caius, et al)
> Web status page improvements
> Fix memory leak (the Sunday problem)
USC
3/03
Page 14
NUMI
MINOS
On the Horizon
• ROP reset and abort implemented through RC
• The ROP will also calculate sparsification
thresholds on pedestal runs (3 widths above
pedestal)
• RC control using new ROP exposed bits
> Making a diagnosis run sequence can then follow
• New RC GUI with added feature
> Upgraded web status page
USC
3/03
Page 15
NUMI
FD HV
MINOS
• Hardware
> Replaced the faulty ferrups
• Only three drop outs since January
> Replaced a HV card
• Software
> Ramp changed to hardware ramp
• No plot yet
> Changes to warning/reset rules
> Testing MySQL logging database
• See DCS section
USC
3/03
Page 16
NUMI
FD DCS
MINOS
• RPS
> Replaced the last “old” units
> Upgrades so fan pack monitoring works (7 of 16 done)
> Adding smoke detectors for under the shield (per
agreement with code officials)
> Change in logic for loss of power at VFBs
(error->warning)
• Lost a fuse the next week… is it a good idea?
> Now have a command-line daemon to query the RPS units
• Enviro
> Need to get north cluster running & prep for SM2 coil
> Radon measurements differ wrt CDMS 2 at factor of two
> Need to replace the north radon counter
• Coil
> First field drop out happened last week
USC
3/03
Page 17
NUMI
MINOS
State of DCS Data
• Erik Beall and rest of DCS working to get the DCS data into
it’s own data stream
>
>
>
>
>
Starting with Nat and the HV
Getting data into a local DCS DB
Exported into rotorooter / data stream
Working on the next step with Robert
Then expand across other subsystems
• Follow on projects
> Web interface for shifters
• Current have to do a lot of work
to check the DCS in a remote shift
– Kerberos/ssh for HV, RPS
– VNC for coil & environmental
– DAQ tells the grossest problems
> Making this data available
in the offline (core group)
• Some issues; see computing talk
USC
3/03
Page 18
NUMI
FD LI – system tune up
MINOS
• Test of a UV pulser box
> Looks better
• See calibration summary
> Light level/focus issues
• LI light level tuning (Hartnell/Falk)
> LIMaster modified to allow
different settings for each LED
> Jeff has tuned the levels
> Need to have a better algorithm for
spacing the steps
USC
3/03
Page 19
NUMI
MINOS
FEE Echoes & LI
• Can’t run like planned now
• Need ~2 hours for one stability point
without data collection
• See Roy’s talk on the progress
• Would need to rethink calibration
mode if not fixable
USC
3/03
Page 20
NUMI
MINOS
LI as a Diagnostic Tool
• Starting to use LI as a diagnosis tool is was always meant to be
< BOX/LED REPORT >
Missing Pulser Box 14 Completely.
Missing Pulser Box 15 Completely.
Very Quiet Box 12 Only 14464361 Associated Hits (59%)
Very Quiet Box 13 Only 14434576 Associated Hits (59%)
< FEE REPORT >
No Hits Beyond Plane 417
Missing Crate --- Crate 14
Missing Crate --- Crate 15
Very Low Channel/Pixel ---- 0 2 2 0 1 16 -- Plane 55 Only 155 Hits ( 0%)
Very Low Channel/Pixel ---- 2 2 3 1 0 12 -- Plane 122 Only 3117 Hits ( 19%)
Very Low Channel/Pixel ---- 11 1 3 1 2 5 -- Plane 347 Only 0 Hits ( 0%)
Low Chip ---- 12 1 1 1 2 -- Planes 403-405 Only 234789 Hits ( 91%)
Very Low Channel/Pixel ---- 13 2 1 1 0 2 -- Plane 383 Only 207 Hits ( 1%)
Very Low Channel/Pixel ---- 13 2 3 1 1 13 -- Plane 393 Only 40
USCHits ( 0%)
3/03
Page 21
NUMI
MINOS
Some Status Plots
(After Recent Detector Sweep)
USC
3/03
Page 22
NUMI
MINOS
A More “Interesting” Plot
(the week before)
USC
3/03
Page 23
NUMI
MINOS
Still some more remedial work
• A number of PIN problems
• 5 single pixels
• Few dead LI fibers
Dead Fibre East: 85
Something wrong with
Something wrong with
Something wrong with
Something wrong with
Something wrong with
Something wrong with
Miswired Fibre East:
Miswired Fibre East:
Dead Fibre East: 363
Dead Fibre East: 375
Miswired Fibre East:
Miswired Fibre East:
Miswired Fibre East:
Miswired Fibre East:
Something wrong with
Miswired Fibre East:
Miswired Fibre East:
East:
East:
West:
East:
East:
East:
351
351
226-228
266-268
314-316
339-341
339-341
342-344
382
382
382
382
East: 402-404
407
407
box
box
box
box
box
box
box
box
box
box
box
box
box
box
box
box
box
box
3
6
8
11
10
10
10
11
11
10
11
12
12
12
12
12
13
13
led
led
led
led
led
led
led
led
led
led
led
led
led
led
led
led
led
led
11
4
18
9
16
20
10
19
20
7
20
2
3
5
20
10
7
8
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Wrong:
Near
Near
Near
Near
Near
Near
Near
Near
Near
Near
Near
Near
Near
Near
Near
Near
Near
Near
14 Far: 8
10 Far: 12
12 Far: 16
12 Far: 14
4 Far: 8
10 Far: 6
4 Far: 12
20 Far: 20
16 Far: 18
14 Far: 8
16 Far: 16
10 Far: 12
10 Far: 10
10 Far: 10
12 Far: 10
4 Far: 14
20 Far: 20
20 Far: 20
USC
3/03
Page 24
NUMI
Noise Problems
MINOS
•
•
•
•
We are seeing a recurring but
spreading noise problem
The affected tubes are
essentially dead
Problem is the HV connector on
the base needs to be reseated
Recurring but new ones coming
up too
>
•
>
>
Have made 3 passes looking for
problems this year
~ half dozen every two weeks
3 have been back each time
Needs a fix to make sure these
are not continuing to make
trouble
>
Doug assigned to the IU MUX
guys
USC
3/03
Page 25
NUMI
MINOS
Update on Electronics Mortality
(Dec-Feb)
VFB swaps
Base swaps
Cable swaps
+
++
*
**
Dec
3++
0
1
Jan
8+
9**
1
Feb
Tot
0
11
5*
14
0
2
(1 power, 1 control)
2 cases swapping didn't solve the problem
1 case swapping didn't solve the problem
All old bases have low R21 (~1kohm)
2 cases the swapping didn't fix the problem &
2 cases with errors while changing the base
USC
3/03
Page 26
NUMI
MINOS
Failure Rates
• At December Karol told us about the base problem
being only being partially to blame for holes (~33%
of bases pulled at FarDet and CalDet really bad in
UT test stand)
> Since then Mike led an effort to come up with a revised
set of diagnostic steps
> Results are lower rate of false swaps
> Base test stand for Soudan later this month
• Recently Alfons told us similar rates were seen
for the VFBs (8 of 23)
> A test stand for VFBs will be revived at Soudan to
reduce the number of false shipments
• Plans for future
> FNAL PREP services all front end electronics
> Test off detector at Soudan before shipping
USC
3/03
Page 27
NUMI
MINOS
Numbers & Spares
Item
VARC VMM
VFB
VAs
Ctrl Cable Pow Cable Sig Cable
MINOS TOTAL
46
259
517
1551
532
519
519
Currently in detector (SM1+SM2)
46
255
409
1227
440
427
427
spares est. for operat.
9
20
50
150
30
20
26
on hand; not installed
3
12
69
207
34
63
118
coming from CERN after const.
4
24
30
90
0
0
0
coming from CERN before end of const.
3
19
30
90
0
0
0
pending / on order
0
0
30
90
100
50
0
needed
55
279
567
1701
562
539
545
total
56
310
568
1704
574
540
545
1
31
1
3
12
1
0
excess
• We evaluated the spares with Roy & Alfons to ensure we
have enough to run for many years of operations
• Total spares seem reasonable … to me
USC
• Table slightly out of date (Ca. early Feb)
3/03
Page 28
NUMI
MINOS
FD Singles
Rates per plane
USC
3/03
Page 29
NUMI
MINOS
Timing System
• There are two features that where
originally advertised but are currently not
implemented:
> A precision oscillator in the TCU that would
maintain accurate absolute timing if the GPS
clock fails
> The option to generate execute signals with an
arbitrary phase with respect to the 40 MHz
clock distributed
• Alfons would like to know if anyone finds
the useful enough to pursue
• Had 1 bad TRC & 1 bad fiber
USC
3/03
in last 2 months
Page 30
NUMI
MINOS
Magnetics
• Coil operates routinely
> Had a supply trip last
week for the first time
since August
• Alex Sousa making good
progress on calibration
control program
> Still some details to work
on
• Cabling nearly up to date
• Starting to plan for SM2
coil
USC
3/03
Page 31
NUMI
Safety at Soudan
MINOS
• We have had 1 safety report at Soudan
since (Dec-Feb)
> Recordable injury related to a previous
injury
> Need to be extra careful with the new crew
> Reviewing & updating procedures in advance
of a refresher course later this month
> Starting to cross train crew as the number
of old timers ramps down
> Have distributed the new crew among
veterans in the crews
USC
3/03
Page 32
NUMI
• A running total of 999
physicist weeks or 20
physicist years at
Soudan on installation
tasks
• Getting there but not
done
> We expect to be
commissioning well into
the summer
> There is also a period
of training the
operations crew later
this summer
Monthly averages
20
18
16
Needed
Shifters + expert
PSL + L2
14
Today
12
10
8
6
4
2
0
A-01
M-01
J-01
J-01
A-01
S-01
O-01
N-01
D-01
J-02
F-02
M-02
A-02
M-02
J-02
J-02
A-02
S-02
O-02
N-02
D-02
J-03
F-03
M-03
A-03
M-03
J-03
J-03
A-03
MINOS
Physicists & Installation
USC
3/03
Page 33
NUMI
• We need people (students,
postdocs) at either full or
50% time to come up to
speed as experts
> Needed starting in mid to
late May
> Very good summer
experience for students who
are taking classes or just
starting
> Needs for shift leaders or
postdocs to help lead
> More senior visitors are
welcome but already a few so
we need to coordinate the
dates
12
10
Needed
Shifters + expert
PSL + L2
8
6
4
2
0
6-Jan
20-Jan
3-Feb
17-Feb
3-Mar
17-Mar
31-Mar
14-Apr
28-Apr
12-May
26-May
9-Jun
23-Jun
7-Jul
21-Jul
4-Aug
18-Aug
MINOS
Needs at Soudan
USC
3/03
Page 34
NUMI
MINOS
Open House
• Soudan Open House scheduled for
Saturday May, 2nd (the week before
fishing opening)
> Last year we had 650 visitors
> Need about 10 tour guides
• Sign up with Bill this weekend
> More on outreach and summer
opportunities/needs from Earl
USC
3/03
Page 35
NUMI
MINOS
Operations Plan
Starting this Summer
• We are going to test operating for max live time (e.g. like we
plan when in beam operations)
Goal is >>95%
• Our plan is to staff during 5 days shifts underground
• We will have two crew (+ hoistman) on call for underground
access 24/7 in case of situations that are not remotely
rectifiable
> Will be used by CDMS and MINOS
• We will operate with local shifts when underground and
remote shifts (FNAL + UK?) for other hours
USC
3/03
Page 36
NUMI
MINOS
FD Documentation
http://www-numi.fnal.gov/cgi-bin/noteFDDocsPrint
• There are currently 26 documents submitted and tagged for
the FD manual
Good guys
Front ends
Environmental
Online monitoring Call lists
Plane assembly
Survey
PMT handling
Magnet
Light injection
Laggards
Magnetic calibration
Safety
DAQ
RPS
HV
• Need these for training the operations crew this
spring/summer
• Please increase the priority on this activity
to ensure efficient operations this summer/fall
> Please add in more detailed user’s (idiot’s) guides
USC
3/03
Page 37
NUMI
30%
20%
10%
0%
Kton-months (fid)
40%
Detector Untilization
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
USC
3/03
Page 38
Feb-03
Dec-02
Oct-02
other/idle %
sched down%
bad data collected %
good data collected %
Aug-02
• Will continue to have 3-4 days per
month for DAQ/RPS
• Hope to get back to some days
without taking the detector offline
50%
Feb-02
Mar-02
Apr-02
May-02
Jun-02
Jul-02
Aug-02
Sep-02
Oct-02
Nov-02
Dec-02
Jan-03
Feb-03
> HV monitoring & control improved;
hardware swaps
> Optimizing automatic recovery
hardware & software
2.0
1.8
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
-
Livetime
Exposure
60%
Jun-02
• Since the summer we have been
collecting data for calibration and
cosmic rays
• We have also been working to
increase the live time
• Addressing potential hardware
impediments
70%
Livetime
MINOS
Detector operations
NUMI
Exposure Summary
MINOS
•
•
•
•
•
•
•
•
Total exposure (since 1/31/02)
Total fiducial exposure (“
“)
Total fiducial ‘good’ exposure (“
SM1 with field (8/8/02)
Old sec 1 (8/16-11/12/02)
Sec 2 (11/27/02)
Reconfig’d sec 1 (12/?/02)
Lower shield walls (1/27/03)
“)
1.52 kty
1.04 kty
0.94 kty
0.76 kty
0.18 kty
0.17 kty
0.13 kty
0.09 kty
USC
3/03
Page 39
NUMI
MINOS
CDMS 2 Update (Part I)
(from Dan Bauer)
First tower of detectors installed in “icebox” at Soudan
Second tower installed this week
Radon
purge
Tower 1
USC
3/03
Page 40
NUMI
MINOS
CDMS 2 Update (Part II)
(from Dan Bauer)
• Dilution refrigerator and
icebox were cooled to
20 mK in Dec, 2002
• Shielding, DAQ, and
Electronics nearly
finished
USC
3/03
Page 41
NUMI
MINOS
CDMS 2 Update (Part III)
(from Dan Bauer)
• Deep site
> Much lower neutron background
> Rapid improvement in sensitivity
• Expect factor of 10 improvement over
CDMS I results by end of 2003
• Deploy remainder of detectors in 2004 and
run until end of 2005 (or longer)
• Detector performance improvements mean
we should still reach original CDMS II
goals
USC
3/03
Page 42
NUMI
MINOS
Zippy Big Picture Summary
• Now 7/8ths complete
• Will finish this summer
> Need roughly 8 additional physicistmonths late spring and summer
• Will shift into operations this summer
> Advanced model for operations
> Physicists at Soudan beyond summer is
an issue
USC
3/03
Page 43