IceCube Project Monthly Report - October 2009
Accomplishments

October was one of the most smooth and trouble-free data taking months this year. The
overall detector uptime was 99.0% and the clean in-ice uptime was 95.8%

An improved program for generating simulated detector data was released in October and
is being tested to ensure compatibility with previously generated simulation data.

IceCube personnel started traveling to the South Pole in October. The first drillers and
the two detector operators that will winter at South Pole in 2010 are scheduled to arrive at
Pole the first week in November 2009.

Three undergraduate students began a trip to the South Pole as part of the Upward Bound
program that encourages science experiences for students from under represented
populations. The students travel on the Swedish Icebreaker Oden with a special IceTop
tank outfitted with DOMs to calibrate the energy response of the tanks to solar particles.
IceCube Project Baseline (M$)
Initial In-Ice Strings & IceTop Tanks Installed
Initial Operational Capability
Project Completion & Closeout
Total Project Cost
Value of Foreign Contributions
NSF Funding
Remaining Contingency
Contingency as % of Remaining Work
Strings installed
Performance and Cost through:
300
250
273.6
275.2
260.8
275.5
268.7
231.6
199.4
200
DOLLARS IN MILLIONS
Jan-2005
Mar-2007
Sep-2011
$275.5
$33.4
$242.1
$6.8
38.4%
59 235.3
30-Sep-09
FULL OPERATIONS
& DATA ANALYSIS
CAPABILITY
150
140.1
INITIAL OPERATIONS & DATA ANALYSIS CAPABILITY
FULL SCALE DOM PRODUCTION
100
DETERMINE FULL DEPLOYMENT RATE
85.7
50
DEPLOY INITIAL STRINGS & TANKS
39.5
15.0
COMPLETE EHWD INTEGRATION & TEST
___Total Funds (US & NonUS)
___Original Budget (BCWS)
___Current Budget (BCWS)
93.4 %
___Earned Value (BCWP)
93.4 %
___Actual Cost (ACWP)
92.6 %
0
FY02
FY03
FY04
FY05
FY06
FY07
FISCAL YEAR
1
FY08
FY09
FY10
FY11
Cost and Schedule Performance – The project is 93.4% complete. Remaining contingency is
$6.8 million. There has been no change to the NSF MREFC funding requirements of $242.1
million since the project performance baseline was established in February 2004.
IceCube Neutrino Observatory
Cost Schedule Status Report
Reporting Period Ending: 9/30/2009
Note 1
At
Completion
Cumulative (AY K$)
Note 4
Budgeted Cost 2
Actual Cost
Variance
Work
Work
of Work
Scheduled Performed Performed Schedule
Cost
Risk ContingencyNote5
% Remain
Assigned
Work
Budgeted
AY $s
Complete (%)
Project Support
25,247.3
25,247.3
25,152.9
0.0
94.5
99.8
5.2%
27,165.7
92.9%
Actl
Cost
92.9% 92.6%
Implementation
40,288.5
40,288.5
39,947.4
0.0
341.0
934.0
14.4%
46,758.6
86.2%
86.2%
85.4%
Instrumentation
72,242.4
72,209.3
72,012.4
-33.1
196.9
60.3
7.0%
73,072.2
98.9%
98.8%
98.5%
Data Acquisition
33,658.3
33,658.3
33,666.3
0.0
-8.0
13.3
22.9%
33,716.4
99.8%
99.8%
99.9%
Data Systems
Detector Comm. &
Verification
Pre Operations
25,362.6
25,362.6
25,379.3
0.0
-16.7
26.1
1.9%
26,707.9
95.0%
95.0%
95.0%
20,046.6
20,046.6
19,991.7
0.0
54.9
48.3
7.8%
20,663.7
97.0%
97.0%
96.7%
2,760.5
2,760.5
2,531.3
0.0
229.2
0.0
0.0%
3,533.9
78.1%
78.1%
71.6%
219,606.2
219,573.1
218,681.3
-33.1
891.8
1,181.8
9.8%
231,618.4
94.8%
94.8%
94.4%
30,181.9
30,206.8
28,967.7
24.9
1,239.2
352.1
6.3%
35,775.3
84.4%
84.4%
81.0%
1,040.2
1,040.2
1,040.2
0.0
0.0
13.4
6.0%
1,263.0
82.4%
82.4%
82.4%
250,828.4
250,820.2
248,689.2
-8.2
2,131.0
1,547.3
8.7%
268,656.8
93.4%
93.4%
92.6%
93.4%
93.4%
92.6%
OBS Structure L2
Subtotal
RPSC SUPPORT
NSF
Total
CONTINGENCY Notes 3
IceCube Total Note 2
Sched
Perf
6,846.7
250,828.4
250,820.2
248,689.2
-8.2
2,131.0
1,547.3
6.3%
275,503.5
Notes: 1 Incorporates approved baseline changes.
2 Total Budget at Completion includes
3,732 K non-US contributions over the amount in the post Hartill III baseline.
3 Budgeted contingency is:
38.4%
of the Budgeted cost of work remaining.
4 The BAC (Budget At Completion), reflects PY8-10 detailed Baseline Review.
5 Contingency is assigned based on the Remaining Work, a bottom-up risk assessment, management judgement, and cost
The Schedule Variance at the end of September 2009 was -$8.2k. The cost variance at the end of
September 2009 was $2,131k (compared to $1,844 last month). This favorable variance is
principally due to lower labor and on-ice support costs for Raytheon Polar Services Corporation
and the Air National Guard in FY2009. In addition, training costs for drilling were less than
budgeted (Implementation) and there are still several lagging invoices for cables
(Instrumentation) and computing infrastructure (Pre Operations).
The baseline schedule for string installation is 18 strings in December 2009 – January 2010 and 3
strings in the final season, December 2010 – January 2011. A total of 19 strings were installed last
season. UW and the IceCube collaboration will propose installing six additional strings to
complete the 80-string array originally proposed plus the DeepCore.
Resources are in place for the successful completion of the MREFC project in FY2011.
UW-Madison and the IceCube collaboration submitted revised budgets for M&O for FY20112015. The revisions were made in response to guidance from the NSF and the IceCube
International Oversight & Finance Group (IOFG). Included in the submittal were responses to the
recommendations from the NSF panel that reviewed the original proposal.
2
Contingency Status and Plans ─ Two change requests were implemented in the past month:
IC0157 Additional computer science support by NPX Designs and IC0159 surface-to-DOM cable
assembly at Sea Con. The available Contingency is now $6,847M.
3
Change Log – IceCube Total Project Budget Baseline ($K)
Description
Date
Approved
Total
Baseline
Allocated
Budget
Allocated
Budget
Change
Contingency
Budget
CR149
UA Graduate Student
07/13/09
275,503
268,095
39
7,408
CR150
07/13/09
275,503
267,808
-287
7,696
07/27/09
275,503
267,908
100
7,596
NA
Rebaseline RPSC FY10
budget
Add Capital Equipment to
1.9.4
Status as of June 2009
275,503
267,908
0
7,596
20,069
NA
Status as of July 2009
275,503
267,908
0
7,596
19,036
No.
CR151
CR154
ICL Rooftop access
09/29/09
275,503
267,908
0
7,596
CR155
Data storage upgrade
09/30/09
275,503
268,371
464
7,132
CR156
SUBR distributed computing
09/28/09
275,503
268,409
37
7,095
275,503
268,409
0
7,095
10/29/09
275,503
268,516
107
6,988
11/03/09
275,503
268,657
141
6,847
275,503
268,657
0
6,847
NA
Status as of August 2009
CR157
CR159
NA
Additional Support for
Monitoring South Pole
Computing Systems
Remaining Surface-to-DOM
cables assembly at Sea Con
Status as of September 2009
Budgeted
Cost of Work
Remaining
18,541
17,837
Risk Assessment & Potential Contingency Adjustments
1.
Contingency potentially required for technical, cost and schedule risks associated with the
approved scope of work. Risk assessments are made at WBS-Level 4 to determine the value
of the risk exposure as a percent of the cost of work remaining.
$1,547K
2.
Pre-Operations costs for additional capacity to the data storage and network systems both at
the South Pole and in the North, and for extending software development efforts.
$2,100K
3.
RPSC Estimated FY09 Closeout (-$700K) and moving the support for one string from FY09
to FY11 (+$250K)
RPSC estimate of base cost to support the installation of six strings during the 2010/11
drilling season. The current RPSC baseline already includes support for 80 strings.
-$450K
4.
$1,340K
5.
The cost to retro IceCube equipment/materials from the South Pole at the end of the project in
FY2012. This is a conservative estimate based on Rev. 8 of the RPSC budget.
6.
Potential cost to retain experienced key personnel in the final season to ensure the success of
the last drilling and installation season.
$500K
7.
Potential additional Fuel Cost exposure during the last season of drilling (FY2011)
$450K
Total
Available Contingency as of Sep 30, 2009
4
$1,050K
$6,537K
$6,847K
Drilling and Installation –
South Pole Activities:
 Season Equipment Site drift surveying was completed and excavation begun.
 First wave of drillers scheduled to arrive on-station early November.
 Hose Heating System monitoring continues – no issues.
 Do Not Freeze Modules (Drill Control Center & Tower Operations Structures 1-2)
power/heat monitoring continues – no issues.
Off-Ice Drilling Activities
 Staffing: All driller positions are filled (w/alternates). One driller failed physical
qualification (NPQ’d) and an alternate was secured.
 Staffing: Full-time and seasonal drillers began deploying, with the 1st wave scheduled to
arrive at South Pole Nov 4th.
 Schedule: The Integrated Master Schedule and Master Site Plans are complete – both
reflecting a 20-hole plan, w/19 surface cable trenches, and 14 IceTop Pits.
 Cargo: The last of planned air and vessel shipments to Pole concluded in Oct.
 Materials: Retrograde materials for 10/11 season were procured and shipped via vessel to
McMurdo for pre-staging.
 The Northern Hemisphere Support Group was re-established with 24/7 support numbers
and contacts to expeditiously provide support for project requests originating from South
Pole.
 Procedure (SOP) updates were completed in October.
 Hazard Analysis updates were completed.
 Development of the EHWD User Manual (Turn-Over Package) continued through
October. Script sequences are complete and ready for video shooting at NPX. Drawings
and documentation updates/collection continued through the month. Related activities for
South Pole are identified and planned.
Off-Ice Installation Activities
 Staffing: The 4 primary deployer positions are filled. Two are still pursuing physical
qualification (PQ), but are expected.
 All equipment is shipped; Schedules and staffing plans are in place; Procedures and
hazard analyses are updated.
 The installation team will begin to arrive at the Pole around November 23rd.
 A list of Special Devices is developed. Weekly status meetings continue to confirm most
of these sub-projects will meet ship dates and be ready for installation at North Pole
Station.
Detector Commissioning and Verification - Verification continues to run smoothly. Modified
versions of the Verification scripts for the new operating mode for the Processing and Filtering
(PnF) were prepared and are ready for testing on the South Pole Test System.
No additional progress to report on tests of waveforms into the standard processing.
5
Calibration and Monitoring – Single-LED flasher runs continue to be taken to aid in the study
of ice properties.
As per the discussion, Standard Candle laser data was taken for energy calibration of the IceCube
59 string configuration. These data are being analyzed.
Data Acquisition Hardware & Software – Three software release candidates (“El Farolito”)
were issued in October 2009. In each case, minor bugs were uncovered when the release
candidate was deployed for the 24-hour test mode at South Pole. The bugs were caught within
the first hour of the run so that the eventual impact on clean detector live time was less than
0.5%. At the end of October the IceCube DAQ is still running the Delfina software release from
July 2009.
Sensitive to even this small deduction from the data quality of IceCube, we are continually trying
to develop better ways to utilize the South Pole Testing System (SPTS) in the north to catch
DAQ bugs before the software is implemented on the South Pole System (SPS). This is an ongoing endeavor.
IceCube Detector Operation for Oct 2009
100%
Unscheduled Downtime
90%
Maintainance
Percentage of Day
80%
C&V
70%
Special Tests / commiss. of new strigs
60%
DOM Calibration
50%
Monthly Benchmark/Calibrations
Partial IceCube + AMANDA
40%
Partial IceCube
30%
AMANDA Only
20%
IceCube Only
10%
IceCube+AMANDA
0%
1
3
5
7
9
11 13 15 17 19 21 23 25 27 29 31
Day
Detector Up-Time: 99.0%
IceCube (in-ice) clean runs Up-Time: 95.8%
Unscheduled Downtime: 0.2%
Definition of the terms:
“Detector Up-Time” is the percentage of the time period for which the pDAQ data acquisition was acquiring data
and delivering at least 500Hz of event rate. This uptime measure therefore includes periods in which the detector
was taking data with a partial detector enabled or with light contamination from calibration sources.
“Clean run Up-Time(s)” is the percentage of the time period considered to have pristine data (standard hardware
and software configurations) with the full nominal detector enabled, not contaminated with light from calibration
sources and for which no serious alerts were generated by the monitoring, verification or other. The criteria applied
are not algorithmic but rather represent the Run Coordinator’s overall impression of the quality (including
uniformity) of the runs/data.
6
October was one of the most smooth and trouble-free data taking months this year. The overall
detector uptime was 99.0% and the clean in-ice uptime was 95.8%.
The few normal occurrences of equipment/systems malfunctions (e.g DOR cards giving bad
data, or a DOM hub crashing or hanging up) were promptly alerted by the automatic monitoring
systems and swiftly corrected by Winterover intervention.
Successful special runs for ice properties using flashers were taken. A special trigger test run was
taken to study trigger configurations for next-year to capture data related to slow moving
particles.
Data Handling - South Pole systems in the IceCube Laboratory began the 59-string
configuration physics operation on May 20th and it continues to run smoothly. The figure below
shows the daily satellite data transfer rates in gigabytes per day for October 2009. The purple is
filtered physics data and the green is calibration and test data.
Information technology work focused in two areas: planning for South Pole systems upgrades to
handle the additional strings and data rate in 2009-2010 installation season, and upgrades to data
warehouse and offline processing infrastructure.
In preparation for pole, a detailed plan for South Pole system upgrade and maintenance was put
into place. Work included an upgrade computer systems and the IceCube Lab to support the
increased data volume from 77 strings. Additionally, major upgrades to the P&F filtering system
and the taping system took place. Changes to the taping system will now allow for two taped
copies of data to be made.
In the northern processing centers, two changes to the data processing machines will cause
significant increases in analyzing data coming off the detector. Commissioning of a new high
performance-computing cluster with 512 cores effectively tripling the processing speed. The
7
new processing power will eliminate a significant delay in processing of 40-string configuration
data, which has backed up 59-string configuration data (run completed in May 09) and the
current 77-string configuration data processing. Expansion of the data warehouse by 234 TByte
will also improve processing speeds.
Online & Offline Filtering, Software & Database - The online filtering system for the 59-string
run, which began in May, continues smoothly with daily satellite transmission of filtered data to
the northern data warehouse at the University of Wisconsin. Offline Level 1 and Level 2
processing (L1 and L2) processing in the north is paced by the availability of resources, both
personnel supported via the Core M&O award and In-Kind service labor distributed across the
collaboration.
As predicted, IceTray version 3 is in full release, with a follow up release containing minor
additions to support simulation already prepared. Preliminary support for the conversion to the
snow leopard operating system is now in place. Preparation for the start of IC77 filtering is
underway, with a planned release for deployment to Pole following test runs.
L1/L2 processing of the 40-string configuration run that ended in April 2008 is nearing
completion. The chart below illustrates that there is only about two months of the total of 13
months of remaining to be processed. Processing of the 59-string data run that ended in May
2009 is being prepared and will begin in November thanks to the processing upgrades.
The IceCube 40 string configuration processing status is indicated in the figure below.
Simulation - Improvements continue to be made in the simulation programs. During October, a
new simulation release candidate was made available. Production tests are currently running to
verify if the current release is compatible with previously produced data and to determine if new
features can be used to improve coincident muon simulation.
8
The simulation production plan for the next 6-12 months is under preparation. Currently,
estimates of CPU time and data size are being established, as well as a plan to distribute jobs
across processing sites.
The figure below shows the global simulation production runtime for the last month:
Education and Outreach –Several undergraduate students began their trip to the South Pole as
part of the Upward Bound program, which encourages science experiences for students from
under represented populations. Two University of Wisconsin River Falls (UWRF) students and
one student from UW Rock County will travel on the Oden Cruise ship from Sweden to
Antarctica. They are traveling with a special IceTop tank outfitted with four DOMs to calibrate
the energy response of the tanks to solar particles. The trip will be completed in three legs with
Drew Anderson (UWRF) traveling on the first leg from Sweden to Montevideo, Uruguay.
Samantha Jakel (UW-Rock County) will take over and accompany the tank to McMurdo Station.
Finally, Kyle Jero (UWRF) will monitor the tank from McMurdo to Punta Arenas, Chile. The
students will blog about their experiences at: http://www.uwrf.edu/icecube/oden%20cruise.htm.
UW-Madison is planning an outreach event aimed at a Spanish-speaking audience. Explorando
las Ciencias will be on November 15th and will include hands-on science activities. In addition
to planning and organizing the event, three native Spanish speakers who are part of the IceCube
research team will be at the event to answer questions and interact with visitors.
Quality Assurance and Safety – As deployment begins, the safety team will review procedures
and check structures and processes at the Pole. A major accomplishment this month was the
creation of Enhanced Hot Water Drill (EHWD) checklists. They were printed poster size for
display in the Tower Operations Structure (TOS) and MECC structures at the Pole. The
following nine posters were sent to the Pole:
 Weight Stack Assembly
 Updated EHWD Hose Inspection Criteria
 EHWD Hose Section Change-Out
 EHWD Hose reel Move
 TOS Pre-Drill Set-Up
 Tower/TOS Move
 Heated Hose Specification Poster
 EHWD/SES Camp Water Flow Diagram
 IceCube Lock-Out/Tag-Out Policy
9
Monthly reports are posted at IceCube Monthly Reports.
10