October 16, 2013 presentation to the NAOJ Workshop




Funds every field of science – mostly as grants to individual researchers at US universities and non-profit organizations
~ $7B annual budget:
 ~$1.3B for Mathematical and Physical Sciences (MPS)
 About $260M for Astronomy


More than $130M annually to operate existing astronomical observatories
Fraction spent on infrastructure increasing due to Atacama Large
Millimeter Array (ALMA) operations and expenditures for mid-scale instrumentation and infrastructure.
Future budget outlook: ~flat
 Physics and Astronomy outlook: declining
 multi-disciplinary initiatives and new applied research programs compete with core disciplinary research, reducing core programs in recent years
2

 Some have been spectacular successes:
Cornell Electron Storage Ring (CESR)
-- high-luminosity 6+6 GeV electronpositron collider at Cornell University
> 30 years operation
And some were not: Project Mohole - an ambitious attempt to drill through the
Earth's crust into the Mohorovicic discontinuity
3

 Six large facilities under construction:
 Advanced LIGO, Advanced Technology Solar Telescope, Alaska Region
Research Vessel, Atacama Large Millimeter Array (with Japan, and nearly complete!), National Ecological Observatory Network, Ocean
Observatories Initiative
 Two recently completed smaller scale facilities:
 National Center for Atmospheric Research’s Wyoming Supercomputer
Center, ALVIN Replacement Human Occupied Vehicle
 Large Synoptic Survey Telescope One may start construction in
FY 2014


Depends on Congressional funding
Planned complete in 2022?
4







NSF doesn’t have a long-term “facility roadmap” of future plans
NSF doesn’t manage big projects or operate big facilities (except in
Antarctica)
NSF oversees planning, construction, and operation, but the skilled project managers and technical staff work for a university or non-profit organization (the “awardee’), who is responsible for project accomplishment
Construction funded through a separate Congressional appropriation
The awardee contracts with major industrial firms to design and build the project. NSF expects awardee to be capable of managing very large
(>$100M) subcontracts to industry
Funding to Awardee via “Cooperative Agreement,” not a contract. This means NSF has substantial project involvement. For example:





NSF shares project risk with Awardee, necessitates risk management plan
NSF can make management changes at Awardee
NSF has latitude to change scope and budget, if necessary
NSF approves major expenditures and significant changes to plan
NSF requires monthly progress reporting of financial and technical status
5




Project planning
 A new project must have a credible budget estimate, a resource-loaded schedule, a sound management plan, and a core team capable of accomplishing the project
Assessment and evaluation
 NSF asks independent experts to evaluate the merits of the proposed science, and the soundness of the project plan, including the budget estimate
NSF prioritization and budget planning


The most important factor for NSF is whether it can afford to operate the new facility after construction is complete
In a flat budget environment, this means that there is a competition between new opportunities and areas of ongoing support – what is the most important science?
 The next few slides provide more details about these 3 topics
6

Horizon planning and
Conceptual Design
Preliminary
Design
Final
Design
Construction Operations
 NSF expects projects to develop planning using sequentially increasing maturity stages:



Horizon: Define Quantitative Science Goals and Objectives; derive functional requirements and top level technical requirements
Develop parametric “top down” cost and schedule estimates, based on scaling historical data  Conceptual Design Report
Develop “bottom up” cost and schedule estimates, based on detailed engineering design, site specific factors, input from industry, interconnection of functional elements, estimates of risk and uncertainty, Project Execution Plan
 Preliminary Design Report
 Develop “ready to build” plan, based substantially on supplier estimates, recruit key staff, secure site permits  Final Design Report
 Conceptual Design is like the outline of a book – with all the structure and chapter headings. The Preliminary Design and Final Design activities fill in the details. Final Design Report = instruction manual for building facility.
 Generally similar processes at US Department of Energy and NASA
7





Horizon Assessment: NSF internally prioritizes among candidate projects. NSF Director authorizes NSF funding of activities leading to
Conceptual Design Report
Conceptual Design Review: Results in rough definition of project scope and coarse estimate (>50% uncertainty) of construction budget. National Science Board (NSB) authorizes significant funds for planning to develop Preliminary Design Report
Preliminary Design Review : Sets “reference baseline” definition of scope, budget, schedule, and risk . If review committee satisfied, and
NSB continues to place high priority on the project, request
Presidential concurrence (Office of Management and Budget) followed by Congressional appropriation of $
Final Design Review: Assesses whether project is ready to start construction. If review committee satisfied, NSB authorizes project team to start spending construction money
8

 National Science Board takes on the very challenging task of prioritizing new facility projects across disciplines


Extremely hard: telescopes vs ships comparison
OMB/Congress expectation that annual construction budget <
$200M, so resource loaded schedule and annual construction budget projections are critical factors
  Need high confidence in Total Project Cost estimate, but also well supported annual budget estimates for each project year!



NSF and NSB are reactive: strong, well organized and continuous community advocacy is very important influence on prioritizing
Broader societal impacts are a big factor in decision: number of users, education and training opportunities, open data access
NSB can and does re-prioritize
9



Cost estimates are inherently biased too low:


They tend to omit or overlook necessary scope
They are usually over-optimistic regarding technical difficulty of planned work
NSF has a “no-overrun policy” – budget shortfalls in construction arising from known risks must be mitigated within budget, and by de-scoping if necessary. Strong reluctance to go back to Congress and ask for more $


But NSF is not crazy – ATST budget augmented by $34M this year to recognize cost increases due to prolonged legal challenges to
Hawaiian site use
Message to communities not to bring an immature project plan and overoptimistic budget request to NSF
10



 But operating cost projections are almost always biased low because they overlook essential scope and underestimate technical difficulty and risk
Perform full life-cycle cost estimation early, and update regularly:
 Operating cost projections


Budgeting for science use
Termination and contingent liabilities (partnership withdrawal, for example)
12




 Construction activity requires a big pre-construction investment (5-
25% of total capital cost, and studies have shown that greater preconstruction spending results in more accurate predictions of actual project cost)
Project management is ~10% of Total Project Cost (TPC)
From idea to groundbreaking requires ~10-20 years
Earned Value Management (EVM) Reporting is one of many important diagnostic indicators:




By the time a project is 10% complete, EVM cost/schedule variances should be meaningful.
Good managers use EVM as a management tool, not just a reporting tool
Early cost/schedule problems are leading indicators of bigger problems later – pay attention to early warnings!
Costs to operate industrial strength project management controls software ~1-2% of total construction cost. It’s worth it!
13

 Risk adjusted cost is for “known unknowns”


Known unknowns = risks known to the cost estimators
This includes provisions for factors such as commodity price volatility, foreign exchange uncertainty, and uncertain start dates due to delays, including those caused by Federal Gov.
 Uncertainty in Federal appropriation process and schedule are part of the landscape for Project
Management and budgeting, include these factors in the risk-adjusted budget
14




Awardee’s Project Execution Plans are expected to follow the
Project Management Body of Knowledge (PMBOK) to develop a robust Risk Management Plan (5 step process)
Budget and schedule estimates are similar to physical measurements; they have values and uncertainties . Each uncertainty can have random and systematic components
NSF develops its own “Internal Management Plan” to manage its own risk in overseeing the project:


Identify risks to agency, especially early indicators of future trouble, possible remediation
“Unknown unknowns” (=risks unknown to the cost estimator) are funding agency’s responsibility, and are dealt with at program policy level.  Maintain flexibility and reserve capacity to adopt alternate courses of action in future planning
15





International collaboration may make it possible to undertake new projects too expensive for any single country alone
Coordination with project planning and assessment becomes even more challenging with multiple funding partners!
Partnership agreements must recognize the unique roles and responsibilities of each partner, and NOT treat partners symmetrically
 For example, NSF funds and oversees large projects, but does not manage or directly operate them
US strongly endorses open researcher access to facilities based on merit review.
 Opposition to European “Pay to play” model. Experimental groups do not contribute to operating costs of US accelerators
(ICFA guidelines), telescopes, etc.
16






US open data access policies may create partnership disincentives in some cases: why pay for operations when you can get the data for free?
Uncertain annual appropriation processes are true for many countries, and complicate construction and operation budgeting
Partnership agreements need to promote long term stability
International partnership agreements must enable individual partners to exercise their expected stewardship roles as overseers of sovereign funds
Big projects are inherently part of political dialogue because of the size of projected budgets
 Projects have foundered when political influence has resulted in premature project start with incomplete plans (ITER, SSC, DUSEL) and there has been painful re-scoping with others (ALMA, …)
17







Partnership agreements are founded on good will – structure must be
carefully planned to align interests to avoid destabilizing collaboration.
Include means for prompt dispute resolution!
International partnerships have intrinsic overhead costs that must be recognized, and different partners have different costs. In-kind contributions may have a cost to the other partners.
Robust systems engineering , spanning partner boundaries, is essential
Conduct and manage a single project rather than parallel activities
Need to synchronize development process when there are other partners or potential partners
Define the appropriate governance models for construction and operation
 Extraordinary projects are successful when led by extraordinary people. Detailed policies and agreements don’t compensate for this.
18

 Annual NSF facility plan – see on-line meeting materials for NSB February board meetings
 NSF Large Facilities Manual – available at http://www.nsf.gov/publications/pub_summ.jsp?ods_key=lfm
 GAO Cost Assessment Guide – see especially the discussions of cost and contingency estimation, and the many helpful case studies
 PMBOK – see especially its discussion of risk management
 AACE-I – international organization that describes best practices for cost and risk estimation
19