Systems/Software ICM Workshop Acquisition and Process Issues Working Group Rick Selby and Rich Turner Systems/Software ICM Workshop July 14-17, 2008 Washington DC Process & Acquisition Participants ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Rick Selby, Northrop Grumman (co-chair) Rich Turner, Stevens Institute of Technology (co-chair) Steven Wong, Northrop Grumman Ernie Gonzalez, SAF Ray Madachy, USC Matt Rainey, Redstone Huntsville Dan Ingold, USC Dave Beshore, Aerospace Lindsay MacDonald, BAE Systems Blake Ireland, Raytheon Bill Bail, Mitre Barry Boehm, USC John Forbes, OSD/SSA Carlos Galdamez, Boeing Gail Haddock, Aerospace Software/Systems Process and Acquisition Initiatives Software/Systems Process & Acquisition Start-up Initiatives teams Prioritized SW requirements list with cut-points at PDR Methods for driving behavior for SW risk reduction 100% Ideal forms of SW evidence Architecturally significant SW requirements Engage right SW decision makers New SW acquisition approaches 1st SW Build at PDR for key components for SW architecture evidence SW leadership meeting with Chief SW Engineers SW start-up teams SW design reviews are risk- not function-centric Independent SW risk team (non-contractor) 67% TRL framework for SW Payoff Enable parallel, open, competitive environment for SW acquisition SW invariants that you must have in order to adopt ICM 33% Learn from CMMI appraisals for evidence-based reviews Develop approaches to minimize/prevent protests 0% 0% 33% 67% Ease 100% Software/Systems Process and Acquisition Initiatives Initiative Review how CMMI appraisals do evidence-based reviews Define the ideal forms of evidence Develop/enhance TRL framework for SW Define new acquisitions approaches Define the “SW invariants” that you must have in order to adopt ICM Enable parallel, open, competitive environment Develop approaches to minimize/prevent protests Implement “Start-Up Teams” Define methods of driving behavior Convene SW Leadership meeting, including Chief SW Engineers, across all ACAT 1 programs Ensure “right decision-makers” attend and are involved prior to the PDR Design review activity around risks rather than functions Employ independent groups (non-contractor) to identify and investigate the risks Require SW Builds for key (high risk) software components to demonstrate functionality, integration, and that the SW people have explored preliminarily design space Define architecturally significant requirements (which need to be resolved in the first SW release) and map these to risks At PDR, require a prioritized list of requirements/capabilities/features so that the customer can select the “cut point” based on the degree of value, risk, budget, and other new information Payoff Low Med High 1 3 9 Ease Diff Med Easy 1 3 9 Payoff Ease Payoff 15 15 #voters 135 135 Ease #voters 9 1 0 1 1 3 8 0 0 0 4 0 6 2 7 4 5 3 0 3 1 12 7 11 2 6 0 10 13 11 2 8 5 8 4 10 13 0 0 1 5 5 4 4 5 2 0 6 7 7 6 0 4 0 0 0 0 7 6 5 24% 93% 69% 84% 44% 59% 20% 85% 100% 86% 61% 20% 45% 19% 23% 15% 11% 69% 64% 57% 14 13 13 14 10 13 13 13 13 14 13 13 13 12 9 12 13 13 13 13 0 0 0 0 3 4 5 2 12 10 9 10 7 0 0 5 5 11 8 3 1 2 4 5 87% 81% 76% 89% 26% 44% 56% 50% 15 14 14 12 13 13 12 13 0 2 10 7 5 1 89% 26% 12 13 0 0 13 2 8 3 100% 45% 13 13 Ideal Program PDR (pre-Milestone B) ● Attendees – – ● Focus of meeting and method of evaluation – – – ● – Level the playing field in terms of technical knowledge, such as embedding engineers in the contractor organization and making the acquisition personnel more knowledgeable about SW Need to have government/FFRDC/UARC SW tiger teams that go into and help acquisitions and programs, such as from Tri-Service Acquisition Initiative including Start-Up Teams to help launch new programs Risks – – ● Focus on risks (vs functionality) of achieving desired functionally within the proposed architecture Evidence-based review The decision making actually occurs before the milestone; how to empower the Technical knowledge – ● “right decision-makers” attend and will involved prior to the PDR [P 12, 3, 0] [E 0, 1, 2] All success-critical stakeholders engaged, including technical warrant holders who will authorize ultimate deployment Rather than listing the system “functions”, we list the “risks” at the review [P 10, 4, 0] [E 0, 3, 0] Independent groups (non-contractor) identify and investigate the risks [P 9, 5, 0] [E 0, 3, 0] Architecture – – At least one SW Build for each key (such as high risk) software component (maybe CSCI) to demonstrate their functionality and integration, which demonstrates that the SW people have explored preliminarily design space [P 10, 2, 0] [E 3, 0, 0] Need to define architecturally significant requirements (by definition, these architecturally significant risks are addressed in the first release) and map these to risks [P 10, 2, 0] [E 0, 0, 3] • Scalability • Performance – – ● If architecturally significant risks are not addressed, the system “will fail” Somehow put in place the architecturally baseline earlier Requirements – Prioritized list of requirements/capabilities/features that the customer can select the “cut point” based on the degree of value, risk, budget, and other new information [P 13, 0, 0] [E 1, 1, 1] • Incorporate some notion of how to change the requirements to reduce risk – Need to be able to assess whether requirements allocated to configurable items make sense Comments on Draft 5000.2 Language [SA 8, 4, SD 3] ● 3.5.10. A [System] Preliminary Design Review[(s)] (PDR[(s)]) shall be conducted for the candidate design(s) to establish the allocated baseline (hardware, software, human/support systems) and underlying architectures and to define a high-confidence design. All system elements (hardware and software) shall be at a level of maturity commensurate with the PDR entry and exit criteria [as defined in the Systems Engineering Plan]. A successful PDR will [provide {independently validated?} evidence that supports] inform requirements trades [decisions]; [substantiates design decisions;] improve[s] cost [, schedule, and performance] estimation; and identify[ies] remaining design, integration, and manufacturing risks. The PDR shall be conducted at the system level and include user representatives [, technical authority,] and associated certification authorities. The PDR Report shall be provided to the MDA at Milestone B {and include recommended requirements trades based upon an assessment of cost, schedule, and performance risk[s]?}. ● Synergy with ICM – Greater emphasis on risk-driven decisions, evidence, and highconfidence designs Comments on Draft 5000.2 Language [SA 7, 6, SD 1] ● 3.5.10. A Preliminary Design Review (PDR) shall be conducted for the candidate design(s) to establish the allocated baseline (hardware, software, human/support systems) and underlying architectures and to define a high-confidence design. [At PDR, evidence shall be provided that {independently?} validates that a]All system elements (hardware and software) [are] shall be at a level of maturity commensurate with the PDR entry and exit criteria. A successful PDR will [support] inform requirements trades [decisions]; [substantiate design decisions;] improve cost [, schedule, and performance] estimation[es]; and identify remaining design, integration, and manufacturing risks. The PDR shall be conducted at the system level and include user representatives [, technical authority,] and associated certification authorities. The PDR Report shall be provided to the MDA at Milestone B {and include recommended requirements trades based upon an assessment of cost, schedule, and performance risk[s]?}. ● Synergy with ICM – Greater emphasis on risk-driven decisions, evidence, and highconfidence designs Some Quotes for Context Setting ● "The only way we will have large acquisition programs on schedule, within budget, and performing as expected, is for everyone - from Congress down to the suppliers - to all stop lying to each other at the same time." ● "Software's just another specialty discipline and doesn't deserve special attention. Integrating software engineering into the development is the job of the chief system engineer." ● "It takes so long for a program to reach deployment that we are essentially acquiring legacy systems." ● "Spiral process is nothing more than the vee chart rolled up." ● "There is no such thing as an emergent requirement." ● "Evolutionary acquisition is just a ploy to excuse the software guys’ incompetence and let programs spiral forever without having to deliver something." Some Topics for Discussion: Acquisition and Process ● Quality Factor Tradeoffs – Integrating hardware and software quality factor evidence planning and preparation guidelines – Coordinating single-quality-factor IPTs ● Cost and Risk – Budgeting for systems and software risk mitigation – Risk-driven earned value management – Translating shortfalls in feasibility evidence into next-increment risk management plans ● Requirements – Concurrently engineering vs. allocating system, hardware, software, and human factors requirements – Methods for dealing with requirements emergence and rapid change ● Competitive Prototyping – Supporting value-adding continuity of prototype development and evaluation teams ● Topic Specifics – Synchronizing different-length hardware and software increments – Early hardware-software integration: hardware surrogates – Contracting for 3-team developer/V&Ver/next-increment rebaseliner incremental development Incremental Commitment Life Cycle Process Stage I: Definition ©USC-CSSE Stage II: Development and Operations Understanding ICM Model for Software ● Reconciling the milestones – Where are LCO/LCA/IOC and SRR/PDR/CDR? – When are the downselects: 3 to 2, 2 to 1? ● How to drive behavior – RFP language – Award fee – Large carrot (sole winner of major program) ● ● How long does the competitive phase last (ends at Milestone B, ends later, etc)? Create a “whole new contractor role” that gets awarded to the 2-to-1 downselect non-winner – External evaluators come into reviews (“air dropped”) and have a high entry barrier and limited context to achieve success – Loss of valuable expertise in the non-winner – Non-winner becomes the “evaluator” of evidence throughout the program ● ● What kinds of evidence/prototypes are needed for what kinds of risks? Funding – Who pays for pre vs post 2-to-1 downselect (what color)? ● How do you use CP to do: – New approaches for model definition and validation – Quality attribute trades (non-functional) Ranked Summary of Initiatives (High to Low) At PDR, require a prioritized list of requirements/capabilities/features so that the customer can select the “cut point” based on the degree of value, risk, budget, and other new information Define methods of driving behavior Ensure “right decision-makers” attend and are involved prior to the PDR Define the ideal forms of evidence Convene SW Leadership meeting, including Chief SW Engineers, across all ACAT 1 programs Define new acquisitions approaches Design review activity around risks rather than functions Implement “Start-Up Teams” Employ independent groups (non-contractor) to identify and investigate the risks Require SW Builds for key (high risk) software components to demonstrate functionality, integration, and that the SW people have explored preliminarily design space Define architecturally significant requirements (which need to be resolved in the first SW release) and map these to risks Develop/enhance TRL framework for SW Enable parallel, open, competitive environment Review how CMMI appraisals do evidence-based reviews Develop approaches to minimize/prevent protests 117 117 117 109 108 106 102 99 96 96 96 81 69 30 23 Issues - 1 ● John Young was seeing CP as a way to “get the HW right” – He did not expect CP to cause all this discussion about SE/SW ● ● ● What is the order of buying down risk? We currently do evidence-based reviews for CMMI appraisals? [P 1,4,9] [E 2, 0 , 1] How do we change the behavior of both the vendor and acquirer? – Reviewers now “tune out” when the SW architecture presentation is given because it is hard to “bring it to life” ● ● ICM ties together goals of reviewers Navy currently has a six-gate review system – Has an emphasis similar to ICM, including both system and software ● ● ● ICM has “sufficient levels of vagueness”; provides opportunity for tailoring – which is a positive flexible How can we figure how the HW-SW touchpoints? SW has the inherent value of changeability Issues - 2 ● ● ● What are the ideal forms of evidence? [P 12, 0, 1] [E 0, 1, 2] Demonstrating is not a complete answer? Needs to be a validated demo that addresses – – ● ● There are already lots of gates and reviews in place now; but the Army had seven Nunn-McGurdy’s last year The decision makers are not attending the early reviews – – – ● These people are needed, not just surrogates “who just take notes” When do you start addressing these issues and when do you push these issues up the chain The review attendees are “going for the show” not “to do the review” The contractor overwhelms the reviewers in terms of technical knowledge – – ● For example, on the early FCS reviews there were many dog-and-pony shows No/little talk about risks Somehow we need to level the playing field in terms of technical knowledge Need some form of parallel teams Risk: PDRs are currently oriented around functions – – – Rather than listing the system “functions”, we list the “risks” at the review This enables something that the reviewers can focus on The “ranked risk list” becomes a first-class document that is at least as important as other design documents ● At the PEO/IWS (Navy), there is an emerging requirement that prior to System PDR, there will have been at least one SW Build for each CSCI to demonstrate that their functionality and integration ● We should define the “invariants” that you must have in order to adopt ICM We need to make sure that the risks that are currently being presented are honest/accurate – ● – – ● The government reviewers somehow identify the risks, and can empower/contract some teams to address these risks Independent groups (non-contractor) identify and investigate the risks Requirements organization and presentation – – ● Demonstrate performance-criticality functionality Need to define architecturally significant requirements and map these to risks The first release (“indivisible build 1”) needs to address all architecturally significant risks Take a fraction of the predictable overruns (50-100%) and spend it up front to reduce risks Issues - 3 ● Need TRL framework for SW? [P 7, 6, 0] [E 0, 2, 1] – – – – ● ● SMC uses Key Decision Points (KDPs) as the major decision milestones, and their KDP-B occurs after the SW reviews now Need to change the attitude of the senior acquisition and policy decision makers (above chief engieer level) – – ● – – – – ● ● Development tools as well as systems How to address assurance??? Government has “open access” to disk farm where all development artifacts (req, design, source code, test code, etc) are stored/developed and therefore can inspect/analyze Have common test beds – – ● Navy sonar systems have periodic re-bidding approach/process where contractors continual re-bid on new capabilities “technology refresh cycle” One inhibitor: How to protect IP that is a discriminator for contractors corporations, including the underlying methods for producing the products Boeing made middleware for UAV open source shared across military contractors ELV Atlas-V Ground Control System uses Linux libraries Naval open architecture for open systems (including shared code) initiative, including contract terms and licensing Domain-based “members-only” open source models How about moving toward an open source model – – ● SW illiteracy exists at the highest levels, such as arguments about whether to do SDPs Need to re-instate the original language that was proposed for the DoD 5000 revision Need to think broadly about new acquisitions approaches, such as moving away from fee-on-labor cost-plus contracting vehicles to “new incentive models” [P 11, 2, 1] [E 0, 0, 3] – – – ● Maybe do not call this framework “TRL” because of confusion with existing HW-centric TRLs MDA has SWRLs (software readiness levels) now and it works pretty well Navy has ratings for process and functionality, analogous to TRLs Interface level readiness too (from IDA workshop April 2008) Such as original Ada validation suite Define common test beds that also provide meeting place and communities to interact DDR&E has several development and test beds environments now that enable the SBIRs teams to develop products How to address unknown unknowns – Making unknown unknowns known Issues - 4 ● ● Enable parallel, open, competitive environment [P 6, 4, 3] [E 0, 0, 3] When acquiring new system, adopt an members-only open source model: – – Standard middleware (think RT Linux for ELV Atlas-V ground system and now NASA Aries ground) Apple-like AppStore for developers to develop and sell applications; Google has gadgets (free), Microsoft has gadgets (free), Yahoo has widgets (free) • Must have some sequence of gates to ensure that SW was “reasonable” / do-no-harm • Members-only contributors • SW is “low cost” but need to pay for support • Multi-tier pricing scheme for execute only, source for customers, source for all – – – ● Acquirers can select/purchase the applications that have the most value What are the incentives for contractors to invest for developing these applications? Example: ACE/TAO is open source middleware that is being used on Navy SSDS large-deck combat systems • Enables new potential competitors because of externally known interfaces Architecture would need to be able to accommodate this “new thinking” Issues - 5 ● What is the earliest to end the Competitive Prototyping? – Sometime between Milestone A and B ● What is the latest to end the Competitive Prototyping? – You can build 2 or more complete systems by keeping competition going throughout the lifecycle ● You continue the competition until the decision makers (and success critical stakeholders) have sufficient evidence that the risks are “acceptable enough” to go forward with one contractor – You can possibly re-open the competition later for some aspects of the program ● ● ● ● ● ● The current working assumption is that you can downselect to one contractor at Milestone B Can we gain knowledge over time? Once a winner is selected, you want to hit-the-ground running and not lose any time and talent How to do we minimize/prevent protests [P 0, 5, 8] [E 0, 0, 3] Will the early rounds of prototyping show you enough evidence to justify going with the a Sole Source award (and therefore avoid protests) Milestone B brings on new requirements and formal briefings to Congress – – – – Right now, the government declares the budget before Milestone B It is very difficult to re-certify programs when you exceed 25% of the original budget Most programs who Nunn-McCurdy once, do it again because of staff loss, etc Government wants cost and schedule realism Issues - 6 ● ● Do we need to have multiple Milestone B’s and/or multiple PDRs? Start-Up Teams: Recommend that SW Tiger Teams are engaged in the SEP development (which occurs prior to Milestone B because the SEP defines the entry and exit criteria for the PDR that occurs prior to Milestone B). [P 10, 3, 0] [E 3, 0, 0] – Update the current SEP preparation guide – Many parts are HW-centric or HW-exclusive – SWAT Team = external team of experts that “help” PEO prior to Milestone B ● Driving behavior [P 13, 0, 0] [E 0, 3, 0] – What to define in the RFP, award fee language – Emphasize usability to government and contractor ● SW Leadership meeting across all ACAT 1 programs including Chief SW Engineers [P 11, 3, 0] [E 3, 0, 0] – Include Chief System Engineers, both government and contactor – Maybe hold this at the NDIA Sys Engr meeting