Identifying Enterprise Leverage Points in Defense Acquisition Program Performance Robb Wirthlin, Major, USAF PhD Candidate January 21, 2009 The views expressed in this work are those of the author and do not reflect the official policy or position of the United States Air Force, the Department of Defense (DoD), or the U.S. Government. Early Research Observations • Anomalous behaviors of acquisition are in-fact emergent behaviors of a much larger, complex system • Prior studies suggest many of the ‘pathologies’ outside of the control of the acquisition managers • These studies point out a need to understand the entire system • To cut any piece out (like acquisition) and study it • exclusively would preclude finding a full set of root causes Therefore, I began to look “outside” of acquisition http://lean.mit.edu © 2009 Massachusetts Institute of Technology LAI Research Summit 01/21/09- 2 Challenge: Find an appropriate representation for the Enterprise • Here’s what I did • Borrowed ideas and concepts from Value Stream • • • diagramming Used commonly accepted understandings of probabilities and occurrence Represented activity in frame of reference understandable to target audience: military and civil servants in AF (used DAU “Wall Chart” organizing framework) Scoped it to a reasonable abstraction and size • • • http://lean.mit.edu Main unit of measurement is a “program” Restricted to ACAT I, II, and III programs Limited to Milestone C and earlier in Acquisition parlance © 2009 Massachusetts Institute of Technology LAI Research Summit 01/21/09- 3 Top Level Abstraction A Representation of the Enterprise of “Cradle to Grave” Acquisition in the US Air Force Swim Lane Pre-MS “A” Pre-MS “B” (Concept Refinement) (Technology Development) Pre-MS “C” (System Development & Demonstration) Pre-Full Rate Production (Production & Deployment) Operations and Sustainment User Requirements Money Acquisition Scope of Model Contractor Time http://lean.mit.edu © 2009 Massachusetts Institute of Technology LAI Research Summit 01/21/09- 4 The Enterprise Model Pre-MS A Pre-MS B Pre-MS C Requirements Swimlane Requirements Swimlane Requirements Swimlane W ait fort echnical results W ait f or tec hnical r esult s Put n i to moderniz at o i n pla nning and send to Ac quisition MDAP Thres hold crossed? t o A cquis ti ion Moder nization/Sus tain ment Ac tivity (n i dividual or g) Yes Pr epar e Concept of Oper at o i n Capability Roadmap "wis h list " for budget s ubmission Thr eshold Object vi e Tradeof f s CCD No Y Draft Br iefing and materials D ecision t o pur sue r equir ements "Coor d" and "Concur " (n i dividual or ganiz at o i ns) (n i dividual or g) MAJCOM "A" Lett er s Update and schedule calendar Y Pre- RSR ( in dividual or g) Y Fn i aliz e RSR & Calendar it ems RSR ( Individual or g) Mr. Harr y Disbr ow H Q USAF A8 MAJCOM A8 Cont n i ue Y I n s cope of exist ing doc uments? Form High Per formance Team (HPT) HQ USAF A8 or g (indiv d i ual org) D raf t C CD Rout e t o pr oper organization Ext er nal input KPP Development Decision to purs ue requir ement s (individual or g) ( Requir ements Policy Shop) Draf t Br e i fing and materials "C oor d" and "Concur " (individual or ganiz ations) Y MAJCOM "A" Lett er s Update and schedule cale ndar Pr e- RSR (individual or g) Finaliz e RS R & Calendar items RSR ( Individual or g) Mr. Harr y Dis br ow H Q USAF A8 N Ot her n i put s Decision t o pur sue r equir ements ( in dividual or g) Proces s St ar t Y MAJCOM A8 Requirement s Pr oc ess Input s fr om v arious sources For exist ing progr am? (Requir ement s Policy shop) No Draft Br iefing and materials "Coord" and "Conc ur" (n i dividual or ganiz at o i ns) MAJ COM "A" Let ters Y Updat e and sc hedule c alendar Pre-RSR ( individual org) MAJCOM A8 Y Finaliz e RSR & Calendar it ems RSR (I ndividual org) Mr. Harr y Disbr ow HQ USAF A8 Fundin g available? Y Conduct St udy or Analysis (individual or g) ( in dividual or g) Decision t o pur sue r equir ement s Dr af t Br e i fing and mater ials (n i dividual or g) (indiv d i ual or ganizations) "Coord" and "C oncur" MAJCOM "A" Lett er s Updat e and schedule c alendar Pre- RSR ( in dividual or g) MAJCOM A8 Rout e to Advanc ed concepts (Requirement s Polic y Shop) No N N N AFROC pr eparations AFR OC decision? MAJCOM or g AFR OC For m High Perf ormance Team ( HPT) HQ USAF A8 or g Update br ief and mater ials ( Individual or g) Y Hg i h P erf or mance Team ( MPT) wor k Dr af t Document ( ICD) H Q USAF A8 org Air Staff Pr oc ess Crit ical Comment s? HQ AF A5R D HQ AF A5R D Finalize RSR & Calendar items RSR ( Individual or g) Mr. Harr y Disbr ow HQ USAF A8 N No ( in dividual or g) Joint inter es t? AFROC OSD N N Air Staff Pr ocess High Perf or mance Team ( MPT) wor k Pr epare f or Milest one Rev iew I nf o t o Acquis ition about CONOPS capabilit y "wish lis ts " Yes Int er act ion with Acquisition Update brief and materials More Detailed r esult s fr om Prot ot ypes/ Engineering Models Acquisition Res ults ( I ndividual or g) Y Dr af t Document ( ICD or draf t CDD) Cr ti ical C omment s? AFROC prepar at o i ns AFROC decision? HQ A F A5RD MAJ COM or g AFROC Prepar e f or Milest one Review Yes CPD? J oint Pr oces s J8 Y C ritical Comments ? J8 Ar chive N Individual or g N MAJCOM org N J8 Dec ide to pursue OSD Comment Res olution MAJ COM approval? N Cr itical Comment s? Inf or mation t o Acquis ition Sys tem for prepat or y w ork Y Join t interest? N Y No Inf or mation t o Acquis ition Sys tem for prepat or y w ork Ar chiv e J8 Per sonnel fr om A cquis ti ion of fice wor king on t his par ticular ac tivity Y Indiv idual or g Join t Process N D ecide t o pur sue I ndividual org HQ AF A5RD Draft Document ( CDD) Decide t o pur sue Y CDD? MAJCOM org MA JCOM appr oval? Y N No No N HQ USAF A8 org Y Comment Resolu tion Dr af t Doc ument ( CDD) HQ USAF A8 org ( Individual or g) HQ USAF A8 or g ( Individual or g) N J8 JROC Prepar ations JROC decision? MAJCOM org JROC Y Details of Pr ef er red Syst em Concept ( Preliminar y) JROC Pr epar at ions N MAJ COM or g Yes J8 Y AFROC decision? MAJC OM or g High Perf ormance Team ( MPT ) wor k Update br ief and materials High Per formance Team (MPT) work Joint Proc ess Cr ti ical Comment s? Comment Resolu tion JROC dec si o i n? Y JROC N I CD? MAJCOM org I nf or mation t o Acquis ti ion Syst em f or prepat ory work J ROC Pr eparat ions N AFROC prepar ations HQ AF A5RD N No Mor e Acquis ti ion Res ults No Y Y MAJCOM appr ov al? Crit ci al Comments ? HQ AF A5RD No F orm High Per for manc e Team (HPT) HQ USAF A8 org Y N N No N Air S taf f Pr oces s Y Update br ief and mater ials I ndividual org A rchiv e N No, because of work alr eady done conduc t AoA? Preliminar y Results fr om Acquis it ion OSD N Obt ain funding f rom Adv anced Conc ept s Branch Decide to purs ue Y Joint int er est ? No Y Form High Per for manc e Team (HPT) HQ USAF A8 org Y N MAJ COM or g JROC dec si o i n? Y No N Ar chive N JROC N Budgeting and Programming Swimlane Budgeting and Programming Swimlane Budgeting and Programming Swimlane Go to "Prepar e Courses of Act ion" ( Acquis ti ion Step) Repeat n+2 t m i es Repeat n+2 times AF lev el pr oc ess MAJCOM PEM Decision to pur sue Adv anc ed Conc ept s Budget Reques t Nex t st ep? MAJCOM Panel MA JCOM Gr oup Next St ep? MAJCOM Council Next St ep? Y Pr ogr ammin g Iss ues & Budget ing Hear ings PBD MAJ COM PEM D ecision t o pursue Advanced Concepts Budget Request Y or N Y or N OMB & Congr ess N law Yes changes t o BES? MAJCOM P EM Decis io n t o pur sue Adv anced Conc ept s Budget Reques t BES DoD POM? Y Next s tep? MAJ COM Panel Y or N Go t o "Pr epare Cour ses of Act o i n" (Ac quisit o i n Step) Go t o "Pr epar e Cour ses of Ac tion" ( Acquisition St ep) Repeat n+2 times Yes Yes - inf or mation only Y PDM? Next St ep? MAJCOM Group Next St ep? MA JCOM Council Nex t Step? AF lev el process Next St ep? OSD Pr ocess Y Next st ep? MAJC OM Panel Y es Yes - infor mat o i n only Y PDM? MAJCOM Gr oup Nex t Step? MAJC OM Council Next St ep? AF lev el proc ess Next St ep? OSD Pr oces s Congr es s a l w N Yes Yes PBD Yes - inf ormation only Y or N Y or N Y or N Y or N c hanges t o BES? Congres s N law Yes changes t o BES? BES BES DoD POM? Y PDM? PBD OSD DoD POM? Y or N OMB Y or N OMB Yes Retr y? R etr y? Ret ry ? No Ar chive Ar chiv e Archive R eleas e moneys to Acquisition To Budget inputs Release money s to Ac quisit o i n Release money s to Acquisition To Budget inputs To Budget inputs To A cquisit ion Fallo ut of B udget Proces s m i pact n i g planning To Acquis ti ion Fallout of Budget Pr ocess m i pact ing plannin g To Acquisition Fallout of Budget Pr oc ess impac ting planning Full Funding in FYDP Full Funding in FYDP KPPs f rom Requirement s Acquisition Pr eparation f or Acquisition Panels Acquisition Acquisition Panels Yes MDA A ppr oval? Milest one A Ac quisit o i n Panel (DAB , MD A, ADM, et c) Af fordabilit y Assessment Pr eparat ion for Acquisition Panels Ac quisit o i n Program Bas elin e Acquisition Acquisition Panels Pr eparation f or Acquisition Panels Ac quisit o i n Panels Yes MDA Appr oval? KPPs f rom Requirement s Milest one C Program Of fice Cos t Est m i ate Cont ract or Cos t Est m i at e No Af fordability Ass essment Independent Cost Est imate Pr epare f or Acquisition Panels No Acquis ti ion Pr ogr am Baseline Y es MDA Approval? N Contr ac t Management F rom Requir ements sy st em: Dr aft document f or basis of wor k RFP r elease and Sourc e Select ion From Requirement s No Dr af t RFP Prepar ation Y Prot es t? To MAJCOM PEM Budget input Yes Funds Redir ect ed RFP r elease and Sourc e S elect ion Dr aft RF P Pr eparation Scope and Awar d Technology D evelopment c ont racts I nt er nally or ex ter nally gener ated mult iple times per y ear Periodic Rev iews RFP c oor dination pr ocess Inter nally or externally gener at ed multiple times per year Pr epar e Progr am budget dec si ion (PBD) Upheld? To MA JCOM PEM Budget input Yes Delay No Ar e Obs & Exp OK? No Yes Upheld? To MAJC OM PEM B udget input Yes Repeat "n" years No Scope and Award Sys tem Development and Demonst ration contr ac ts (or AoA) Yes RFP coor dination pr oc ess Repeat "n" y ear s Yes Are Obs & Exp OK? No A cquis ti ion Planning Ac tivit e i s No Pr otest ? Contr act Management No RFP r eleas e and Sourc e Select o i n Repeat "n" years From Planning & Budgeting C ont ract Management RFP coordination proc ess Int er nally or ex ternally gener at ed mult iple times per year Scope and Award St udy contr ac ts ( or AoA) Delay Yes Funds Redirec ted No Per o i dic Reviews Des ign Rediness Review?/ MDA Appr oval Yes YES No scope growth or tec hnical pr oblems Seek funds? PEM/other st aff find moneys Yes No Obt ain Funds? ( timely) No Ar e Obs & Exp OK? No No Sourc e Select io n Pla ns Yes Funds Redir ect ed No Per iodic Rev e i ws Yes Yes To Requirement s sc ope gr owt h or t echnic al problems Prepar e C our ses of Ac tion Prefer red Sys tem Concept Funding Shor t fall Cont ract Change Requir ed? s cope growth or t echnical pr oblems Seek f unds? PE M/ ot her s taff f ind moneys Yes No Obtain Funds? ( timely) No Seek funds ? Yes PEM/other st af f fn i d money s No Obtain Funds? (t m i ely ) No Yes Prepar e Cour ses of Act o i n Source Sele ct o i nP a l ns Y es Yes Funding Short fall P repar e Cour ses of Ac tion Sys tem Per for manc e Specif ci ation Contr act Change Required? To R equir ements Init ial Rate pr oduct o i n Baseline Yes Ac quisit o i n Panels Fundin g Shor tf all Cont rac t Change R equir ed? End loop Yes Adjus t Program and Cont ract (c ost /s chedule est imates rev si ed) .. . End loop P repar ation f or Ac quisit o i n Panels E nd o l op To Requir ement s Yes Change Cont rac t/ Resc ope ef for t ( cost / schedule impact ) Bes t Mat erial Appr oac hes Yes Change Cont ract /Res cope eff ort ( cost/ sc hedule impact) Dif fer ent Distr ibut ion pr ofile and e l ngt h dependin g upon path t aken SRR? Adjust P rogram and Cont rac t (cost /sc hedule est m i ates r evis ed) .. . Change Contr ac t/ Resc ope eff or t (c ost /s chedule impact ) Pr eliminary Integrated Ar chetic tur e No No Diff er ent Dis tr b i ution profile and lengt h depending upon path taken To Requir ements Engin eer ing Reviews? Sys tems Engineer ing To Ev ent Happens st ep To Sy stems E ngineering s tep Rewor k and Dela y Yes Dif fer ent Dis tr ibut o i n profile and e l ngt h depending upon path t aken Acquisition Planning Act vi ti ies Syst em Perf ormance Specific ation A djust Pr ogr am and Contr act ( cost/ sc hedule est m i at es r evised). . . No To Event Happens s tep Sys tems Engineer ing Dat a f rom User St udie s Test Readeiness Review Yes Yes Make Trades ? Pr ef er red Sys tem Concept No No Sys tem Ver ification Review? To Requirements Acquis ition Plannin g Act vi ti ies Sy st ems Engineering P rot ot ypes/ Engineer n i g Models Syst ems Engin eer ing To Ev ent Happens st ep To Requirement s To Event Happens To Event Happens To Sys tems Engin eer ing To S yst ems Engineer n i g St ep To Event Happens To Sys tems Engineer ing To Event Happens To Syst ems Engin eer ing Pr ot ot ypes/ Engineering Models Contractors Contractors Perf or m wor k Cont r act St ar t- up Event Happens Contr ac t Start -up Sc ope Growth/ Technic al Pr oblems ? Rew ork and Delay Perf or m wor k To Ev ent Happens st ep Rework and Delay Rewor k and Dela y Rework and Dela y Perf or m wor k Cont ract St art - up Event Happens Contractors Sc ope Growth/ Technic al Pr oblems ? YES YES YES and NO NO Fabricat ion No T ask Complete? Assembly I ntegrated t es ting Development al t est ing (S yst em), Liv e Fir e Test , Operat ional Ass essment Combined Test ing (DT&E, OT& E, LFT&E) To "Prepar e Cour ses of Action" in Acquisition Swimlane No Task Comple te? To "Prepar e Cour ses of Act o i n" in Acquisition Swimlane Ev ent Happens Scope Growth/ Tec hnical Problems? Yes To Pr eferr ed Sys tem Concept YES Yes Yes of NO To SRR No Task Complet e? To "P repar e Cour ses of Ac tion" in Acquis ti ion Swimla ne Yes To Prototype Engineering Models Follow pat h at appr opr a i te st age Output consists of time elapsed and actual cost for single program - and tracks comparison to proposed time required and original proposed costs http://lean.mit.edu Sour ce Select ion Plans Draft RFP Pr eparat ion Milest one B Approval? © 2009 Massachusetts Institute of Technology LAI Research Summit 01/21/09- 5 Close-up view RSR – Decision Point -Sources: Official Docs, Interviews (Krussel, Moen) -Probability: 98% Funding Available? – Decision point -Sources: Interviews (Krussel, Schilling, Duda ) -Probability: 80% http://lean.mit.edu Model Design: Every decision point, every process task, is documented and sourced; now validated RSR Mr. Harry Disbrow HQ USAF A8 N Funding available? (individual org) Y Conduct study or analysis – Task -Sources: Official docs, Interviews (Krussel, Duda) -- Time Distribution: 45 to 180 days; binomial, p=0.40 Conduct Study or Analysis (individual org) N © 2009 Massachusetts Institute of Technology LAI Research Summit 01/21/09- 6 Discussion and Policy Implications • A Few examples • PDR (and new DoD 5000.2 response) • Rework modeling (explicit vs. implicit) • Variance at local levels vs. Enterprise variance • Undocumented handoffs, waiting periods • Policy Implications • Consider ^ overlay of staffing profiles to Enterprise • • model behavior Consider ^ error bars of program vs. point estimates ^ Oversight functions, managing risk, value-added govt workforce activities http://lean.mit.edu © 2009 Massachusetts Institute of Technology LAI Research Summit 01/21/09- 7 Future Work • Candidate issues to pursue: • How much time is spent “waiting” in the system? • What is the cost of waiting? • How can the ratio of waiting to non-waiting taks be improved? • Which tasks do the data suggest have a more influential role in taking time (or cost) in the overall process? • • What is the overall impact of a process’ time to Enterprise outcomes? Where can an Enterprise leader suggest “investments” into the process with the greatest impact to cycle time or cost reduction? http://lean.mit.edu © 2009 Massachusetts Institute of Technology LAI Research Summit 01/21/09- 8 Summary • • • Introduction to problem and motivation Model development Discussion and Implications http://lean.mit.edu © 2009 Massachusetts Institute of Technology LAI Research Summit 01/21/09- 9