ESS
2008
Poster Session
MIT ESD
Characterizing the Enterprise of
Military Systems Acquisition
Ph.D. Program
Start date: September 2005
Thesis advisor: W. Seering, Professor
Robb Wirthlin, Ph.D. Candidate
Research Group: Lean Product Development, Lean
Committee: W. Seering, Professor; S. Widnall, Professor;
Advancement Initiative
D. Lessard, Professor
"The views expressed on this poster are those of the author and do not reflect the official policy or position of the United States Air Force, Department of Defense, or the U.S. Government. "
Motivation / Problem
Methodology
Chronic execution problems for large, complex, systems
acquisitions
• Department of Defense Acquisition
• A systemic issue: 30% - 40% cost and schedule overrun on
average (Source: Summary Selected Acquisition Reports,
last 30 years; Biery, Miller & Lessard, others)
Conducting grounded research using Social Science Research methods to
characterize the AF Acquisition system. Over 50 interviews of key players
within system talking about risk and portfolios of systems
> Borrowed ideas and concepts from Value Stream mapping
> Using commonly accepted understandings of risk; probabilities and
occurrence
> Represented activity in frame of reference understandable to target
audience: military and civil servants in AF
– Main unit of measurement is a “program”
– Restricted to ACAT I, II, and III programs; Limited to Milestone C and
earlier in Acquisition phase parlance
Literature suggests Risk is part of the problem
• Risk is underappreciated in large, complex projects (see
Megaprojects and Risk by Flyvbjerg)
• The GAO, RAND, IDA and other groups have recently told DoD
this too.
Preliminary Results
•Interview analyses suggests many root causes of Acquisition
anomalous behaviors originate outside of the formal Acquisition
system
•Concept of managing through portfolios is immature and
portfolio risk understanding is primitive outside Acquisition as
well
Remaining Research
•Finish the model Verification and Validation, program the model;
run and debug
•Examine 2 or 3 key questions using the model with priority given
to questions dealing with portfolios and risk
• For example: What is the overall process yield? What fraction
of time spent is value-added? What is the cost of waiting?
What interventions require the least disturbance to the
existing system?
The Research
Risk
• For some, it means “almost anything that can go wrong”
• Others see opportunities in Risk
• Still others specify it further: political risk, technical risk,
organizational risk, technology risk, etc.
Model Design: Every decision point, every process task, where
possible, is thoroughly documented and sourced
Key Question(s)
RSR – Decision
Point
-Sources: Official
Docs, Interviews
-Probability: 98%
Is Risk at the Heart of Acquisition Problems?
> Is risk really the problem? If so, we should manage risk
better, right?
> And how do we manage risk better? What is the answer?
> Would using portfolio theory help manage Enterprise risk?
Funding Available?
– Decision point
-Sources:
Interviews
-Probability: 80%
A Representation of the Enterprise of “Cradle to
Grave” Acquisition in the US Air Force
Wrap Up
Conduct study or analysis – Task
-Sources: Official docs, Interviews
-- Time Distribution: 45 to 180 days;
binomial, p=0.40
RSR
Funding available?
Mr. Harry Disbrow
HQ USAF A8
•Model tacitly accounts for portfolio “interdependencies” – a
problem identified in all interviews but deemed impossible to
quantify
•Model reflects “things as they really are”, not theory
•Model can be programmed and will lend itself to simulation
exercises and sensitivity analyses
•Work represents first Enterprise Systems Analysis for Military
Systems Acquisition using this methodology
Conduct Study or Analysis
Y
(individual org)
(individual org)
N
N
Characterizing the Enterprise of Acquisition: A Model for Analysis
Requirements Swimlane
Requirements Swimlane
Requirements Swimlane
Wa it fortec hni ca l res ult s
Wa ti fortec hn ci a l res ul ts
Put n
i to mod erni za it on pla nni ng
and sen d to Acq ui sit i on
M DAP
Th resh ol d
cros sed ?
to Acq ui sit i on M ode rniz ati on/Sus tain men t Ac tivi ty
(ind iv idu al org)
Yes
Pre pa re Conc ept o f O pera it on
Cap abi il ty Road map "wish l is t"fo r bud get su bmi ss ion
T hres hol d Ob jec tiv e T a
r de offs
CCD
No
Y
Deci si on to p ursu e req ui re men ts
Dra ft Bri efin g an d ma te ria sl
(i ndi vi dua l org )
"Coo rd " and "Co ncu r"
(in di vid ual orga ni zati ons )
MAJCOM "A" Lette rs
Up da te and sch edu e
l cal end ar
Y
Pre-RSR
(i ndi vid ua l org )
Y
Fn
i al zi e RSR & Cale nda r ti em s
RSR
(Ind vi d
i u al o rg)
Mr. Harry Dis brow
HQ USAF A8
MAJCOM A8
Co nti nue
Y
In sc ope of exi sti ng d ocu men ts?
Fo rm High Pe rforma nce T e am (HPT )
HQ USAF A8 org
(in di vid ua l org)
Draft
CCD
Ro ute to p rope r orga niz atio n
Extern al i npu t
Re qui reme nts Proc ess
De ci sio n to p ursu e requ ri e men ts
(in di vid ua l org)
Draft Brie fing and mate ir al s
"Coord "a nd "Con cur"
(ind vi d
i u al orga niz atio ns)
Y
MAJCOM A
" " Le tters
Up date a nd s ch edu e
l c ale nd ar
Ai r Sta ff Pro ces s
Pre-RSR
(in div d
i ual org)
Y
M AJ COM A8
Fn
i ali ze RSR & Ca e
l nda r i tems
RSR
(Indi vi dua l o rg )
Mr. Ha rry Disb row
HQ USAF A8
N
Inp uts from vari ous sou rces
O ther in pu ts
De cis io n to pu rsue re qu ir em ents
(i ndi vi dua l org )
Pro ce ss Start
KPP Dev elo pme nt
(Requ ri e men ts Po il cy Sh op)
F or e xis tin g prog ram?
(Req uir e men ts Poli cy sho p)
No
Dra ft Bri efin g an d mate rial s
"Coo rd" an d "Co nc ur"
(in div id ual orga niz ati ons )
MAJCOM "A" Le tters
Y
Up da e
t and sch edu e
l cal end ar
Pre-RSR
(i ndi vid ua l org )
MAJCOM A8
Y
RSR
Fn
i al zi e RSR & Cale nda r ti em s
(In di vid ua l org)
Mr. Harry Dis brow
HQ USAF A8
F und ing ava il abl e?
Y
Cond uct Stud y or Ana lys is
Dec si o
i n to pu rs ue re qui reme nts
(i nd ivi du al o rg)
(i ndi vi dua l o rg )
(ind vi d
i u al o rg)
Dra ft Brie fing and mate rial s
(Re qu ir em ents Po il cy Sho p)
MAJCOM "A" Le t e rs
Y
Up date and sch edu e
l c al end ar
Pre -RSR
(in di vid ual org)
MAJCOM A8
N
N
(Indi vi dua l org )
Y
RSR
(I n di vid ual org)
Mr. Harry Di sbro w
HQ USAF A8
Draft Doc ume nt (ICD)
HQ USAF A8 org
N
No
HQ AF A5RD
AFRO Cp repa ra tio ns
M AJ COM org
AF ROC de ci sio n?
Jo n
i t i ntere st?
AF ROC
O SD
N
No
N
No
N
Ai r Sta ff Pro ces s
Hi gh Perfo rman ce T ea m (MPT ) wo rk
Pre pare fo r M il esto ne Rev ie w
Info to Ac qui si ti on abo ut
CONOPS
cap ab il ty "wis h il sts"
Ye s
In terac tion with Acq uis ti i on
Up date brie f and materi al s
M ore Deta li ed res ul ts
fro m Pro totyp es/Eng in eeri ng M ode ls
Ac qui si ti on Re sul ts
Pe rson ne l from Acq ui sit i on o ffice
wo rki ng o n this parti cul ar ac tiv it y
Indi vi dua l org
Y
Crit i ca l Comm ents ?
AF RO Cp repa ratio ns
HQ AF A5 RD
MAJCOM o g
r
Y
Deci de to purs ue
Y
(In div d
i ual org)
De cid e to pu rsue
N
N
O SD
J RO Cd eci si on?
M AJCOM org
J RO C
N
Pre pare fo r Mil es to ne Re vie w
Ye s
CPD?
MAJCOM org
Jo int Proc ess
J8
N
J RO CPre parati on s
Crit i cal Comm ents?
J8
Y
De tail s of Prefe rre d
Sys tem Con cep t (P re m
il n
i a ry)
N
Ye s
Jo n
i t i nte re st?
N
Comme nt Res olu tion
MAJCOM a ppro val ?
N
J8
Y
Arch vi e
In di vid ua l org
AF ROC
No
Inform atio n to Acq uis ti i on Sys tem
for pre pato ry wo rk
Arc hi ve
Crit i cal Comme nts?
Info rmati on to Ac qui si ti on Sy stem
for p re pa o
t ry work
Y
AFROC de ci sio n?
Y
Ind vi i du al o rg
CDD?
Joi nt Proc ess
N
J8
De ci de to p ursu e
HQ AF A5RD
Draft Do cu ment ( CDD)
HQ USAF A8 o rg
N
MAJCOM o rg
MAJCOM a ppro val ?
Y
Dra ft Do cum ent (ICDo r draft CDD)
HQ USAF A8 org
(Ind vi i du al o rg)
J8
J8
Y
HQ AF A5RD
Co mme nt Reso u
l it on
Draft Docu men t (CDD)
HQ USAF A8 o g
r
(Ind vi i du al o rg)
Hig h Performa nce T eam (M PT ) work
Up date b rief a nd m ateri als
J oin t Pro ces s
Crit i cal Comm ents?
Comme nt Reso u
l tion
J ROC Pre para tion s
J ROCd ec si i on ?
MAJCOM o rg
JROC
Y
N
ICD?
MAJ CO M org
In forma it on to Ac qu si i tio n Syste m
for pre pato ry work
J ROC Pre para tion s
N
Crit i cal Comme nts?
No
Hig h Perform anc e T eam (MPT) work
Up date b rie f a nd m ateri als
N
No
Y
Y
MAJ CO M app rova l?
Y
Y
N
F orm Hig h Performa nce T eam (HPT )
HQ USAF A8 org
Y
N
No
N
N
HQ AF A5 RD
HQ AF A5 RD
No
Mo re Acq uis ti i on Res ult s
Prel m
i i nary Re sul ts from Ac qui si tion
OSD
Crit i cal Comm ents?
Air Staff Proce ss
Indi vi dua l org
Arch vi e
N
No, bec au se o f wo rk alr e ady don e
con du ct Ao A?
(i ndi vi dua l org )
Joi nt n
i teres t?
High Performa nce Te am (MPT ) work
Ob tain fund n
i g from
Ad va nce d Conc epts Bra nch
De cid e to p ursu e
F in ali ze RSR & Ca e
l nda r i tems
AFROC
MAJCOM o rg
N
F orm Hig h Performa nce Te am (HPT )
HQ USAF A8 org
Upda te bri ef an d mate rial s
Y
AF RO Cd eci si on?
AF ROC prep arati ons
No
No
Y
Fo m
r High Pe rfo rman ce T ea m (HPT)
HQ USAF A8 o rg
"Coo rd " and "Co ncu r"
(ind iv idu al orga niz atio ns)
Ro ute to Adv anc ed c on cep st
N
MAJ CO M org
JROC dec si o
in?
Y
No
N
Archi ve
N
J ROC
N
Budgeting and Programming Swimlane
Budgeting and Programming Swimlane
Budgeting and Programming Swimlane
Go to "Prep are
Co urse s of Acti on"
(Ac qui si tion Ste p)
Re pea t n + 2 time s
Re pea t n+ 2 tim es
AF e
l vel proc ess
MAJCOM PEM
Deci si on to
pu rsue
Adv anc ed Co nce pts Bud get Req ues t
Ne xt step ?
M AJ COM Pan el
M AJCOM Gro up
Next Step ?
MAJCOM Co unc li
Ne xt Ste p?
Y
Pro gram min g Iss ues &
Bu dge ting He ari ngs
Ne xt Step?
Y
PDM?
MAJCO M PEM
Dec si o
i n to
pu rsue
Adv an ced Co nce pts Bud ge t Req ue st
Yo r N
O MB & Con gres s
N
a
l w
Yes
c han ge s to
BES?
MAJ CO M PEM
De cis o
i n to
p ursue
Ad van ced Co nc epts Bu dge t Re que st
BES
Y
Nex t step ?
MAJCOM Pan el
Yo r N
OSD
Go to "Prep are
Cours es of Acti on "
(Ac qu si i tion Ste p)
Go to "Prep are
Co urse s of Acti on"
( c qui si ti on Ste p)
A
Re pea t n+ 2 tim es
Yes
Yes - n
i fo rmatio n on yl
PBD
Y or N
DoD PO M?
MAJ CO M Grou p
Nex t Ste p?
MAJCOM Cou nci l
Next Step ?
AF e
l ve l proc es s
Nex t Ste p?
O SD Proc ess
Y
Nex t s tep?
M AJCOM Pan el
Ye s
Yes - i nforma tio n on yl
Y
PDM?
MAJCO M Grou p
Next Step ?
M AJ COM Cou nci l
Ne xt Step?
AF l ev el p roce ss
Ne xt Step?
O SD Proc ess
Cong ress
N
l aw
Yes - n
i forma tion onl y
Y or N
cha ng es to
BES?
Yes
Co ngre ss
N
l aw
Yes
cha nge s to
BES?
BES
BES
Do DPO M?
PBD
Y or N
Y o rN
Yes
Y
PDM?
PBD
Y or N
DoD POM?
Y or N
OMB
Y o rN
OM B
Ye s
Re tr y ?
Retry?
Retry?
No
Archi ve
Arc hiv e
Arc hiv e
Rele ase mon eys
to Acqu si i tio n
T o Bud get i npu ts
Re e
l a se m one ys
to Ac qui si ti on
Rele as e mon eys
to Acq ui sit i on
To Budg et n
i puts
T o Bu dge t n
i p uts
T o Acqu si ti o
i n
F all ou t o f Bu dge t Pro ces s
im pac ting pla nni ng
T o Ac qui si tion
Fa o
l ut of Bud get Proc ess
i mpa cti ng p lan ni ng
To Ac qui si tion
Fa l out of Bud ge t Proc ess
m
i p acti ng p a
l n ni ng
F u l F u ndi ng i n F YDP
F ul l F und n
i gn
i FYDP
KPPs from
Req uir e men ts
Acquisition
Prepa ratio n for Acq uis it i on Pan els
Acquisition
Ac qui si ti on Pa nel s
Ye s
M DA App rova ?
l
Mil es tone A
Ac qui si tion Pa ne l (DAB, MDA, ADM , e tc)
Afforda bil ti y As ses sme nt
Prep arati on fo r Ac qui si tion Pa ne sl
Acqu si ti o
i n Progra m Base il ne
Acquisition
Acqu si ti o
i n Pane sl
Pre para tion for Acqu isi tio n Pane ls
Ac qui si ti on Pa nel s
Yes
MDA Ap prov al?
KPPs fro m
Requ ri e men ts
M il eston e C
Progra m Offic e Cost Esti mate
Co ntrac to r Co st Esti mate
Pre pare for Ac qu si i tio n Pane sl
No
Afford abi il ty Asse ssm ent
Inde pen den t Co st Estim ate
No
Ac qui sit i on Pro gram Ba sel n
ie
Yes
MDA Appro val ?
N
Sou rce Sel ecti on Pl ans
RFP rel ea se a nd Sou rce Sel ec ti on
Draft RF P Pre para ti on
Mil esto ne B
F rom Req ui re men ts
Ap prov al?
Con tra ct Ma nag eme nt
F rom Re qui reme nts s ystem :
Dra ft d ocu men t fo r bas si o f wo rk
No
Draft RF P Prep arati on
Y
F un ds
Red ri ec ted
RF P re lea se a nd So urce Se lec tion
No
RF P re e
l a se a nd Sou rce Sel ec tion
T o MAJCOM PEM
Bu dge t n
i p ut
Pro test?
Draft RF P Prep arati on
Sco pe a nd Award T e chn olo gy De vel opm ent c ontrac ts
In terna l y or ex terna l y gen erate d
mul tipl e tim es p er ye ar
Pe ir od ci Re vie ws
Scop e an d Award Sys tem Dev elo pme nt an d Demo nstra ti on con tra cts (o r Ao A)
RFP c oord ina tion proc ess
Yes
Intern all y or e xterna l y ge nera ted
mul tip e
l ti mes p er ye ar
Pre pare Pro gram b udg et de cis io n (PBD)
Up hel d?
Uphe d
l?
T o M AJ COM PEM
Budg et in put
Yes
Yes
T o MAJ CO M PEM
Bu dg et n
i put
Yes
Rep eat "n" y ears
No
RF P co ordi nati on p roce ss
Repe at "n" ye ars
Yes
Yes
Acq ui sit i on Pl ann n
i g Activ ti e
i s
No
Protes ?
t
Co ntrac t M ana ge ment
Re pea t "n" yea rs
Are Ob s &
Exp O K?
No
F rom Pl ann n
i g & Bu dge ting
Contrac t Man age men t
RF P c oo rd n
i atio n pro ces s
Inte rnal yl o r exte n
r al yl g ene rated
mu tl i ple time s pe r y ear
Scop e an d Award Study con tracts (o r Ao A)
Del ay
No
Are Ob s &
Exp OK?
No
De a
ly
Yes
F u nds
Redi recte d
No
Pe rio dic Re vi ews
Desi gn Re din ess
Re vi ew?/MDA
Appro val
Ye s
YES
No
sc ope growth or
tech ni cal prob lem s
Seek fu nds ?
PEM/othe r staff
fi nd mo ne ys
Yes
No
O btai n
F und s?
(time yl )
No
Are O bs &
Ex p OK?
No
No
Sou rc e Sel ecti on Pl ans
Ye s
Fu nd s
Re di e
r cted
No
Perio di c Revi ews
Ye s
Yes
T o Req ui e
r men ts
sc op e growth o r
tec hni ca l pro ble ms
Prepa re Cours es
of Acti on
Preferre d
Sy stem
Co nce pt
F un din g
Sho rtf a l
Contrac t Ch an ge
Requ ri ed ?
s cop e gro wth or
te chn ci a l p ro bl ems
PEM/othe r s taff
find mon eys
Ye s
See k fun ds?
No
Obta n
i
F u nds ?
(ti mel y)
No
Ye s
Se ek fun ds ?
Yes
PEM /o ther s a
t ff
fi nd m one ys
No
Obtai n
F un ds?
(tim ely )
No
Yes
Pre pare Co urse s
of Actio n
So urce Se e
l ctio n Pla ns
Yes
F u ndi ng
Sh ortfall
Prep are Cou rses
o f Ac tion
Sys tem Perfo m
r anc e
Spe cif i ca ti on
T o Req ui re men ts
Co ntract Cha ng e
Re qui red?
In ti i al Rate prod ucti on
Base li ne
Ye s
Acq uis ti i on Pan el s
Fu ndi ng
Sh ortfal l
Contrac t Ch ang e
Req ui red?
End l oop
“This ain’t rocket
science. It’s much,
much harder than that.”
BGen in Acquisition
Yes
Adj ust Prog ram an d Con tr a ct
(co st/sc hed ule esti mate s rev si e d)...
End l oo p
Prep arati on for Acq ui sit i on Pa nel s
End o
lop
T o Req ui e
r men ts
Yes
Chan ge Co ntract/Res cop e
effort (co st/sc hed ul e m
i pac t)
Be st Ma te ria l
Ap proa che s
Ye s
Ch ang e Con tr a ct/Resc ope
e f o rt (c ost/s che dul e i mpac t)
Dif fere nt Distri buti on p rofil e
an d le ngth dep end ing upo n
pa th take n
SRR?
Adju st Progra m an d Contrac t
(cos t/sch edu e
l e sti mates revi sed )...
Pre li min ary
In tegra ted
Arch etic ture
Cha ng e Contrac t/Re sc ope
e ffort (cos t/s ch edu e
l i mpa ct)
No
Differen t Di strib utio n pro fie
l
and len gth d epe ndi ng u po n
path ta ken
No
T o Requ ir e ments
Eng n
i eeri ng
Re vi ews?
Sys tems En gi nee ri ng
T o Eve nt Happ ens step
T o Sys e
t ms En gin eeri ng s tep
Rework a nd Del ay
Di f e rent Dis tribu tion profi e
l
a nd l eng th de pen di ng u pon
p ath tak en
Ac qui si ti on Pl an nin g Activ ti i es
Yes
Syste m
Performa nce
Spec fi ci ati on
Adj us t Pro gram a nd Con tract
(c ost/sc he dul e es tima tes rev is ed)...
No
T o Ev ent Hap pen s ste p
Sy stem s Engi ne erin g
Data from
User Stud e
is
Prefe rre d
Sys tem
Conc ept
T est Rea dei ne ss
Re vie w
Yes
Yes
Ma ke T rad es?
No
No
Sys tem
Veri fi ca tion
Revi ew?
T o Req ui re men ts
Acqu si ti o
i n Pla nni ng Ac ti vi ties
Syste ms Eng n
i e eri ng
Pro totyp es/
Eng n
i eeri ng
Mo del s
Sys tems En gin ee ri ng
T o Ev ent Hap pen s ste p
T o Req uir e men ts
T o Eve nt Ha pp ens
T o Eve nt Happ ens
T o Sys tems En gin ee ri ng
T o Syste ms Eng n
i eeri ng Step
T o Ev ent Hap pen s
T o Sy stem s Engi ne erin g
T o Eve nt Happ ens
T o Sys tems En gin eeri ng
Prototy pes /
Eng n
i e erin g
Mod el s
Contractors
Contractors
Perform work
Co ntrac t Sta tr -u p
Eve nt Happ ens
Co ntract Start-up
Sc ope Growth/
T e chn ci a l Prob e
l m s?
Rework a nd De a
ly
Perfo m
r wo rk
T o Even t Ha ppe ns s tep
Contractors
Re wo rk an d Del ay
Re wo rk an d Del ay
Re work a nd Del ay
Perform work
Con tr a ct Start-up
Ev ent Hap pe ns
Sc ope Growth/
T ech ni cal Pro bl ems ?
YES
YES
YES an d NO
NO
F ab rica tion
No
T ask
Com ple te?
Asse mbl y
Inte grate d tes ti ng
Deve o
l pme ntal testi ng (Sys tem), Li ve F i re T est, Op erati ona l Ass ess men t
Comb n
i e d T es ti ng (DT &E, OT &E, LF T &E)
T o "P re pare Co urse s of Acti on"
i n Acq ui sit i on Swim a
lne
No
T as k
Co mpl ete?
T o "P re pare Co urse s of Acti on"
i n Acq ui sit i on Swim lan e
Ev ent Hap pen s
Sc ope G rowth/
T ech nic al Pro ble ms?
Yes
To Pre ferred Sy stem Con ce pt
YES
Yes
Yes of NO
T o SRR
No
Ta sk
Co mpl ete?
T o "Prep are Cou rses of Actio n"
i n Acq uis ti i on Swim a
lne
Yes
T o Prototy pe Eng n
i eeri ng Mo de sl
F ol low pa th at ap prop ria te stag e
Swim Lanes are used to show key processes. A “rectangle” is a task with a time distribution associated with it (represented by a binomial distribution and a designated p-value)
A “diamond” is a decision point with a branching probability to “yes” or “no”. A “oval” is information and serves to connect processes between swim lanes. A “parallelogram” shows the product of a process
Robb Wirthlin