Large Requirements-Driven vs. Small Informal IT Projects
Do you ever wonder how many projects succeed using the traditional requirements-driven model?
Small teams often dive in, start collecting data, and write code to process their data with only vague requirements such as a basic description of
the problem (and usually get all kinds of successful results).
Traditional software companies spend far too much time determining the requirements and rarely produce anything (even if they are merely
reconfiguring or tailoring an existing product to the new requirements).
Yes, this line of thinking is right on target (i.e., the best code is often formed by small informal teams of subject matter experts).
The fields of systems and software engineering have DEVOLVED into the requirements-driven model in the last five decades (i.e., MIL-STD1521B, DoD-STD-2167A, MIL-STD-498, J-STD-016, ISO 12207, ISO 15288, SW-CMM, CMMI, SEBoK, SWEBoK, PSP/TSP, etc.).
(Agile experts call this Linear thinking, Waterfall life cycle, Big Design Up Front, etc. ...)
 My first project was a toolset for the human space program in the 1980s (i.e., $50M requirements phase, $250M development phase, and
nothing to show for it after a decade).
 My second project was the human space program's avionics software itself (i.e., a few crackerjack programmers successfully created small
software subsystems, while armies of systems/software engineers did not do the same using requirements-driven models for $2B per year).
 My third project was a corporate software project management tool like MS-Project or Primavera (that was successfully created by a single
programmer with one or two years of experience).
 My fourth project was a $50M radar command and control modernization project (that was saved by a single MSEE with crackerjack C
programming skills who interfaced to radar libraries single-handedly created and commercialized by a physicist with a PhD).
 My fifth project was a $25 million proof-of-concept to simulate telemetry-over-IP for a large satellite constellation (in which a college intern
with poor C programming skills unsuccessfully tried to compensate for defective hardware resulting in the loss of a $30B contract).
(The initial software systems for air-to-air avionics weapon systems were successfully developed using small teams of physicists and MSEEs
who were crackerjack C and Assembly language programmers ...)
There are many case studies of individuals and small teams who have successfully created great software products in weeks and months (i.e.,
online trading systems handling billions of dollars in transactions, Internet browsers, battleship command and control systems, etc.).
(Commercial software packages are often created, maintained, enhanced, and evolved by a single, highly-talented programmer ...)
On the contrary, many terrible software-intensive projects involved small armies of programmers (i.e., FBI fingerprint system, FBI casefile
system, FAA air traffic control, USAF fighter jets avionics systems, etc.).
There's a name for these terrible projects (i.e., runaways, death marches, etc.) and there are numerous articles on this phenomenon called
"escalation theory" (i.e., psychological need to continue investing millions and billions of dollars in bad projects, which make them worse).
 A government agency spent about $50M on the perfect software-GPS prototype in the 1990s for all of its flying aircraft, only to have the
contractor deliver some FORTRAN code from the 1970s (because the contract had a "material substitution clause").
 Most large government IT projects cost around $1B, are staffed by hundreds of people, and end in failure (while the smallest projects staffed
by a few people and created for a few hundred thousand or million dollars almost always end in success).
 Many systems/software engineers are part of $500 to $600M runaway government projects that could have been completed by a handful of
crackerjack Java programmers (i.e., PMs invest millions of dollars in these projects to gain notoriety, fame, and political standing).
Most big software-intensive projects end in failure (i.e., cost and schedule overruns, poor technical performance, or extremely poor software
quality and reliability).
In some cases, small teams save large projects at a fraction of the cost/time utilized by system/software engineers over a period of years and
billions of dollars (i.e., modern fighter jet programs, software-intensive communication systems, law enforcement systems, etc.).
Small teams using little or no systems/software engineering discipline are NOT always successful (i.e., a buggy software-driven therapeutic Xray machine created by a single physicist contributed to the death of dozens of people over many years until its flaws were uncovered).
We have an official PARADOX here (i.e., have large software teams use too much systems/software engineering discipline and almost
certainly fail vs. small software teams with little or no discipline which is also a volatile, high-risk proposition).
Thus, enter "Agile methods," which are small, lightweight, but highly-disciplined software methods created by programmers for programmers
(with built-in continuous improvement cycles to accelerate the learning curve and rapidly improve performance by an order-of-magnitude).
Ironically, Agile methods are often considered "hacking" by systems/software engineers who want to kill projects with too much discipline
(and Agile methods are too bureaucratic for small teams who want design and code software without any lightweight discipline at all).
The reason small projects fail is due to lack of lightweight discipline, while the reason large projects fail is due to too much heavyweight
discipline, the latter of which is explained by queuing theory (i.e., too many processes, products, and people cause decisions to deadlock).
Public sector projects have too many resources available to them and often fail for spending too much money just because they can, often in the
name of discipline, quality, reliability, and mission assurance (whereas Agile methods can exceed these expectations at a fraction of the cost).
The bottom line is that heavyweight traditional methods are not best solution for large public sector projects, while Agile methods are a
legitimate alternative that simultaneously exhibit the characteristics of flexibility, efficiency, discipline, and most importantly, success.