Menu
Searc
Read the Tea Leaves
Software and other dark arts, by Nolan Lawson
The collapse of complex
software
In 1988, the anthropologist Joseph Tainter
published a book called The Collapse of Complex
Societies. In it, he described the rise and fall of
great civilizations such as the Romans, the
Mayans, and the Chacoans. His goal was to
answer a question that had vexed thinkers over
the centuries: why did such mighty societies
collapse?
In his analysis, Tainter found the primary enemy
of these societies to be complexity. As
civilizations grow, they add more and more
complexity: more hierarchies, more
bureaucracies, deeper intertwinings of social
structures. Early on, this makes sense: each new
level of complexity brings rewards, in terms of
increased economic output, tax revenue, etc. But
at a certain point, the law of diminishing returns
sets in, and each new level of complexity brings
fewer and fewer net benefits, dwindling down to
zero and beyond.
But since complexity has worked so well for so
long, societies are unable to adapt. Even when
each new layer of complexity starts to bring zero
or even negative returns on investment, people
continue trying to do what worked in the past. At
some point, the morass they’ve built becomes so
dysfunctional and unwieldy that the only solution
is collapse: i.e., a rapid decrease in complexity,
usually by abolishing the old system and starting
from scratch.
What I find fascinating about this (besides the
obvious implications for modern civilization) is
that Tainter could have been writing about
software.
Anyone who’s worked in the tech industry for
long enough, especially at larger organizations,
has seen it before. A legacy system exists: it’s
big, it’s complex, and no one fully understands
how it works. Architects are brought in to “fix”
the system. They might wheel out a big
whiteboard showing a lot of boxes and arrows
pointing at other boxes, and inevitably, their
solution is… to add more boxes and arrows.
Nobody can subtract from the system; everyone
just adds.
Endpoint
AGEREEATRA
CLOGGER
ZURTI
GARY
BINGO
NPP|RATMAN
/onl
RACCOON
EKS
TRY
Paper°
Balon
e
MAsP
MI
(ABS)
OMEGA
STAR
HELL
BREAD
REPLICA
“EKS is being deprecated at the end of the month
for Omega Star, but Omega Star still doesn’t
support ISO timestamps.” We’ve all been there.
(Via Krazam)
This might go on for several years. At some point,
though, an organizational shakeup probably
occurs – a merger, a reorg, the polite release of
some senior executive to go focus on their
painting hobby for a while. A new band of
architects is brought in, and their solution to the
“big diagram of boxes and arrows” problem is
much simpler: draw a big red X through the
whole thing. The old system is sunset or
deprecated, the haggard veterans who worked
on it either leave or are reshuffled to other
projects, and a fresh-faced team is brought in to,
blessedly, design a new system from scratch.
As disappointing as it may be for those of us who
might aspire to write the kind of software that is
timeless and enduring, you have to admit that
this system works. For all its wastefulness,
inefficiency, and pure mendacity (“The old code
works fine!” “No wait, the old code is terrible!”),
this is the model that has sustained a lot of
software companies over the past few decades.
Will this cycle go on forever, though? I’m not so
sure. Right now, the software industry has been
in a nearly two-decade economic boom (with
some fits and starts), but the one sure thing in
economics is that booms eventually turn to
busts. During the boom, software companies can
keep hiring new headcount to manage their
existing software (i.e. more engineers to
understand more boxes and arrows), but if their
labor force is forced to contract, then that same
system may become unmaintainable. A rapid and
permanent reduction in complexity may be the
only long-term solution.
One thing working in complexity’s favor, though,
is that engineers like complexity. Admit it: as
much as we complain about other people’s
complexity, we love our own. We love sitting
around and dreaming up new architectural
diagrams that can comfortably sit inside our own
heads – it’s only when these diagrams leave our
heads, take shape in the real world, and outgrow
the size of any one person’s head that the
problems begin.
It takes a lot of discipline to resist complexity, to
say “no” to new boxes and arrows. To say, “No,
we won’t solve that problem, because that will
just introduce 10 new problems that we haven’t
imagined yet.” Or to say, “Let’s go with a much
simpler design, even if it seems amateurish,
because at least we can understand it.” Or to just
say, “Let’s do less instead of more.”
Simplicity of design sounds great in theory, but it
might not win you many plaudits from your peers.
A complex design means more teams to manage
more parts of the system, more for the engineers
to do, more meetings and planning sessions,
maybe some more patents to file. A simple
design might make it seem like you’re not really
doing your job. “That’s it? We’re done? We can
clock out?” And when promotion season comes
around, it might be easier to make a case for
yourself with a dazzling new design than a
boring, well-understood solution.
Ultimately, I think whether software follows the
boom-and-bust model, or a more sustainable
model, will depend on the economic pressures of
the organization that is producing the software. A
software company that values growth at all cost,
like the Romans eagerly gobbling up more and
more of Gaul, will likely fall into the “addcomplexity-and-collapse” cycle. A software
company with more modest aims, that has a
stable customer base and doesn’t change much
over time (does such a thing exist?) will be more
like the humble tribe that follows the yearly
migration of the antelope and focuses on
sustainable, tried-and-true techniques. (Whether
such companies will end up like the hapless
Gauls, overrun by Caesar and his armies, is
another question.)
Personally, I try to maintain a good sense of
humor about this situation, and to avoid giving in
to cynicism or despair. Software is fun to write,
but it’s also very impermanent in the current
industry. If the code you wrote 10 years ago is
still in use, then you have a lot to crow about. If
not, then hey, at least you’re in good company
with the rest of us, who probably make up the
majority of software developers. Just keep doing
the best you can, and try to have a healthy
degree of skepticism when some wild-eyed
architect wheels out a big diagram with a lot of
boxes and arrows.
Related
The balance has shifted away from SPAs
May 21, 2022
In "Web"
More thoughts on SPAs
May 25, 2022
In "Web"
Introducing the Cordova SQLite Plugin 2
April 10, 2016
In "Webapps"
 10 Replies
June 9, 2022
« Previous
Leave a Reply
Your email address will not be published.
Required fields are marked *
Comment *
Name *
Email *
Website
Post Comment
Notify me of new comments via email.
Notify me of new posts via email.
This site uses Akismet to reduce spam. Learn
how your comment data is processed.
Jim Boyland on June 9, 2022 at 1:43 PM
This article hits the nail on the head,
well done.
 Reply
Pingback: Futureseek Daily Link Review; 10
June 2022 | Futureseek Link Digest
cataraducan on June 9, 2022 at 4:54 PM
I do not agree with this premodern
boom-and-bust cyclical take,
something else much more interesting
and profound is going on (think of
imanent antagonisms between Kants’s
contingent pathological objects and
universalities, the OOP vs FP debate is
problably fits well here). For a guy who
worked in software industry and
perceived it as a mess, the real
question to ask is “how that anything
works at all?”
 Reply
Rich Astone on June 9, 2022 at 5:02 PM
Some great food for thought and an
example of the value of applying
knowledge of the humanities (History)
to the world of business and
technology. I’m sharing on LinkedIn if
that’s ok. I’m very appreciative of the
insights from folks like you in the HN
community.
 Reply
James Parker on June 9, 2022 at 6:16 PM
Not all engineers embrace complexity,
even in their own work. I often quote
Turing award winner CAR Hoare, who
said a system has either “no obvious
defects” (due to over complexity) or ”
obviously no defects”. It is typically
those outside of engineering
(marketing or management) that push
for complexity, even if they don’t intend
to.
 Reply
Nolan Lawson on June 10, 2022 at
12:17 PM
That’s fair. OTOH I do think it’s
easy to be seduced by your own
design, at least while it fits in your
own head, and to forget about the
potential unintended
consequences.
You’re right that business
requirements often drive
complexity. I think this is
something I was trying to get at:
sometimes software complexity is
a symptom, not a cause, e.g.
because the organization that
produced it is complex (think:
Conway’s Law).
 Reply
tetsuoii on June 10, 2022 at 10:09 AM
Yes, this is all true and for this exact
reason I only write software in C and
avoid modern paradigms. It’s better to
write small programs with low
complexity and high performance. That
requires skill and craftmanship.
Programs should be minimalist and
have as few dependencies as possible.
It’s often better to cannibalize a
program than to extend it. The socalled modularity and extensibility of
object orientation often causes the
exact opposite result. Data is hidden
behind abstractions, unnecessary
layers of complexity, copied and
handled adding performance loss to
obscurity. Then this is iterated upon
across components creating a tangled
mess.
 Reply
Pingback: Links 10/06/2022: ExTiX Deepin 22.6
and Diffoscope 216 | Techrights
Louis on June 10, 2022 at 6:45 PM
Lookin at you, Dependency Injection!
 Reply
Pingback: Links 11/06/2022: Qt 6 in Debian |
Techrights
Recent Posts
The collapse of complex software
State is hard: why SPAs will persist
More thoughts on SPAs
The balance has shifted away from SPAs
The struggle of using native emoji on the web
About Me
Hi, I'm Nolan. I'm a web developer
living in Seattle and working for
Salesforce. Opinions expressed in
this blog are mine and frequently
wrong.
Archives
June 2022 (1)
May 2022 (3)
April 2022 (1)
February 2022 (1)
January 2022 (1)
December 2021 (3)
September 2021 (1)
August 2021 (6)
February 2021 (2)
January 2021 (2)
December 2020 (1)
July 2020 (1)
June 2020 (1)
May 2020 (2)
February 2020 (1)
December 2019 (1)
November 2019 (1)
September 2019 (1)
August 2019 (2)
June 2019 (4)
May 2019 (3)
February 2019 (2)
January 2019 (1)
November 2018 (1)
September 2018 (5)
August 2018 (1)
May 2018 (1)
April 2018 (1)
March 2018 (1)
January 2018 (1)
December 2017 (1)
November 2017 (2)
October 2017 (1)
August 2017 (1)
May 2017 (1)
March 2017 (1)
January 2017 (1)
October 2016 (1)
August 2016 (1)
June 2016 (1)
April 2016 (1)
February 2016 (2)
December 2015 (1)
October 2015 (1)
September 2015 (1)
July 2015 (1)
June 2015 (2)
October 2014 (1)
September 2014 (1)
April 2014 (1)
March 2014 (1)
December 2013 (2)
November 2013 (3)
August 2013 (1)
May 2013 (3)
January 2013 (1)
December 2012 (1)
November 2012 (1)
October 2012 (1)
September 2012 (3)
June 2012 (2)
March 2012 (3)
February 2012 (1)
January 2012 (1)
November 2011 (1)
August 2011 (1)
July 2011 (1)
June 2011 (3)
May 2011 (2)
April 2011 (4)
March 2011 (1)
Tags
accessibility alogcat
android android market
apple app tracker blobs boost bootstrap browsers bug reports
catlog chord reader code contacts continuous integration
copyright couch apps couchdb couchdroid developers
development grails html5 indexeddb information
retrieval japanese name converter javascript jenkins
keepscore listview localstorage logcat logviewer lucene
mobile web modules nginx
nlp node nodejs npm offline-first
open source passwords performance
pokedroid pouchdb pouchdroid query expansion
relatedness calculator relatedness coefficient s3
sectioned listview
security semver social
safari satire
media socket.io
software development solr spas
supersaiyanscrollview synonyms
crossword
twitter ui design ultimate
w3c webapp webapps web platform web sockets
websql web workers
Links
Mastodon
GitHub
npm
Keybase
View Full Site
Blog at WordPress.com.