Toyota’s Woes Linked to Software Bugs and Raises Questions
about Auto Electronics
It is now official that the problems with the 2010 Toyota Hybrid Prius
cars are directly related to defects in the braking control software. The
accelerator fiasco was a serious one indeed, causing 52 deaths since
2000.So how can Toyota and other automobile companies prevent
such devastating malfunctions from happening again? By making sure
the software that controls all the electronic devices in today's cars do
not have any problems in the first place. Dave Peterson from Coverity
discusses the issue.
There’s been a lot of coverage about the Toyota recalls and the role of
software in automobiles. What hasn’t been clearly defined, however, is the
sheer complexity of the software code involved and the challenges involved in
checking for problems.
Today’s auto software is not a standalone package. Most drivers think that
turning the steering wheel right and applying pressure on the brakes are
physical acts – but often they are literally computer commands telling your car
to slow down and turn. If drivers knew this, it could impact the way they react
to problems. If the accelerator 'sticks' they should throw the car in neutral because pulling on the pedal will not do anything.
Indeed, modern automobiles have as much as 100 million lines of code in
embedded software running inside the car – that’s more than an F-35 or a
Dreamliner 737 jumbo plane. From connecting and controlling mechanics,
ABS engagement systems to the traditional electronics such as navigation,
audio, heating and cooling, software affects various parts of the automobile.
The modern cars have brake by wire, adaptive cruise control, active steering,
tire sensors and many other parts all controlled by embedded systems and
software.
To complicate things further, most of the software code often comes from
multiple suppliers as well. In the Toyota case, it’s not clear whether the faulty
brake software in the Prius was coded by Toyota engineers or by a supplier
down the chain. And even if each individual code base was 100% bug free,
there’s no guarantee that when the different pieces are glued together into a
vehicle, there won’t be ‘glitches.’ To complicate things further, time to market
pressure and the rush to add more electronic features in modern automobiles
is creating a software integrity problem. Are we actually spending enough time
making sure the software is as perfect as the look of a new car?
1
This lack of software testing is probably the real challenge for auto
manufacturers like Toyota. How did things end up this way? What started as
a manufacturing assembly line process allowing many vehicles to be built in a
predictable way eventually evolved into a Deming quality control process
driving quality procedures for predictable, repeatable, scalable manufacturing.
The Deming approach was created by Edward Deming, the business
management guru who advised Japan on how to rebuild its shattered
industries after the Second World War. Simply put, Deming argued that
enlisting the efforts of willing workers to do things properly the first time and
giving them the right tools were the real secrets of improving quality -- not
teams of inspectors.
Sadly the automotive assembly has reached a point where well-built vehicles
are failing miserably because there is no Deming-like rigor on quality in the
new software assembly line. Vehicles at any speed — just like computer
systems at wire speeds — are unsafe without an engine of high integrity
software driving it.
So, it should follow that in the new software assembly line we too have an
Office of Defect Investigation – only, build it early into the development cycle,
and not after products have gone into the field. Once software has rolled into
the field, it’s too late – and the results can be damaging (as Toyota knows
only too well). In particular, this office may want to focus on three
“departments”:
Architecture Analysis as we the design software.
Static Analysis as we write the code.
Dynamic Analysis and as we detect and fix functional issues.
A while back, there was an urban legend making the round in news cycles.
According to the most popular version, Bill Gates allegedly stated, “If GM had
kept up with the technology like the computer industry has, we would all be
driving $25.00 cars that got 1,000 miles to the gallon.” In this story, GM
responded with a smart press release asserting that if it developed products
the way Microsoft did, its cars would have all the problems of Microsoft’s
software. This has been proven to be an untrue story but in context of the
Toyota recall, the debate still stands on its own.
Software professionals cannot be on the receiving end of such debates. They
need their own Offices of Defect Investigation and emphasis on production
quality early and often to ensure software integrity in every piece of software
delivered to the industry.
2