6th IFAC Symposium Advances in Automotive Control
Munich, Germany, July 12-14, 2010
Automated Validation Process
for On-Board-Diagnosis Functions
Dr. Felix Richert*, Daniel Brückner**
* BMW Group, Hufelandstraße 4, 80788 Munich
Germany (Tel: +49-89-382-18794; e-mail: felix.richert@ bmw.de).
** BMW Group, Hufelandstraße 4, 80788 Munich
Germany (Tel: +49-89-382-37592; e-mail: daniel.brueckner@ bmw.de).
Abstract: Half of all ECU software functions are meanwhile monitoring functions. For a better and faster
development and calibration of these diagnostic functions, an automated validation process has been developed. Due to specific OBD regulations, failure simulations without calibration changes in the ECU
software are needed. The approach to design these failure simulations to be used on series-production vehicle and the according tool chain for a fully automated validation of all OBD functions are presented.
Keywords: On-Board-Diagnosis, Validation, Hardware-in-the-Loop-Simulation, Failure simulation,
Product Vehicle Evaluation.
not only electrical monitors with simple shortcut or open
circuit failures, but also high level rationality monitoring
functions, these failure simulation methods are an essential
part of developing validation methods for monitoring functions and include not only the needed failure simulation models but as well additional sensor specific hardware.
1. INTRODUCTION
During the last years, BMW has developed a rising number
of engines and vehicle models. Cutting edge technology for
improvements in fuel consumption and power output at the
same time, more and more demanding emission regulations
and faster development processes require new and more efficient validation methods. Calibration of the monitoring functions has to take place at the very end of the development
process and is influenced by late changes of in emission control and drivability; hence the need for improvement in validation quality and less needed time for the validation process
at the same is significantly high.
For automated software testing there are several known approaches and commercial software tools available. Being
aware of this, the testing automation on its own is meanwhile
standardized technique in software development processes
and used for many use cases in automotive industry and
elsewhere. Nevertheless, the combination of all particular
elements of the tool chain and failure simulation methods
described in this paper, this approach for the validation of
high level automotive monitoring functions does show a
strong improvement to nowadays OBD validation, building a
testing framework capable of covering all needs around testing of all monitoring functions which is fully usable with a
HIL simulator as test bench as well as a real vehicle.
For emission relevant systems there are particular legislation
called “OBD regulation”. In contrary to the entire amount of
diagnostic monitors running online during vehicle operation
“OBD” in the automotive language use usually means only
these emission relevant monitoring functions. Since more and
more efforts have to be taken to fulfil the current emission
regulations, a significant part of the software function in the
engine control unit (ECU) are related to OBD. Aside the
specification which components and systems have to be
monitored, the obligation for specific validation tests for
every OBD function is included.
After an introduction to the relevant OBD regulations for the
validation process the used methods for failure simulation are
presented as well as the entire tool chain for the automated
validation process. All necessary steps are explained in detail
using the catalyst monitoring as an example.
For both needs – fast diagnosis validation and mandatory
OBD specific validation tests, an automated validation process is needed. Since every diagnostic function is designed to
recognize a particular system fault, for every fault a suitable
failure simulation method is needed. Due to specific OBD
legislations the ECU has to remain unchanged (i.e. no “testing calibration” is allowed), and real hardware changes cannot be done automatically, hence these failure simulations
have to be built on signal manipulation only. Since there are
978-3-902661-72-2/10/$20.00 © 2010 IFAC
2. ON-BOARD-DIAGNOSIS
On-Board-Diagnosis is on the one hand the sum of all monitoring functions of an engine control unit running during
vehicle operation (contrary to off-board-diagnosis which is
only performed e.g. at the garage). On the other hand in
automotive language use, “OBD” usually means this particular subset of monitoring functions which are related to emission control and emission control regulation. In this article,
727
10.3182/20100712-3-DE-2013.00102
AAC 2010
Munich, Germany, July 12-14, 2010
selected by CARB for which validation results have to be
presented to CARB (three with “demo” and “PVE”, three
with “PVE” only, see below).
also only those diagnostic functions for emission relevant
components are included, if On-Board-Diagnosis or OBD is
mentioned. Contrary to this, “general diagnosis” may be used
for the entity of all online monitoring functions.
This validation is divided into two main parts: During the so
called “demo”, all faults, which are calibrated regarding to
emission thresholds, have to be applied during the standard
driving cycle FTP72 and monitor completion as well as fault
detection before exceeding certain thresholds have to be
proven. These tests have to be done by the manufacturer with
a pre-series vehicle on the emission test bench and are (for
the selected vehicle test groups) part of the mandatory certification process before start of production.
2.1 OBD Legislation
There are several different OBD regulations worldwide. The
most important one (and as well the first one in 1986) is the
one of the US state California, provided by “California Air
Resource Board” CARB. The so called “final regulation
order” [CARB] contains quite detailed instructions, which
engine components are to be monitored and which kind of
faults have to be covered. This includes in the first step electrical faults and out of range monitors which both have to
operate continuously and are usually covered by quite simple
functions with only several thresholds to be calibrated.
The “Product Vehicle Evaluation” (PVE) takes place with
one of the final series production vehicles, must also be done
by the manufacturer, and has to be presented the latest six
month after start of production. Here, the entire communication with the Scan Tool and all OBD faults have to be tested,
but not with a certain driving cycle or related to thresholds,
but only as a functional check whether a fault is recognized
under suitable conditions or not. The official PVE is divided
into three sub-test-areas: The communication, the monitor
functionality and additional standardized field data about the
average of the functional completion during in use driving
conditions. In this article, the main focus lies on the functional validation, hence in the following “PVE” is used (not entirely correctly) as a synonym for only this fraction of all its
parts (”PVE(j)(2)”).
To understand the complexity of these tests, some details
about the PVE and demo have to be known: The manufacturer has to provide a description for each fault, how it can be
stimulated artificially. Several approaches are allowed, e.g.
the usage of a defect component like a stuck valve or a defect
catalyst, removing a certain signal cable or tuning a characteristic line of a certain sensor with an external device – as
long as no change in calibration of the ECU is made. Generally speaking, any method is allowed which does not destroy
the vehicle irreversibly and which is usable on a conventional
vehicle out of the serial production. Faults which cannot be
provoked without destruction of parts of the vehicle can be
approved by CARB as not mandatory for PVE testing, e.g.
ECU internal faults.
Furthermore, a rationality check for all values and systems
has to be performed. These functions have to be realized
often with quite an effort; very different approaches have to
be taken, depending on the system and the exact fault to be
monitored. Hence a lot of functions with many different
structures exist, including multiple sensor usage, model calculation or intrusive checks with active impact on the engine
management for monitoring purpose only. Many of them
need very specific driving situations or engine conditions and
a certain amount of time, until the algorithm has entirely
finished and the diagnostic result is available.
There are other regulations like E-OBD for Europe or particular laws in Japan or Korea, but at the end of the day they
all point in similar directions and follow mainly the ideas of
the US Californian instructions, which hence are mainly
taken into account in the following.
The OBD regulation does not only describe, which systems
have to be monitored, but also detailed circumstances, under
which conditions and how fast a fault must be recognized,
how and with which tools the needed information has to be
delivered to the mechanic: All OBD faults have to be stored
in a standardized way and with particular additional information about the engine and vehicle states at the moment of
fault recognition. Furthermore, this information has to be
readable with a certain, also standardized tool, the so called
“Scan Tool”. The standardization of this tool and the fault
codes is done by the SAE and referred by CARB.
In a state of the art drive train, there are several hundred fault
codes (“diagnostic trouble code”, DTC) related to OBD
which have to be tested. For each DTC a standardized procedure has to be applied to ensure correct fault recognition,
storage of pending and confirmed fault code as well as failure
healing, see figure 1. Since this includes already several driving cycles for the testing of only one fault code, it is obvious,
that a lot of time is needed for the realization of an entire
PVE. In the PVE context, a “driving cycle“ is defined from
first vehicle start (“ignition on and start of driving”) until
turning off the vehicle. Hence depending on the particular
fault code, there are very short driving cycles (for testing fast
monitors) and some very long driving cycles for those monitors which need certain driving situations or engine states
(e.g. warmed up engine).
Nevertheless, since typical methods to apply the faults include changing particular engine components and only one
The regulations require the storage of a “pending“ fault code
after recognition during the first driving cycle and a “confirmed” fault code after recognition during the second driving
cycle. Additionally to the confirmed fault code, the “malfunction indicator lamp” (MIL) is lightened. After three fault free
driving cycles (DC), the MIL can be turned off, after 40 DCs
the fault code can be removed (“healing” of a fault).
2.2 OBD Testing
Furthermore, there are detailed guidelines, which tests have
to be shown to CARB to prove the correct functionality of
the monitor functions, which means nothing else than official
instructions for a validation of all OBD functions. For every
model year, six so called “test groups” per manufacturer are
728
AAC 2010
Munich, Germany, July 12-14, 2010
fault at a time can be tested, the most time is needed for these
mechanical work steps. For an official PVE of a modern
passenger car, typically two to three month are needed for the
entire completion.
Fig. 1. Standard procedure of PVE testing for one fault.
3. VALIDATION PROCESS
Besides the mandatory PVE described above, there are several necessary validation steps during the entire schedule of
the development process. The realization of this “official”
PVE takes long time, but since it takes place after the start of
production (and hence the preliminary end of the development), this is not the most crucial part. More important are all
those work steps which are integrated strictly in the entire
chronology of software development and calibration. This
working sequence takes place with alternating steps of ECU
software releases and calibration in which particularly the
drivability, the emission control and the OBD strongly interfere with each other.
Two different major objectives have to be distinguished: At
first, a “Pre-PVE” usually has to be done, which includes the
entire described process at an earlier time of the development
process. The objective is to ensure that all regulations are met
and the final PVE will be passed. The testing environment
should be as close to the final scenario as possible. Nevertheless, any validation which needs several months for completion cannot be feasibly integrated in the development process
itself. Hence some modifications are needed to speed up the
testing a lot, but without changing the important issues. Since
most of the time is needed for mechanical work, a good approach is to avoid failure methods which do need hardware
changes. Furthermore, only without hardware changes, automation of the entire process is really possible. Nevertheless
it’s important for the PVE that the validation can be done
without any re-calibration of other ECU-internal modifications and all tests for a pre-PVE can be applied at a production vehicle.
Fig. 2. Failure simulation methods for automated validation.
validation, a tool chain with suitable failure simulation methods is described below.
4. FAILURE SIMULATION
There are different possibilities to stimulate a monitoring
function. The easiest way usually is a change of the calibrated
thresholds, but this is suitable only for testing the correct
operation of the software code, but not the particular calibration. The change of characteristic curves of engine sensors is
often used during the calibration process itself, but usually
has to be realized also with changes in ECU re-calibration.
On the other hand, for the continuous validation a very fast
process for a check of all functionalities is needed; here it is
more important to get a result as fast as possible with as few
needed resources as possible. With development tools including Hardware-in-the-Loop-Simulators there is no need to
manipulate a real car for this, but failure simulation with
changes in a HIL simulation model (e.g. parameter change o
fthe engine model) is not usable for a transfer to vehicle related tests.
As described above, for the official validation of all OBD
functions, a typical failure simulation is done by hardware
changes, i.e. a particular component is exchanged with a
defect one or a limiting sample.
A simulation of a defect component itself can only be done, if
the system to be tested is simulated itself, i.e. if the test is
done at a HIL simulator. This is for some functions the easiest way, but can exclusively be used for HIL based testing
e.g. by change of model parameters (Fig 2a). Furthermore
detailed knowledge of the simulation model is needed. In an
To meet both needs, system validation which is applicable at
an arbitrary series production vehicle and fast functional
729
AAC 2010
Munich, Germany, July 12-14, 2010
environment, where a HIL is used as common part of the
development chain, it is often used as black box system for
ECU software development by the calibration engineers. The
model development on the other hand may be done by simulation experts who are not entirely familiar with the ECU
software.
5. TOOL CHAIN
Several components are needed for the basic realisation of the
described failure simulation as well as for the automation and
validation testing. The following steps have to be performed
to apply a simulated (sensor) failure. (In this example, failure
method 2c is chosen):
To be able to apply failure simulations automatically not
depending on the target system, another approach has to be
chosen. The entire connection of vehicle and ECU takes
place via electrical signals of sensors and actuators. If a failure shall be simulated without changing the ECU or even the
engine, only the signals remain to be manipulated. The easiest way for signal manipulation is with an open loop system
(Fig. 2b). Here an entirely new sensor signal is applied instead of the real one, but the information about the real measurement is not used or maybe not even available anymore.
This is suitable only for sensors which show usually nearly
fixed values, since the basic measurement of the particular
value is taken offline with strong impact to every control loop
which might use it.
• Break open the wiring harness.
• Read signal value externally on “engine side” of wiring.
• Manipulate the read signal.
•
• Automation of switching and manipulating.
The resulting component structure can be seen in figure 3.
For actuator failure simulation only the direction is changed,
i.e. reading takes place on the “ECU side” of the wiring and
applying the signal on the engine side.
To be able to deal with transient signals, a closed loop manipulation is needed (Fig. 2c). Only with this approach, relative manipulation can take place. e.g. “the air fuel ratio
(‘lambda’) shows 20% too much fuel (‘20% rich mixture’)”.
Breaking open the wiring harness for single signals is a
common task; so called break out boxes (BOB) are often
used during calibration or other engineering tasks on a vehicle or the HIL simulator. For the automated OBD validation a
particular version of a break out box is needed: On the one
hand, it is necessary to switch each signal to several states:
Short to ground, short to battery and open circuit are the
common electric faults and can be simulated directly with a
suitable failure insertion unit. Additionally for the more sophisticated signal manipulation a communication port (“com.
port”) is needed, where the raw signal can be read externally.
On the other hand the signal switching has to be done automatically; hence the entire failure insertion unit (i.e. the
switches for all signals) must be controlled by software and
not manually. For this switching purpose, an electronic break
out box and failure insertion unit (FIU) was built, consisting
mainly of several dSpace DS291 FIU boards. Each of these
boards can switch ten signals separately to ground, to battery,
to open circuit or to the com port of the board. Since each
board has only this one com port, it is important during layout design not to share the same board for those signals
which must be manipulated at the same time.
Both signal manipulation approaches are possible also for
actuator signals. There are two differences compared to sensor manipulation: On the one hand, a manipulated actuator
signal is closer to a real defective system (regarding nonsensor faults), since the affected actuator really changes the
system behaviour, while a manipulated sensor signal affects
the system itself only, if it is used for control reasons additionally to the monitoring functions. (Which often is the case,
but then the impact may even be bigger than with actuator
manipulation). On the other hand, most actuator signals need
an additional power stage and particular hardware is needed.
With suitable hardware, all the signal manipulation can be
realized without changing the ECU or the engine, but only as
additional components included in the wiring harness. In this
case, the manipulation system can be used without changes
on both vehicle and HIL simulator. If used on the HIL, the
same manipulation methods can even be used on the simulated signals – as part of the HIL model, but without need to
know deep dive details about the HIL simulation model except for the signal interface module (Fig. 2d).
Reading the signal and manipulating it is done with a dSpace
Micro-Autobox (MABX) [dSpace]. With the digital and
analogue inputs, all signals of the com ports are read. Inside
the Micro-Autobox, as on a HIL simulator, compiled Simulink models can be simulated and are used for signal manipulation. These manipulation models can be very simple, e.g.
consisting only of a gain, or more complex involving the
combination of several sensors at the same time. Since the
HIL simulation models are also based on compiled Simulink
models, these manipulation models can directly be integrated
in the HIL sensor models; hence without developing alternative failure simulation, the shown systems in figures 2c) and
2d) can be realized.
BOB with Switch
ECU
Vehicle / HiL
Com. Port
Manipulator
Apply manipulated signal to “ECU side” of wiring.
Automation
and Control
Fig. 3. Needed components for automated validation process.
730
While direct analogue or digital voltage signals can be used
directly (e.g. typical pressure sensors), for some sensors addi-
AAC 2010
Munich, Germany, July 12-14, 2010
Last but not least, not only the failure simulation, but also the
engine has to be controlled automatically. Using a HIL simulator, this is quite simple, but for the validation with the real
vehicle, this includes at first engine starting, stopping and
applying a certain angle on the accelerator pedal (in idle!).
For some high level monitors, particular driving situations
are needed; hence also full driving control has to be integrated e.g. control of a driving robot while driving on a test
bench. For full vehicle control, acceleration pedal and brake
pedal have to be manipulated. While the manipulation of the
acceleration pedal i.e. the signal generated from the acceleration pedal can easily be realized with manipulating the sensor
readings with the described methods, the brake pedal really
has to be pressed since the hydraulic brake pressure is mechanically generated. Hence a controllable unit already developed earlier at BMW has been used. Once these steps are
realized, the vehicle control is integrated also in the automation software using the existing manipulating capabilities of
the described tool chain.
Test Control
ECU-Test
FIU
ECU
„Write Addon“:
Sensor Specific
Sensor Simulation
Vehicle
HiL Simulator
„Read Addon“:
Sensor Specific
Measurement Device
Micro AutoBox
Control Wiring
Signal Wiring
Fig. 4. Realized tool chain for automated validation process.
tional hardware is needed. E.g. temperature sensors often
work as NTC (Negative Temperature Coefficient Thermistors) via measurement of a (temperature dependent) electric
resistance. Connected directly to the ECU, the evaluation of
this resistance needs a voltage divider inside the ECU as part
of the signal evaluation. This signal evaluation has to be rebuilt externally (“Sensor Specific Measurement Device”),
before the signal can be used with the analogue input of the
MABX. For some other sensors there are more complex
electric evaluation circuits integrated in the ECU; each of
them has to be re-built externally. One of the most complex
sensors is the wide range oxygen sensor, where the wanted
signal is the Nernst Current depending nonlinear on the
measured oxygen in the surrounding exhaust gas and where
additional fast control loops are included in the ECU only for
operating this sensor.
6. APPLICATION EXAMPLE
For illustration purpose, the described methodology is applied on the catalyst monitoring. As main parameter for an
evaluation of the conversion capability of a catalyst, the so
called “oxygen storage capacity” (OSC) is used. Roughly
speaking, the more oxygen a catalyst can store, the better it
is. During the calibration process, depending on emission test
results, a catalyst with an OSC below a specific value is defined as “defect”. Hence, to recognize this threshold OSC, is
an essential task of the monitoring function. One possibility
to identify the OSC, is to apply a constant “lean mixture”
(i.e. inject less fuel than could be burned with full usage of
the currently available oxygen in the fresh air provided to the
engine). Since the downstream sensor is only reached by
measureable oxygen, if all storage capacity of the catalyst has
been exceeded, the time the lean mixture reaches the sensor
behind the catalyst, is a measure for the OSC.
The output of the manipulated signals works vice versa. The
same DS291 FIU boards are used for re-applying these signals, hence for one signal manipulation, two boards are
needed (but with these two boards, up to ten different signals
can be manipulated). Here again, some signal conversion has
to be done depending on the sensor signal (“Sensor Specific
Sensor Simulation”): If e.g. the signal is not voltage but current, there has to be an external converter, since the MABX
does not include current outputs.
The mixture before the catalyst is measured by the wide
range “upstream oxygen sensor”, the mixture behind the
catalyst by the two point “downstream oxygen sensor”. Depending the applied lean mixture, the current engine air mass
flow and particularly on the time which the oxygen needs to
“break through” the catalyst, the OSC is calculated. If it exceeds a certain limit, a fault code is set. The fault free case is
shown in fig. 5: The calculated OSC value is integrated during the “oxygen storage phase” beginning with measured lead
mixture upstream until the breakthrough downstream is recognized. For reassurance this process is repeated several
times, here six times.
Finally the entire system has to be controlled. At first this is
the software to manipulate the model parameters on the
MABX and the switching of the FIU. This is done with the
dSpace software “ControlDesk”. Furthermore, the main
automation tasks have to be included. Here the software
“ECU-Test” is used [Tracetronic], which mainly acts as a
remote control for all other used software, like ControlDesk,
the calibration software INCA, the Scan Tool software DTS
and some other BMW internally software tools. Which tools
besides the Scan Tool exactly are used for measuring purpose
depends on the desired evaluation. Once a general reference
test has been designed, it is easy to include additional measurements or test scenarios, e.g. to include desired information
about some ECU internal values about the functions or resulting limp home modes. However, ECU-internal measurements
are only available, if a calibratable development ECU is used,
not one for a series production vehicle.
For validation purpose, usually the working catalyst of an
engine is exchanged with a known, malfunctioning one. In
this case, only for testing one single monitoring function,
several days of mechanical work have to be included. For the
official validation (PVE) where the failure methods do have
to match exactly the approved ones, this effort strictly has to
be done, but for internal testing on the vehicle as well as on
the HIL, this mechanical work can be avoided by manipulating the signals of the oxygen sensors. A simulated downstream “lean spike” directly after the upstream sensor has
731
Begin of upstream
lean mixture
1.08
1.04
1.00
0.96
Air-Fuel Ratio [-]
0.92
lean „break through“
0.2
180
0.0
failure threshold
120
60
0
0.96
0.90
voltage of downstream
oxygen sensor [V]
0.6
0.4
1.02
Begin of upstream
lean mixture (AFR>1)
1.02
1.00
0.98
0.96
0.92
0.8
0
0.94
0.8
0.6
0.4
0.2
lean „break through“
without previous
storage phase
80
0.0
failure threshold
40
0
-40
time[sec]
Air-Fuel Ratio [-]
0.96
1.08
OSC value [mg]
1.00
oxygen storage phase
voltage of downstream
oxygen sensor [V]
Air-Fuel Ratio setpoint [-]
1.04
error bit [-]
1
1.08
OSC value [mg]
Air-Fuel Ratio setpoint [-]
AAC 2010
Munich, Germany, July 12-14, 2010
time[sec]
Fig. 5. Catalyst monitoring (fault free).
Fig. 6. Catalyst monitoring (with simulated defect catalyst).
detected lean mixture (AFR > 1) simulates a catalyst without
any OSC at all; additional waiting time until the spike enables “simulated OSC tuning”. The result of a “no OSC
fault” is shown in fig. 6; after the sixth stimulation without
reaching the OSC threshold due to the direct breakthrough,
an error is set. To stimulate the monitor function correctly,
the manipulation of the downstream sensor signal has to be
applied exactly depending on the upstream signal.
for even a real vehicle does show a strong improvement in
testing automotive monitoring functions.
Despite the main motivation for the presented methodology
and tool chain was the validation of OBD monitoring functions of the ECU, further steps are self-evident: The transfer
to non-OBD monitoring functions and validation of other
control units (e.g. hybrid control units, transmission control
units) have already been planned or realisation has already
begun, discussions about limp home testing are ongoing.
7. CONCLUSION AND OUTLOOK
For quality improvement in the software development and
calibration process despite the entire development time getting shorter, an automated validation process particularly
suitable for monitoring related testing has been presented. For
the validation of diagnostic functions, failure simulation is a
essential task. Depending on the used system, different methods of failure simulation and resulting tool chains can be
used, the presented approach is capable of re-using the failure
simulation methods not depending on the test bench. With the
described system, the entire validation of all approx. 400
OBD functions of BMW series production vehicles has been
realized. Several projects have meanwhile been tested; due to
the highly improved speed for this testing, validation of these
monitoring functions can take place more often and earlier in
the entire development process. Hence, the influence of even
late changes can be detected easily and development quality
is improved. Furthermore, it is possible to realize a preliminary version of the for OBD functions mandatory validation
at an early development stage with a system as close as possible to the final serial vehicle. Many of the single elements
of the presented approach are already used in several areas of
software testing; the entity and combination of all these parts
to the fully automated tool chain including failure simulation
8. REFERENCES
CARB: California Air Resource Board (2007). Final regulation Order for OBD II and Emission Warranty Regulations, Section 1968.2. Approved on November 9, 2007.
OBD II regulations and Rulemaking:
http://www.arb.ca.gov/msprog/obdprog/obdregs.htm.
dSpace: Description of MicroAutobox (2009).
http://www.dspace.de/ww/de/gmb/home/products/hw/mi
cautob.cfm
Tracetronic (2009): Description of ECU-Test.
http://www.tracetronic.de/produkte/ecutest.html
732