Software BIT Design and Testing for Embedded
Software
Yichen WANG
Zhenzhen ZHOU
Department of System Engineering
Beihang University
Beijing, 100191, China
Department of System Engineering
Beihang University
Beijing, 100191, China
•
Abstract—Build-in-test of software is designed in embedded
system widely. This paper analyses the general architecture of the
embedded system, introduces three types of BIT structure
pattern, for each pattern, the arithmetic design is described in
details with real practice. This paper presents a new s/w BIT
approach based on fault injection. Four important phrases:
testing requirement analysis, fault injection, testing design and
result analysis are described thoroughly in this approach. The
ending part is the summery of the paper.
Since more and more functions are implemented by software
now, the number of hardware related increases rapidly. The
self-examination of those hardware devices must be done with
the help of software. Up to now, the research on software BIT
just start. In fact, it is still an expansion of hardware testability
design, which is different from software BIST (build-in-selftest) slightly.
Keywords- embedded software; software testability; software
Build-In-Test; fault injection; software testing
I.
Fault injection was first proposed in the early stage of last
70s, and then adopted to validate the design of Fault-tolerant
system by industry. After being introduced to universities and
research departments, it attracted more people’s attention in
last 90s, leading to the result of deeper researches and wider
applications of fault injection. Actually, because of different
approaches during its application, fault injection technique can
be sorted in many ways.
INTRODUCTION
In china, the researches and applications on testability/BIT
are developed a bit later than some other foreign countries.
However, many accomplishments have been achieved,
especially in the field of national defense where testability is
required in important systems and equipments. Nowadays,
great efforts relative to the design and analysis on
testability/BIT have been made, coming along with a series of
national military standards. Also, a problem begins to attract
people’s eyes that our researches of testability are confined to
hardware without enough attention paid to software, although
the importance of design for BIT in the field of aeronautics and
astronautics and weaponry increases rapidly.
In this paper, we focus on the design pattern and test
approaches of software BIT. Our research, which is based on
the analysis of typical software BIT algorithm selected from
real equipments, describes the architecture and algorithm
patterns of embedded software and proposed software BIT test
method based on fault injection technique. We first analyze the
requirements of software BIT according to the design patterns
and fault patterns of embedded software, and then complete the
test designs combining software/hardware fault injection
technique, and evaluate the results of software BIT with test
credibility factors being considered finally.
Built-in test (BIT for short) can generally be described as a
set of evaluation and diagnostic tests that use resources that are
integral parts of the system under test, in order to facilitate
diagnosis and improve testing level. A system may tell us
whether itself is healthy and locate faults, using hardware and
software units or self-test devices. Thus, we can monitor the
system status and detect and isolate faults of it. In other words,
BIT is a kind of auto testing capability provided by the system
and fault detection/isolation devices. Generally speaking, there
are four kinds of BIT divided by their working mechanisms
and time:
•
Consecutive BIT—monitoring working status of the
system consecutively.
•
Periodic BIT—carrying out test in a certain frequency.
•
Set-up BIT—the monitoring operation is activated only
by the outside event, a startup from the operator, for
example.
978-1-4244-4905-7/09/$25.00©2009 IEEE
Maintenance BIT—executing maintenance test after the
system task is finished, for instance, after landing of
plane. It can start up any kind of BIT of the system,
including itself, of course
II.
SOFTWARE BIT DESIGN
A. Software BIT Structure Pattern
The BIT of embedded software realizes the functions of
fault detection and isolation and fault alarming for embedded
system and interconnected system by using software algorithm
As figure1 shows, the hardware environment required for
embedded software can be divided into three levels:
•
703
On-board component level: including CPU, RAM,
NVM, and Timer and so on.
•
System inner interface level: including all the
input/output interfaces, all the sensors and actor, and
specified graphic interface.
•
Outer environment level: including mainly physical
environment and all the devices which are connected
with embedded system through outer bus or interface.
board components so much that on-board detecting capability
of software BIT to such on board components is very limited.
The principle of software BIT design bases on hardware
failure models. Software takes effect in this way: check if
failure has occurred with the algorithm designed according
failure model; if it does, raise an alarm with proper failure
handling mechanism designed. Considering RAM on which the
execution of software depends highly. In common sense, the
capability of writing and reading should be tested for all the
memory devices, including RAM, ROM, and NVRA etc. In
fact, the common failure model of RAM is writing and reading
failure. So, a typical RAM self test algorithm can be described
as follows:
{…
int N1 = 0xAAAA;
write N1;
read N1;
int N2 = 0x5555;
write N2;
read N2;
int N = N1 + N2;
if(N != 0)
send_fault_info(); /* report the selt testing fault
of RAM */
else
…
…
}
.
Figure 1. Typical structure of embedded system
Generally speaking, embedded software BIT implements
hardware self test of embedded system with the help of
software algorithm. According to the structure shown in figure
1, embedded software BIT can be classified into three types:
•
On-board Component BIT: Software implements self
test for on-board component, including CPU, all the
RAM and Timer and so on.
•
System inner Interface BIT: Software implements self
test for input/output interface board.
•
System BIT: Software implements self test of
interconnected devices that are connected with the
system.
This algorithm writes some constant data into RAM first, in
order to validate writing capability. Then it reads this data from
RAM to validate reading capability. At last, it adds up the two
constant data read from RAM to validate accuracy of the RAM.
In some systems which require high safety this algorithm needs
loop mechanism further, for the purpose of checking every
memory units. What’s more, for some special memory devices,
such as non volatile memory, extra algorithm is needed to
validate their specialties.
2) The Algorithm Design for System Inner Interface BIT
Inner interface is the key part of a system, playing the role
of exchanging data between software to be tested and outer
devices or environment. From the viewpoint of failure model,
we focus our attention about the design of system inner
interface BIT on two aspects, the correctness of hardware
connecting with inner interfaces and the correctness of sending
and receiving data from interfaces. Consider the design of 422
bus BIT as an example, which is common in embedded system.
The algorithm can be designed to check if the hardware link is
correct by initializing hardware first and then sending and
receiving a frame of data from the 422 bus. The next step is to
compare sending with receiving data to determine the
correctness of transfers. At last, record the test result or go
down to the fault solving process.
Figure 2. Embedded software BIT architecture model
B. Software BIT Algorithm Pattern
1) The algorithm design for on-board component BIT
On board components influenced the execution of software
most, for instance, if CPU, RAM and times or other important
components fail, the whole software will stop or fatal error will
occur. But the execution of software is dependent on those on
A typical 422 bus self test algorithm:
{…
Set_BIT_lock; /* set BIT lock */
init(); /* initiate the 422 interface */
704
if (BIT data frame1 != BIT data frame2)
send_fault_info();/*report the fault of interconnected
equipment A */
else
return the final result of BIT between systems;
…
}
write (BIT data frame1);
send(BIT data frame1 ) to 422bus;/* write data into the
interface of 422 bus */
receive(BIT data frame2);/* receive the data from 422
bus */
read(BIT data frame2);/* read out BIT test data */
if (BIT data frame1 != BIT data frame2)
send_fault_info(); /* report self testing fault of 422
interface */
else
return to the result of interface examination;
…
}
III.
THE BIT TEST APPROACHES BASED ON FAULT
INJECTION
From the analysis of structure and algorithm patterns of
software BIT, we may draw the conclusion that the test and
validation for embedded software should put more emphasis on
estimating software’s ability for detecting and handling faults,
when the function requirements of software BIT are taken into
consideration. To be specific, on one hand, we should validate
the design of software BIT and its capability of detecting
hardware failure from embedded system. On the other hand,
we should examine whether software BIT has a prominent fault
handling process in which the detected hardware fault can be
informed to the user or the last caller correctly.
3) The Algorithm Design for BIT among Systems
BIT among systems is a kind of examination for the
working status of interconnected devices, which is always
carried out after system inner interface test has been completed.
This algorithm goes step by step in the following way. First, let
the system being tested send a preliminary signal or a frame of
data to the interconnected device to be tested through a fixed
bus interface. Secondly, wait until the device sends back self
test result. Thirdly, evaluate the return value to tell whether the
device is in right status. Finally, record the test result or go
down to the fault handling process.
In order to satisfy all the test requirements mentioned above,
this paper proposes a test approach for embedded software BIT
based on the fault injection technique, considering the character
of software BIT design at the same time. This approach tests
and validates the two main capabilities of software BIT by the
way of simulating hardware failure of embedded system. There
are two ways on how to simulate hardware failure, software
fault injection and hardware fault injection.
A typical algorithm of BIT between systems is as follows:
{…
write (BIT data frame);
send(BIT data frame ) to equip; /*send a frame data to
the bus interface of equipment A */
receive(BIT data result) from equpA;/* receive bus data
from equipment A */
read(BIT data result);/* read out the data from BIT test
result */
The key aspects of the test approach based on fault
injection are specifying the hardware failure models of system,
acquiring test requirements and designs according to these
models, as well as the execution of this test following. Four
main steps of this approach are shown in figure 3.
Figure 3. The test approach based on fault injection
705
RAM correctly”, we need to break the pins of memory unit.
No need to say, this devastating approach could not be adopted
widely in practice.
A. Specify Test Requiremen
Regarding to the function requirements and design patterns
of software BIT, a complete BIT test requirement should
contain two aspects:
•
•
Software fault injection simulates hardware faults by
changing the source code of the software to be tested. In
common sense, it is an operational and effective approach.
According to the failure model in software test requirements
and BIT algorithm, we can determine where and how to inject
code for simulating this model.
Whether software BIT could detect all the possible
hardware failure models of system correctly?
The final goal of software BIT design, which is also the
main content of BIT software test, is detecting all the hardware
faults of system and handling of them correctly. Acquiring
software test requirements that require us to analyze all these
faults is on the basis of software BIT design requirements and
the library of hardware failure models. After specifying the
failure model, we should analyze the implementation process
of BIT in order to build the relationships between failure
models and software system status. Obviously, this is the
foundation of determining expected test results in test design
phase.
Different from hardware fault injection, software fault
injection concerns two extra aspects.
•
•
Testing credibility
Fault injection is a kind of emulation technology. So the
credibility of software fault injection should be analyzed in
two aspects: the simulation fidelity determined by the code
inserted by program instrument and its location; the influence
to the running and status of original code affected by program
instrument.
C. Design Test Cases
Since test requirements have determined the goal and
content and fault injection technique has provided proper
approach for test, the next stage—test case design, will
describe operation steps in details. In this stage, several points
should be concerned.
Whether software BIT has alarmed the hardware faults
which do not exit in fact?
False alarm which means that BIT raises alarm for fault
that does not really exit is an important concept in the research
field of software testability. This topic is out of the scope of
this paper, we’ll discuss it in another one.
B. Specify Fault Injection Mechanism
From software BIT and test requirements, we can tell that
hardware failure model is a critical factor which affects the
performance of software BIT. Whether software BIT is
complete and right quite relies on collecting failure models in
design stage. Moreover, whether software BIT can be tested
completely and correctly depends on simulating failure models
in test stage.
As software test requirements have been specified, we
should select a proper injection mechanism for each failure
model according to the test condition, in other words, fault
simulated approach. There are two fault simulated approaches:
software fault injection and hardware fault injection.
•
Testing adequacy
Software BIT aims at detecting hardware faults and then
dealing with or alarming them. Actually, the failure model
simulated by software can not take the place of real hardware
failure model. Their manifestations are similar but not equal.
The difference needing to be analyzed in test process depends
on the place where the failure model is injected.
Considering the RAM example mentioned before, we can
analyze the failure model in the following ways. According to
the design requirement of software BIT—examining RAM, we
can get two failure models: “can not write into RAM
correctly” and “can not read from RAM correctly”. As
concerning with the type of RAM and with the help of our
experiences, we can get another failure models, such as
“NVRAM loses its non-volatile property”, “Erasable RAM
fails to erase data”. Then referring to software requirements,
we can obtain explicit connections between faults and system
status.
•
Software fault injection
Hardware fault injection
Hardware fault injection is a more realistic test approach,
which requires that the desired fault exits in hardware. For
instance, in order to get the failure model—“can not write into
•
Test case illustration: the requirement of test and the
illustration of the hardware failure model to be
simulated.
•
Test condition: the fault injection mechanism adopted
as well as its location and method.
•
Test input: the critical part of test case design which
requires us to list the steps for test case execution in
details, including detailed fault injection operations,
related data input and specific operation steps of
software.
•
Result expected: the criterion for determining whether
a test case has passed correctly and whether software
contains faults. It demands us to list the software
behavior or data exhibition for each input, according to
the test input and software requirement.
Other information about test case is shown in Table 1.
706
TABLE I.
A TEST CASE TABLE BASED ON FAULT INJECTION
Editor:
Investigator:
Date:
Test case illustration
the requirement of test and the illustration of the hardware failure model to be simulated
Test case label
the unique label of test case
Test condition
the fault injection mechanism adopted as well as its location and method
Test input:
The specific illustration of input for the running of test case, including:
Input specification: the name of each test input and explanations of some special cases.
Input source: the illustration of where the input comes from, such as the output of the program being tested, files on hard disk, network, or keyboard.
Input character: the illustration of whether it is real or simulated.
Input steps: the input time or order and detailed operations.
Expected output: the expected output for above input.
the terminal condition of the test case.
Terminal condition
Criterion
the criterion for determining whether the test case has passed correctly.
Now, the software BIT test based on fault injection has
been successfully applied to several software test projects of
weapon equipment. And many critical faults have been
detected, such as the self test of “NVRAM loses its nonvolatile property”.
D. Analyze Test Results
Each designed test case should be executed in the
appropriate test environment, which requires its execution data
can be obtained at the same time. The execution processes and
results of test cases should be described in records. Comparing
the result with expected output, we can figure out that whether
or not a test case has passed correctly.
REFERENCES
[1]
Since software BIT test based on fault injection is an
expansion of fault injection test which belongs to simulation
test, the analysis of test results should take the simulation
approach and the location of fault injection into consideration.
IV.
[2]
[3]
CONCLUSION
[4]
BIT is an exterior character of software testability which is
an important attribute of software quality. This paper has
analyzed and summarized the familiar BIT design patterns and
then proposed a new approach for software BIT test. This
approach can analyze and more importantly test the function of
software BIT completely.
707
Z. Tian, and J.Y. Shi, The Design and Validation of System Testability.
Beihang University Press, April 2003.
J.Z. Sun, J.Y. Wang, and X.Z. Yang, “The Present Situation For
Research of Fault Injection Methodology and Tools,” Journal of
Astronautics, 2001, Vol.22, pp.99-103.
W. Li, S.Y. Xu, and H.S. Bi, “Mounting and Statistic of Checkpoint in
Built-in-self-test for Software,” Computer Engineering, 2005, Vol.31,
pp.96-98.
R.Z. Jia, L. Liu, X.R. Wang, and W.J. Zhan, “Automatic Test Software
of Verification and Validation for Simulation System,” Acta Simulata
Systematica Sinica, 2000, Vol.12, pp.617-620.