Test Process
Management (TPM)
Software Tool Suite for Verification
& Validation of Electronic Equipment
Product Overview
1
Table of Contents
Introduction 3
Test Life Cycle Overview
4
5
TPM Modules
6
TPM Concept
TPM Creation
7
8
TPM Editor
TPM Test Definition Elements
9
12
TPM SIgnal Browser 13
TPM Executor
14
Test Documentation
15
Test Execution
TPM Reporter
Test Scenario Example Summary
2
16
16
18
Introduction
Introduction
TPM Overview
TechSAT’s Test Process Management (TPM) application is a comprehensive software tool
suite for managing the entire life-cycle involved in the testing of electronic equipment.
TPM supports you in all phases of the test process, from importing test requirements
and system ICDs, creating test cases and test plans, executing tests, to generating test
reports and performing impact analyses.
As an Automatic Test Equipment (ATE) for testing, validating, and certifying electronic
equipment, TPM can perform both interactive and fully automated tests. Therefore, TPM
is ideally suited for production, integration, and maintenance testing.
TPM Benefits
TPM offers the following benefits:
>> Reduced
costs through the use of a single tool suite for the entire test life-cycle
test implementation effort through the use of intuitive form-based test definitions
>> Complete traceability from system requirement to test result
>> Increased productivity through a high degree of automation during all stages of the test
process
>> Improved test quality through consistent use of the same test process
>> Reduced
TPM
Test Process Management
3
Test Life-Cycle Overview
The figure below shows the test life-cycle
using DOORS™ together with TechSAT’s
TPM and Avionics Development System
ADS2.
DOORS is used for managing system requirements, test requirements, test results, and all
interrelations between them, thereby providing
traceability.
DOORS
By importing test requirements from DOORS
into TPM, it is possible to define test cases,
based on which test plans can be established.
The test plans are then compiled and built
into test executables. These can be executed
on ADS2, which serves as the test execution
engine.
During test execution, the test results are fed
back to TPM to generate a test result report.
Subsequently, the test results can be exported
to DOORS to create result and traceability
matrices.
Functional/
Functional/
System
System
Requirements
Requirements
Traceability
Traceability
Matrix
Matrix
Result
Result
Matrix
Matrix
Test
Test
Requirements
Requirements
Test
TestCases
Cases
&&Test
TestPlan
Plan
Test
Test
Result
ResultReport
Report
TechSAT TPM
Test
Test
Executable
Executable
TechSAT ADS2
4
Test
Test
Results
Results
Test
Test
Execution
Execution
TPM Life-Cycle Overview and Concept
TPM Concept
The test category covered by TPM is »black
box testing« of any equipment that is connected to its environment through I/O connections.
The idea is to stimulate the inputs of the unit
under test (UUT) and verify the outputs against
the expected values.
TPM test programs do not access the I/O
lines directly, but use the concept of signals as
an interface. Signals are named data objects
of different data types carrying a value and
a timestamp. TPM manipulates and verifies
only such signals. The transfer of signals to/
from physical I/O and thus to/from the UUT
is performed by the underlying test execution
engine based on ADS2. The following diagram
shows the data flow between TPM and a unit
under test:
ADS2 Execution Engine
TPM
Test Program
created with TPM Editor,
then compiled to executable test
program
Signals
I/O Hardware
I/O Connections
unit under test
PM
mpiler
5
TPM Modules
The TPM software suite is comprised of the following modules:
„„ TPM Editor – used to create and modify
test cases based on the built-in test definition elements. These can be linked to
test requirements and access signals as
defined in the system ICD. Test cases can
be combined to test plans.
„„ TPM Test Definition Elements - used
to create test cases. TPM provides different types of test definition elements, e.g.
for signal stimulation and verification, flow
control, and event handling.
„„ TPM Compiler – generates C code from
the defined test plan which is then built to
a test executable that can be executed on
the test rig. The compiler is an integral part
of the TPM Editor.
„„ TPM Signal Browser – used to select
signals and insert them in test definition
elements. The browser allows importing
system ICDs and provides powerful search
and filter options.
„„ TPM Executor – manages the test execution and offers monitoring, debugging, and
tracing capabilities.
„„ TPM Reporter – creates customizable test
result reports based on the test case definitions and the test results. Additionally, the
TPM Reporter can generate test description reports before the tests are executed.
The following diagram shows the TPM modules and the data flow between them.
Requirements
TPM
Editor
ICD
Test
Plan
TPM
Reporter
Test
Description
Report
TPM
Compiler
Test
Executable
TPM
Executor
Test
Result
TPM
Reporter
Test Result
Report
6
TPM Modules and Test Creation
Test Creation
Test Creation
Requirements
(e.g. DOORS)
Test
Concept
System ICD
Test Case
Editor
Test Plan
Compiler
Test
Executable
Test Job
Signal
Browser
The flowchart above shows the test creation process with TPM.
The TPM Editor can import test requirements from requirement engineering tools like
DOORS through XML or CSV files.
By importing a requirement document (e.g. a
DOORS module), a test definition document
is created, which has the same structure as
the imported requirement document. For each
requirement an empty test case is created,
which is linked to the corresponding requirement. The test definition document can be
navigated through a tree browser, which
reflects the structure of the document.
According to the associated requirement each
test case has to perform certain operations
on the system under test to determine if the
requirement is fulfilled. These operations are
carried out on process variables or signals,
which are defined in the system Interface
Control Document (ICD). With the TPM Signal Browser, the ICD can be imported in XML
format and stored in a signal database, which
can be accessed by the test cases. The signals can therefore be referenced directly in the
test cases or inserted from the signal browser
via drag and drop.
When requirements are imported, TPM creates
merely the test case structure. It is therefore
up to the test engineer to implement each test
case based on the test concepts. The test
cases are implemented using the test definition forms in the TPM Editor.
Once the test cases are implemented, it is
possible to package subsets into a test plan.
After invoking the editor‘s integrated compiler,
it first converts the selected test plan into
C-code. Subsequently a C-compiler compiles
and builds the test plan into a test executable
which will execute on the test engine, perform
the specified system operations, and deliver
the test results. Currently, the TPM Compiler
can generate test executables for Windows,
Linux, and VxWorks platforms.
An additional output of the TPM Compiler
is the test job, which is a complete description of all test plan components, including the
name of the author, revision number, etc. This
information is used to create test description
reports as well as test result reports.
7
TPM Editor
The TPM Editor GUI and its main components are depicted below.
1
2
3
1 Test Project View
2 Test Definition Element
Property View
3 Online Help View
4
The test project view (see above) displays the
components of a test project: the »test case«
list and the »test plan« list. All test cases are
organized in the test case list, and all test
plans are organized in the test plan list.
By clicking the expand/collapse icons, you
can expand or collapse the content of the test
definition element, thereby changing the level
of detail displayed in the view.
8
4 Console Output View
When you select a particular definition element
in the test project view, its properties are displayed in the test definition element property
view for editing.
The online help view provides detailed information on how the definition element works and
how it can be configured.
The console output view lists diagnostic information of the TPM Editor and compiler.
TPM Editor and Test Definition Elements
TPM Test Definition Elements
Test definition elements are used to implement test cases. They are represented in
corresponding forms provided by the TPM Editor. The main test definition elements are
depicted below.
9
Test Case
Epilogue
The Test Case is a container for the test procedure and the corresponding test requirement. It is usually created automatically when
you import test requirements into the TPM
Editor. However, you can create additional test
cases if desired. Test cases can be referenced
in test plans to create specific test scenarios.
The Epilogue is a non-realtime element which
concludes the Test Procedure. It is typically
used to reset the system to its initial state and
to unload previously loaded resources.
Test Requirement
A Test Requirement element is created for
every imported test requirement. Test Requirements contain a general description of the
requirement as well as a description of the
required system conditions and the expected
results.
Time Slot
The Test Step contains one or more Time Slot
definitions. Time Slots are defined by absolute
or relative time, by events, or by conditions.
Within a Time Slot, multiple signal stimulation
or verification operations are performed concurrently.
Time Slots may overlap, thus allowing a very
natural way of defining test conditions.
Examples:
Test Procedure
The Test Procedure is a container for the
executable part of the test case. It typically
contains an initialization component called a
Prologue, one or more Test Steps, and an Epilogue.
Test Step
The Test Step implements a sequence of system operations executed in real time. Typically,
it first stimulates the system, then waits for an
event to occur, and finally verifies that the system reacts as expected, i.e. that it fulfills the
system requirement associated with the Test
Step.
Prologue
The Prologue is a non-realtime element used
to initialize and prepare the system for the
subsequent Test Steps. The Prologue is typically used to preset signals and load resources, such as simulations and panels.
10
>> 2
sec after PowerOn until PowerOff:
verify voltage > 10V
>> on WOW == False until WOW == True
verify doors all closed
Time Slots are event controlled, i.e. the execution of the contained statements depends on
the occurrence of the event that triggers the
time slot. Typical events are, for instance, the
elapse of a time delay or a signal reaching a
certain value. Per default, a Time Slot is triggered when the previous Time Slot is finished.
To trigger the execution of Time Slots, you can
either use system events or define your own
events.
TPM Test Definition Elements
Statements
Statements are actions such as signal stimulation, signal verification, and dialog pop-ups. All
Statements grouped in the same Time Slot are
executed in parallel. By default, a Time Slot
finishes when the last active statement in the
Time Slot has finished. However, if you specify
a timeout, the Time Slot is forced to finish
upon the elapse of the timeout period, regardless whether there are still running statements.
TPM supports the following types of statements:
„„ Stimulation Statements
Stimulation statements are used to set
signals to constant values or to values
computed by a function generator.
Examples:
- set A to 5
- hold B at 7
- ramp A from 0 to 100
The function generator outputs are continously assigned to the signal as long as
the statement is running.
„„ Verify Statements
Verify Statements are used to compare
actual signal values with their expected values for deciding whether a requirement is
fulfilled or not. TPM offers several types of
Verify Statements, including:
>> VerifyLimits
– verifies whether the signal
value is between the given minimum and
maximum value.
>> VerifyTolerance – verifies whether the
signal value is within the tolerance of the
given nominal value.
>> VerifyExact – verifies whether the signal
value exactly matches the nominal value.
>> VerifyExpression – verifies whether the
given expression is true.
>> VerifyNoChange
– verifies that the signal
value remains unchanged during the
time slot period.
„„ Trigger Statements
Trigger statements are used to generate
events on certain conditions. When an trigger expression evaluates to true, its associated event is raised possibly starting or
terminating a time slot.
„„ BranchCondition Statement
This statement is used to define a list of
trigger statements. The BranchCondition
Statement can raise one event only.
„„ Dialog Statements
These statements are used to create
message boxes and input dialogs.
„„ UserCode Statement
The available TPM statements are sufficient to implement most test cases.
However, for special complex behavior, the
TPM statements may not be sufficient to
define a test. In this case TPM provides the
framework to integrate customized C-code
through the UserCode Statement. Using
this statement, TPM will automatically
generate templates for the include file, the
source file, and the make file. This makes
it simple to add the necessary code and
integrate it into the Test Case.
Libraries
Often test cases have certain repeating elements. For this purpose, TPM provides the
Library Definition Element, which makes it
possible to create libraries of Test Procedures,
Test Steps, Prologues, and Epilogues. These
can be referenced (called) in different test
cases.
11
TPM Signal Browser
The Signal Browser is a support utility for the Test Editor. It is used to select and insert
signals into test definition elements. For this purpose, the TPM Signal Browser can import
the ICD of the system under test. The Signal Browser GUI is shown below.
In addition to its ICD import functionality,
the Signal Browser can be used to search
for required signals in the database. Found
signals can be selected with the mouse and
inserted into the test cases via drag and drop.
You can search the database for particular
signals by applying custom match patterns
on signal names or on information which is
specific to the signal type. For example, you
can search for AFDX signals with a certain VL
identifier and for MIL-STD-1553 signals with a
certain RT address.
The Signal Browser has four built-in filter
types:
1)
2)
3)
4)
CVT
AFDX
A429
MIL-STD-1553
The CVT filter is a search filter based on signal
names only. All other filters provide search criteria that are specific to a particular I/O type.
12
TPM Signal Browser and Test Execution
Test Execution
Test Creation
Requirements
(e.g. DOORS)
Test
Concept
Test Execution
HW & SW
Resources
Test Case
Editor
Test Plan
Compiler
Test
Executable
Test Log
Executor
Test Result
Signal Browser
The flowchart above illustrates the Test Execution phase, which follows immediately after
the Test Creation. The Test Execution is managed by the TPM Executor, which uses, as
described earlier, the test executable and the
test job as inputs. To perform the execution,
the Executor additionally needs hardware and
software resources, such as the test rig with
the test execution engine (ADS2), simulations,
etc.
The Executor does not only allow loading and
executing different test executables, but also
provides means to control the execution using
step execution with user-defined step time
intervals.
The TPM Executor generates a test log file,
which contains the value history of all signals
referenced in the executable. The test log file
is very useful for tracing and post-analyzing
signal values as they were recorded during the
test run. The test log file can be customized to
contain only information on a selected number
of signals. Besides in ASCII format, the test
log file is also stored in the SQLite database
format (sld) for better browse, filter, and search
capabilities.
Test Job
Finally, the Executor generates a test result file
in XML format. The test result file is based on
the test job file and the output of the test executable. The following information is contained
in the test result file:
„„ Exectuted test components (test components that were not executed are discarded) and their associated results (passed/
failed)
„„ Information on potential user interference
during execution, for example, aborting,
pausing to step through, etc.
„„ Absolute timestamp information regarding
the start and end of individual test components as well as the entire test
„„ Miscellaneous information such as the
name of the test person, the TPM Executor
revision, etc.
From this pool of information, the TPM Test
Reporter builds customizable reports.
13
TPM Executor
The TPM Executor GUI is divided into six areas as shown below.
2
1 Test Job View
3
6
4
2 Test Component Properties View
3 Execution Console View
4 Execution Control View
1
5
5 Execution Engine (ADS2) Log View
6 Signal View
The Test Job View on the left-hand side displays the tree structure of the loaded test job.
By clicking a test component in the tree structure, you can display the corresponding properties in the Test Component Properties View.
You start the test execution by clicking the
„START“ button in the Execution Control View,
and you terminate it by clicking the „STOP“
button. When you stop the execution, the
Executor will finish the current cycle and then
execute the Epilogue component before terminating the test. Furthermore, you may pause
the automatic execution and continue the test
manually step by step in time intervals specified either in seconds or number of cycle times
(or time frames) of the underlying execution
engine. You can resume the normal execution
at any time.
14
During execution the execution details are
automatically displayed in the Execution Console View. This view does not only list the test
component currently executed, but also the
absolute timestamp of its execution and the
associated events. For better visualization,
the Execution Console View highlights passed
verify statements in green and failed verify
statements in red.
During test execution the tree structure in
the Test Job View is automatically expanded,
thereby indicating the execution progress.
Analog to the Execution Console View, executed test procedures are highlighted either in
green or red.
Finally, the Execution Engine Log View provides information on the status of the ADS2
execution engine. You can set the level of
information detail by selecting the message
categories to be logged: errors, warnings, and
information.
Test Execution and Test Documentation
Test Documentation
Test Creation
Requirements
(e.g. DOORS)
Test
Concept
System ICD
Test Execution
HW & SW
Resources
Test Case
Editor
Test Plan
Compiler
Test
Executable
Test Log
Executor
Test Result
Test Job
Signal
Browser
Test
Documentation
The last step of the test process involves the
test documentation, i.e. the generation of a
test report in PDF format, and the transfer of
the test results back to the requirements tool,
if such a tool is used.
As depicted in the flowchart above, test
reports can be generated at two points in the
test process.
First, you can create a test description report
based on the test job file generated by the
TPM Compiler. Usually, you create the test
description report before you start the test.
This way you can print an overview of the test
you intend to execute.
Second, the TPM Reporter allows you to
generate a test result report based on the
test result file. The test description report and
the test result report are identical in structure. However, only the latter contains actual
results.
Reporter
Reporter
Test
Description
Report
Test Result
Report
TRG
The test result report serves to evaluate the
test run. Only if each individual verify statement evaluates to „passed“, the test as a
whole will do so, too. If an error occurs, the
report indicates its location and, if possible,
also its cause.
Besides generating the test result report, you
can also use the test result file to transfer the
results back to a requirements tool for conducting statistical analyses. With a configurable script included in TPM, you can extract
result information from the test result file and
store it to a separate file in comma-separated
values (CSV) format. Then you can import
the CSV file to a requirements tools such as
DOORS.
15
TPM Reporter
The TPM Reporter GUI is divided into five sections as depicted in the screenshot below.
1
2
3
1 Icon Bar Section
2 Report Selection Section
4
3 Result Overview Section
5
4 Report Configuration Section
5 Log and Control Section
The Icon Bar Section contains two icons to
choose between the two modes of report generation: test description report („Description
mode“) or test result report („Result mode“).
Additionally, it contains an icon to start the
report generation.
The Report Selection Section lets you select
particular test jobs from the available test projects. In „Description mode“ a test description
report can be generated. In „Result mode“,
however, you can select a particular test run
of a test job and generate the corresponding
test result report. When you select a test run,
the Result Overview Section provides a result
overview without the need to generate the
report beforehand.
16
In the Report Configuration Section you can
tailor the report content and layout according to your wishes. On the one hand, you can
select from numerous check boxes to decide
what type of information you want to include in
the report. On the other hand, you can define
XML style sheets (XSL) to customize every
aspect of the report page layout, including
logos, headers, and footers.
TPM comes with various default style sheets
suitable for different purposes, such as »debug
level« reporting, »management level« reporting,
etc.
During report generation, the generation progress is displayed in the Log and Control Section of the GUI. In this section, you can also
stop the ongoing report generation.
TPM Reporter and Test Scenario
Test Scenario Example
The following is an example of how to formulate a test requirement with TPM.
Power Switch Verification
Create Test Case
„„ Test Requirements
Now we define the test case corresponding to
the above test requirements:
>> (1)
Verify that the power_ok condition
becomes true within one second after
turning power on.
>> (2) Verify that the voltage is 10 V +/0.1 V and the current is > the 2.5 A after
power_ok for at least 10 seconds.
>> (3) Verify that 0.4 sec after switching
power off, the voltage is 0 V +/- 0.1 V
and the current is < 0.1 A.
We divide the whole requirement into two
test steps, one is power_on and the other is
power_off.
TestStep power_on:
„„ Start:
>> trigger
PowerOnFail on timeout==1.0sec
>> set power = 1
>> Trigger PowerReady on power_ok==1
„„ Ver: OnEvent PowerReady, for 10.0sec
>> Verify,
ALWAYS, Voltage == 10.0V,
Tolerance=0.1V
>> Verify, ALWAYS, Current > 2.5A
„„ OnEvent PowerOnFail:
>> Popup
Message “FAILED”
TestStep power_off:
„„ Start: Delay = 1.0,
>> Set
power = 0
„„ Ver: After 0.4sec for 10.0sec:
>> Verify,
ALWAYS, Voltage == 0.0, Tolerance = 0.1
>> Verify, ALWAYS, Current < 2.5
17
Within the Test Step power_on, there are three
Time Slots: Start, Ver, and Fail.
Within the Time Slot Start, the signal power
is set to 1 (which means power is turned on).
Furthermore, an event called PowerReady will
be triggered when the condition power_ok ==
1 becomes true.
18
The Time Slot Fail is only used to display an
error message in case of a timeout of Time
Slot Start, that means that the power_ok signal is not raised for 1.0 second after setting
the power to 1.
The power_off test step is similar.
The two Statements Set and Trigger should
be finished within one second; else, the event
PowerOnFail will be triggered.
Within the Time Slot Ver voltage and current
values are verified against the expected values
for 10 seconds.
The following screenshot shows the result of
the test definition of this example in the TPM
Editor.
After defining the test case, you can click the
Build icon in the menu bar to call the TPM
Compiler to build the executable C file from
the test case.
Executor to select the required test job. As
soon as you click the Start button, the test job
is executed automatically in the ADS2 environment.
When the TPM Compiler has successfully
compiled the test case, you can use the TPM
The test results will be saved automatically to
the specified path and according to the layout
defined before.
Summery
Summary
The following lists some of the highlights of TPM:
„„ All-in-one software tool suite covering the entire test lifecycle of electronic equipment
„„ High degree of automation in the entire test process
„„ Easy and intuitive test implementation based on input forms for standard test components
„„ Full traceability between test requirements, test case(s)/plan(s), and test results
„„ C-Code interface for complex test cases
„„ Support for libraries for code-reusability
„„ SQLite database with GUI for easy import and handling of the system ICD
„„ Test execution control with debugging capabilities
„„ Tailored test report generation
„„ Integrated version control (Clear Case, PVCS, and CVS)
„„ Support for Windows, Linux, and VxWorks platforms
19
19
techSAT GmbH
Technical Systems for Avionics and Test
Gruber Strasse 46c • 85586 Poing • Germany
Tel +49 (8121) 703-0
Fax +49 (8121) 703-177
ts-info@techsat.com
techSAT Hamburg GmbH
Hein-Saß-Weg 36 • 21129 Hamburg • Germany
Tel +49 (40) 333-979-45
Fax +49 (40) 333-979 54
hamburg@techsat.com
techSAT North America
Lake Washington Air Harbor • Hangar Three
3849 N.E. 98th Street • Seattle WA 98115 • USA
Tel +1 (206) 910-3908
wm@techsatnorthamerica.com
www.techsat.com
This document describes TechSAT's Test Process Management (TPM) tool suite.
The information in this document is subject to change without further notice. TechSAT assumes no responsibility
for any error that may appear in this document.
All product names are (registered) trademarks of the referenced companies.
Copyright © 2010 TechSAT GmbH / Rev-1002. All rights reserved.