10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Facing Up
to the Future of Testing
AUTOSAR Conformance Tests
- Feedback on their development and utilization
Alain Feudjio-Vouffo, Fraunhofer Fokus (Germany), alain.vouffo@fokus.fraunhofer.de
Alexandra Faucheux, Kereval (France), alexandra.giret@kereval.com
Mathieu Morvan, Kereval (France), mathieu.morvan@kereval.com
Alain Ribault, Kereval (France), alain.ribault@kereval.com
Ashish Jadhav, Patni Computer Systems (India), ashish.jadhav@patni.com
Philippe Robin, Technoveo (France), philippe.robin@technoveo.com
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Content of the presentation





Introduction: AUTOSAR CTSpecs release status
AUTOSAR CTSpecs & abstraction from hardware platform
Metrics on AUTOSAR CTSpecs TTCN-3 code and test environment
Combination of conformance test and source code coverage
Looking into the future
 Theoretical view on combination and reuse of TTCN-3 suites to create new ones
 From ATS for module to ATS for stack and integrated software platform
 Example of deployment in the automotive industry
 Conclusion
2
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Introduction - AUTOSAR CTSpecs release status
 AUTOSAR conformance test specifications (CTSpecs) have been released on 15 April as part of
Release4.0R0002
 Conformance tests are available for 43 modules (including the RTE)
 A CTspec contains
 the specification of the conformance test suite
 the TTCN-3 scripts for dynamic test cases
 the configuration sets to be used when executing the Conformance Test Suite
 Number of test cases
 Number of dynamic tests (TTCN3)
 Number of static tests
3
6820
2754
4066
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
AUTOSAR CTSpecs & abstraction from hardware platform (1/2)
TTCN-3
Compiler
TTCN-3
Executable
 Adapters are generated
 ETS is hardware independent
TTCN-3 files
Adapters
Generator
Adapters
COM
Execution of the CTS
ETS
on PC
COM
COM
COM
COM
HW Specific
HW Specific
HW Specific
HW Specific
Module
Under Test
Module
Under Test
Module
Under Test
Module
Under Test
KEREVAL' Platform
4
Module
Under Test
Instrumented MUT
on board
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Metrics on AUTOSAR CTSpecs TTCN-3 code and test environment
CUSTOMER
CAPITALIZED or GENERATED
Software, Hardware and Test design tool
Configurations and
environment
MANUAL EFFORT
Reception
Integration
BSW preparation (configuration)
Test suite
BSW integration
Sources
execution
Source Code Inspection
Effort during Execution
Configuration Inspection
Operation Signature Tests
Dynamic Tests
5%
Test suite
execution
results
Test Management System
Reporting
Attestation
Manual Effort
Capitalized or
Generated
95%
5
CTSpec, TTCN-3 scripts and Configuration Set(s)
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
AUTOSAR CTSpecs & abstraction from hardware platform (2/2)
 Test Memory Management allows Test Suite to access memory of the board
TTCN-3 files
TTCN-3
Executable
module FeeJo
Allocate / Deallocate
/* ... */
Adapters
Read / Write / Compare
var PointerAddr_ v_AddrPointer;
pt_TestMemoryAccess.call(TestMemoryAccess_Allocate:{v_Size}, nowait);
alt
{
[]pt_TestMemoryAccess.getreply(TestMemoryAccess_Allocate:{v_Size}) ->
value v_AddrPointer
{
/* ... */
}
}
COM
COM
ETS
on PC
Test Memory
Management
HW Specific
/* ... */
pt_Fee.call(
Fee_Read:{v_BlockNumber, v_BlockOffset, v_AddrPointer, v_Size},
nowait
);
/* ... */
6
Module
Under Test
Instrumented MUT
on board
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Combination of conformance test and source code coverage
 A Code Coverage Analysis tool (RTRT) is used
during Conformance Test execution
 Provides
 An indicator of tests relevance according
to structure
 An effort estimation to go over functional
validation from conformance testing
 Information on dead code and unspecified
code
 Required in Safety Validation context
 ISO 26262 and DO178B
 Thanks to coverage measurement, conformance
testing can be part of validation process
7
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Reuse of the TTCN-3 suites
 Two types of reuse are envisaged for the module-level TTCN-3 ATS that are available
for the 43 modules.

1) Reuse of the suites for creating the ATS for the later revisions of the AUTOSAR standard
during the maintenance phase of the current release, and
 2) Reuse of the suites for creating the ATS for the future releases of the AUTOSAR standard
AUTOSAR Rel 4.0
Rev 2
Rev 3
AUTOSAR Rel 4.1
…
Rev 0
Rev 1
…
…
RE-USE of Artifacts in the ATS
 The ATS has been created having maintainability as a major design criterion. TTCN-3
provides the high level artifacts in the ATS to enable the reuse.
 Traceability between the specifications and the test cases and configuration parameters
has been created to simplify the upgrade process.
8
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Combination of TTCN-3 suites to create new ones – extensibility
 Different users of AUTOSAR might
be interested in extending the
existing ATS by having more tests
that are vendor specific and do not
exist in the current ATS for internal
test compliance
 The TTCN-3 suites are well
structured and provide an ease for
combining with other vendor
specific test cases to extend the
tests for internal test compliance
 The TTCN-3 suite is well structured
and provides ease of extensibility
for
test
cases,
configuration
parameters and combinations of
configuration parameters
9
Extensibility of TTCN3
AUTOSAR ATS
(for internal test compliance)
AUTOSAR
ATS
Vendor
Specific
Module
Tests
Additional
Tests
Required
by OEMs
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Higher level Test Suites
 The ATS is currently at module level
 Using TTCN-3 provides ease of extensibility to combine
test-cases and reuse code with additional test cases to
obtain a test suite for a set of inter-connected modules
 Some of the stubs and drivers would have to be replaced by
actual modules and suites used for testing of stacks like the
CAN stack, the Flexray stack, the communication stack, etc.
can thus be prepared
 In such type of higher level suites various AUTOSAR
modules can be simultaneously tested
 The test cases of the ATS can also be used to develop test
suite for testing of a full AUTOSAR stack
 The test cases of the ATS can also be used for developing
test suites for testing of AUTOSAR compliant software
components
10
Full
AUTOSAR STACK
ATS
for
Combination
of
Modules
Module
ATS
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Example of deployment in the automotive industry
 Feedback on executions
 Average of 10 non conformities detected per Module Under Test at first CT execution
 Typical non conformities detected on each module
Configuration data
set
• Parameter container
error
• Parameter range error
• Parameter type error
• Parameter missing
Source Code
inspection
• Enum definition error
• Parameter type error
• include missing
Static Compile
Time
Dynamic test suite
• signatures or prototypes
error
• parameters code
generation error (from
XML information)
• bad condition of function
deactivation
• argument return
• unexpected API invoked
• expected API not
invoked
 Such non conformities have not been detected during validation of the module
11
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Example of deployment in the automotive industry
Integrated with customers development plan
 Multi
run conformance tests
Functional
design
Functional
Tests
SW Architecture
design
Coding
Integration
Tests
Unit Tests
Confirmation test and
attestation delivery
Non-regression tests for any changes in:
- Configuration or Implementation,
- HW platform
Module Conformance checking
Conformance checking according to AUTOSAR description files (non-TTCN-3 tests)
12
AUTOSAR conformance tests - feedback on their development and utilization
10th TTCN-3 User Conference, 7-9 June 2011, Bled, Slovenia
Conclusions
 Hardware dependent ETS is obtained from ATS and is used for testing the MUT, this ensures that
the AUTOSAR CTSpecs are portable
 The use of AUTOSAR CTSpecs TTCN-3 code and test environment by CTAs need a very low
percentage of manual activities and up to 95% of activities can be tool-based
 AUTOSAR Conformance testing can be combined with other tests like source code coverage so
that independent test runs can be reduced resulting in overall reduction in test execution time
 AUTOSAR ATS has maintainability and extensibility enabled by use of TTCN3 for efficient reuse in
creating new ATS for future releases, ehancements and higher level testing
 AUTOSAR Conformance testing can be integrated in the overall AUTOSAR development process
so that testing can be performed in early stages of the development and can result in detecting
non-conformances even at design stage which would improve the development efficiency
13