Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Toyota: James Kapinski, Jyotirmoy Deshmukh,
Xiaoqing Jin, Hisahiro Ito, Ken Butts
December 11, 2014
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Toyota MBD Group
• Our group focus
Toyota Technical Center
– Advanced research in V&V for
powertrain controller designs
• Our group background
– Cyber-physical systems
(hybrid systems)
– Formal verification methods
• Our perspective
– Focus is on techniques for
application-level real-time
controller development
Powertrain Control
Division
Model-Based
Development
Group
Verification & Validation
2
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Ever-Increasing Complexities of Powertrain Control System
Fuel economy
Emissions
Safety
Driveability
1988
•
•
1997
2002
2009
Need to meet ever-increasing standards -> more complex control code
Engine control code in modern engines can be measured in millions of lines of code!
3
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Features of Powertrain Control
Software Development
•
•
Safety critical
Single core
– But multicore is coming!
•
Hard real-time
– Time-triggered tasks
• E.g., P+I control, table lookups
– Event triggered tasks
• E.g., crank angle events
•
Not much connectivity
– Distribution of features across processors not as significant
•
•
Performance and functionality critically depends on environment
Exhaustive test is impossible
– Continuous variables over unbounded time ⇒ infinite test cases
4
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Verification Challenges
• Complex models
–
–
–
–
–
Large number of states/inputs
Nonlinearities and lots of switching behavior
Variable time delays (delay differential equations)
Look-up-tables
Can contain legacy code or other black-box components
• Inconvenient model formats
– Many formal tools require format that can be translated into a discrete-state
representation or a hybrid automaton
– Simulink semantics are closed
– Translating formats is time consuming and error prone
• Lack of formal requirements
– More on this later…
5
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Value of Simulation
• Helps design validation
– Vital part of control law development
• Can uncover bugs
• Does not require verification domain
knowledge
– Engineers are not familiar with
© The MathWorks
• Temporal logic, bounded model checking, theorem
provers
• Simulations are cheap and usually fast
• Test-suites can be shared and built up
across models
6
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Using Simulation for
Test and Verification
• Let’s use simulation to guide verification and
testing approaches
• NOT a fundamentally new idea:
–
–
–
–
–
–
–
Concolic Testing: Sen et al, Kanade et al
Proofs from tests: (Gupta, Rupak, Rybalchenko)
Falsification analysis: (S-TaLiRo: Georgios, Sriram)
Sensitivity-based analysis (Breach: Donzé, Maler)
Coverage-guided simulation (Thao Dang et al)
Sciduction – combining induction and deduction (Seshia, Jha)
…. (please pardon the omissions)
7
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Spectrum of Analysis Techniques
More Scalable
Testing/Control Techniques
• Simulation
• Linear Analysis
(numerical)
• Test Vector
Generation for
Model Coverage
Less Scalable
• Linear Analysis
(symbolic)
• Concolic
Testing
• (Bounded) Model
Checking
• Stability
Proofs
• Reachability
Analysis
Less formal/exhaustive
8/60
Verification
• Theorem
Proving
More formal/exhaustive
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Spectrum of Analysis Techniques
More Scalable
Testing/Control Techniques
• Simulation
• Linear Analysis
(numerical)
• Test Vector
Generation for
Model Coverage
Less Scalable
• Linear Analysis
(symbolic)
• Trajectory
Splicing
• Coverage-based Testing
• Concolic
Testing
• Simulation-Guided
Lyapunov/Contraction
Analysis
• (Bounded) Model
Checking
• Stability
Proofs
• Reachability
Analysis
Less formal/exhaustive
9/60
Verification
• Theorem
Proving
More formal/exhaustive
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Spectrum of Analysis Techniques
More Scalable
Testing/Control Techniques
Simulation traces to learn contraction
metrics for dynamical systems
Using simulation
segments to efficiently
• Simulation
search for counterexamples
A. Zutshi, S. Sankaranarayanan, J. Deshmukh,
and J. Kapinski. Multiple Shooting, CEGAR-based
• Linear
Falsification for Hybrid
Systems.Analysis
Best Paper
in EMSOFT 2014.
(numerical)
• Test Vector
Generation for
Model Coverage
Less Scalable
T. Dreossi, T. Dang, A. Donze, J. Kapinski, X. Jin, J. Deshmukh. Efficient
Guiding Strategies for Testing of Temporal Properties of Hybrid Systems.
Submitted to the 2015 NASA Formal Methods Symposium.
10/60
A. Balkan, J. Deshmukh, J. Kapinski, P. Tabuada.
Simulation-guided Contraction Analysis. To
appear in the 2015 Indian Control Conference.
• Trajectory
Splicing
• Coverage-based Testing
LineartoAnalysis
Simulation-based •testing
maximize coverage of infinite
state-space
(symbolic)
Less formal/exhaustive
Verification
• Concolic
Testing
• Simulation-Guided
Lyapunov/Contraction
Analysis
• (Bounded)
Model
Using simulation traces
Checking
• Stability
to learn Lyapunov
functions andProofs
barrier
certificates
• Theorem
Kapinski,• J.
V. Deshmukh, S.
Reachability
Proving
Sankaranarayanan, and N.
Analysis
Aŕechiga. Simulation-guided
Lyapunov Analysis for Hybrid
Dynamical Systems. In HybridMore formal/exhaustive
Systems: Computation and
Control, 2014.
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
CPS Requirement Challenges
?
Implementation ⊨ Requirements
Implementation
Implementation
Requirements
Verification Tool
Classic Verification Assumption
11
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
CPS Requirement Challenges
Results from
Integration Tests
Informal
Engineering Insight
Implementation
Implementation
Simulationbased checks
Incomplete
Requirements
The Reality for CPS
12
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
CPS Requirement Challenges
• Requirements are evolving due to CPS-related issues
– Environment/software designs evolve concurrently
– Not possible to create a plant model that captures all
behaviors
– Subtle interactions between states/signals are not known
before integration test
• Definition of correct behaviors exist only in engineer’s
brain
– Formal requirements are hard for engineers to develop
– Existing requirements do not capture all of the desired
behaviors
• Model may capture appropriate/expected behavior but
requirements do not
13
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
CPS Requirement Challenges
• Requirements are evolving due to CPS-related issues
– Environment/software designs evolve concurrently
– Not possible to create a plant model that captures all
behaviors
– Subtle interactions between states/signals are not known
before integration test
• Definition of correct behaviors exist only in engineer’s
brain
– Formal requirements are hard for engineers to develop
– Existing requirements do not capture all of the desired
behaviors
Let’s look at
some ideas to
address this
• Model may capture appropriate/expected behavior but
requirements do not
14
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Requirement Mining†
• Sometimes requirements are
not in format needed to
perform formal verification
Simulink
Model
Seed Traces
– Would be useful to automatically
obtain formal specifications
• Our approach is simulationbased
Simulation
Traces
Obtain tightest
parameter for
given traces
Counter-example
Traces
Counterexample
Found
Falsify requirement
using a global
optimizer
Candidate
Requirement
No Counter-example
e.g., Overshoot=?,
Settling time=?
Template
Requirement
e.g., Overshoot=5%,
Settling time=0.2 sec.
Inferred
Requirement
† X. Jin, A. Donze, J. V. Deshmukh, and S. A. Seshia. Mining Requirements from Closed-Loop
Control Models. In Hybrid Systems: Computation and Control 2013.
15
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Learning Requirements
• Learning STL requirements from traces
– Optimization-guided learning
• Enumerate PSTL formulas up to a certain length (i.e., number
of nodes in parse tree of formula)
• Use Requirement Mining to mine parameter values from traces
• Select best feasible formula
Learning Tool
STL requirement
Traces from
Engineer
16
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Summary
• Many V&V challenges for powertrain systems
– Due to CPS nature of systems & high complexity
– We are encouraged by simulation-guided approaches
• Requirements engineering poses significant challenges
– Can’t assume we have a thorough set of formal requirements
– Let’s consider simulation-guided approaches
17
Ongoing Challenges in Applying V&V
Technologies to Automotive Engine Control
Jim Kapinski
Thank You!
• A benchmark powertrain control model described in:
– X. Jin, J. Deshmukh, J. Kapinski, K. Ueda, and K. Butts.
Powertrain Control Verification Benchmark. In Hybrid Systems:
Computation and Control, 2014.
• A version of the benchmark model can be found on the
Applied Verification for Continuous and Hybrid Systems
(ARCH) site:
– http://cps-vo.org/group/ARCH
– Paper: http://cps-vo.org/node/12108
– Models: http://cps-vo.org/node/12119
18