rateEFFECT
Effectiveness Evaluation of Active Safety Systems
Jörn Marten Wille, Dr. Michael Zatloukal
Volkswagen Group Research
Group research
rateEFFECT: Effectiveness Evaluation of Active Safety
Systems
1
Intention and Motivation
2
Evaluation Approach - Introduction to rateEFFECT
3
Evaluation of Crash Avoiding Systems
4
Database – GIDAS-preCrashMatrix
5
Opportunities for US Pre-crash Data
Group research
Slide 2
rateEFFECT: Effectiveness Evaluation of Active Safety
Systems
1
Intention and Motivation
2
Evaluation Approach - Introduction to rateEFFECT
3
Evaluation of Crash Avoiding Systems
4
Database – GIDAS-preCrashMatrix
5
Opportunities for an US Pre-crash Data
Group research
Slide 3
Intention and Motivation
A
B
Which active safety systems should be developed to maximize
safety benefit in real traffic accidents?
What is the effectiveness of a specific active safety system in the
real world? How many casualties could be avoided by such a
system?
Group research
Slide 4
Status
quo
Forecast
with
System
Casualties / Fatalities
Rating Process
System configuration
Algorithm
Sensor
Driver
model
Injury risk function
rateEFFECT
3
Actuator
PC-Crash
PC-Crash
PC-Crash
4
100%
100%
200
70%
50%
60%
40%-50%
50%
30%-40%
70
60
50
40
30
20
150
Anzahl Fälle
20%-30%
40%
40%
MAIS2+
50%-60%
60%
0%
100
10%-20%
30%
0%-10%
20%
50
10%
Geschwindigkeitsänderung
dv in km/h
0%
10
0
0
5
10
15
20
25
30
35
40
45
55
60
65
70
75
80
Annäherungsgeschwindigkeit in km/h
2
Target population
5
Effectiveness evaluation
1830 Berechnete Unfälle
(Wirkfeld)
1
Beeinflusste
Unfälle
8%
739
1091
4%
addressable accidents
1
Initial accident world
2
Target population – case selection
3
FAS&IS system
4
Modified accidents (∆v, collision angle etc.)
5
Lowered injury severity
6
Translation into a rating
Group research
50
0
Real world accidents (Germany)
6
Optional rating
Slide 5
tat. rel. Häufigkeit
60%-70%
10%
Current level of
safety
70%-80%
70%
20%
GIDAS
PreCrash-Matrix
Konfidenzband
80%-90%
80%
80%
30%
Conceptual, model or real algorithms
250
90%-100%
90%
90%
Nicht
beeinflusste
Unfälle
Effectiveness vs. Target Population – Need for Simulation
Questions the target population alone cannot answer:
How does a specific system design influence its performance?
Is driver reaction relevant for this scenario?
What is the performance of a warning system compared to an AEB?
Group research
Slide 6
The Effectiveness Analysis is based on the In-the-LoopMethod
System Description
Corresponds to the behavior of
real vehicle components
System Modelling
Sensor 1
Classical simulator approach:
Interaction in every integration
step
Sensor 2
Algorithm 1
Sensor 3
Algorithm 2
Warning 1
Based on PC-Crash (vehicle
dynamics and scenery)
Modelling of arbitrary systems
possible (including continuous
feedback control systems)
Driver model 1
Actuator 1
Integration of arbitrary algorithms
and complex driver models
PC-Crash
Adaption of the database to current
safety level
Group research
Actuator 2
Slide 7
FAS&IS-system
Driver Interaction
rateEFFECT Analysis with Different Levels of Detail
1
n
2
Target
population
 Potentially
addressable
accidents
 Best-case-analysis
Target population
Simplified
system
Real system
 rateEFFECT
analysis of a
simplified system
 Maximum of the
achievable effect of
a modelled System
 rateEFFECT analysis
of a real approximated
system
 Approximated effect of
a real systems
Effectiveness
evaluation
Effectiveness
evaluation
1830 Berechnete Unfälle
(Wirkfeld)
1830 Berechnete Unfälle
(Wirkfeld)
Beeinflusste
Unfälle
Beeinflusste
Unfälle
8%
4%
Group research
8%
739
1091
Nicht
beeinflusste
Unfälle
739
4%
Slide 8
1091
Nicht
beeinflusste
Unfälle
rateEFFECT Analysis of a Fictitious Emergency Brake System
System description:
Detectable:
PreCrash system with distance sensors to
avoid or mitigate frontal impacts
Not detectable:
Velocity range:
0 – 200 km/h
Sensor properties:
System action:
Three step braking up to full emergency
brake in case an impact is about to happen
deceleration a
Range 80m
Opening angle :
45°
Group research
Slide 9
time t
Accidents
altered by system
Accidents not
altered by
system
Performance figures
Ø dV original accident data (1830):
15,64 m/s
Ø dV altered accidents (1091):
13,82 m/s
Ø dV reduction altered accidents (1091): 7,07 m/s
Group research
Casualties per year
Number of accidents simulated
(target population)
Injury risk (MAIS2+)
Effectiveness Evaluation via Reduction of Injury Risk (MAIS2+)
Vehicle - vehicle
0,72%
5.4%
Adjusted to meet current
safety level
System effectiveness
Difference between reference
and estimated scenario
-352
Without System
Without
system:
2638
Forecast
2286
Severely injured+
Slide 10
Target population
Adjusted to meet current
safety level
System
Difference between reference
and estimated scenario
Summary of the Evaluation: Altered Accidents
1 Basic Configuration: Emergency Brake System
2 BAS + Autonomous Brake
3 “…” + Warning + Driver‘s Reaction
4 “…” + always detect stationary vehicles
Group research
Slide 11
Summary of the Evaluation: System Effectiveness
Injury risk (MAIS2+)
0,34%
5.4%
5.06%
≙
6.3 % of
Target
Population
3 “…” + Warning + Driver‘s Reaction
Injury risk (MAIS2+)
2 BAS + Autonomous Brake
0,81%
5.4%
Group research
4.59%
≙
0,68%
5.4%
4.72%
≙
12.6 % of
Target
Population
4 “…” + always detect stationary vehicles
Injury risk (MAIS2+)
Injury risk (MAIS2+)
1 Basic Configuration: Emergency Brake System
15 % of
Target
Population
Slide 12
1,9%
5.4%
4.59%
≙
35.2 % of
Target
Population
Backup
Group research
Slide 13
rateEFFECT: Effectiveness Evaluation of Active Safety
Systems
1
Intention and Motivation
2
Evaluation Approach - Introduction to rateEFFECT
3
Evaluation of Crash Avoiding Systems
4
Database – GIDAS-preCrashMatrix
5
Opportunities for US Pre-crash Data
Group research
Slide 14
Integrating Real World Tests
System configuration
Algorithm
Sensor
Driver
model
Injury risk function
rateEFFECT
3
Actuator
PC-Crash
PC-Crash
PC-Crash
4
100%
100%
70%
50%
40%
70
60
50
40
30
20
Test results
3
FAS&IS system
4
Modified accidents (∆v, collision angle etc.)
5
Lowered injury severity
6
Translation into a rating
Group research
30%-40%
150
Anzahl Fälle
20%-30%
100
10%-20%
30%
0%-10%
20%
50
10%
Geschwindigkeitsänderung
dv in km/h
0%
10
0
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
Annäherungsgeschwindigkeit in km/h
5
Effectiveness evaluation
1830 Berechnete Unfälle
(Wirkfeld)
Beeinflusste
Unfälle
739
1091
4%
2
40%-50%
50%
0
8%
Translation into tests
60%
40%
0%
Real world test
6
Optional rating
Slide 15
tat. rel. Häufigkeit
MAIS2+
50%-60%
60%
2
1
200
60%-70%
10%
1
70%-80%
70%
20%
Real world accidents
Konfidenzband
80%-90%
80%
80%
30%
Conceptual, model or real algorithms
250
90%-100%
90%
90%
Nicht
beeinflusste
Unfälle
rateEFFECT Analysis of a Fictitious Emergency Brake System
System description:
Detectable:
PreCrash system with distance sensor to
avoid or mitigate frontal impacts
Not detectable:
Velocity range: 0 – 200 km/h
Detects stationary vehicles at v < 30 km/h
Sensor properties:
Three step braking up to full emergency brake in
case an impact is about to happen
BAS to full brake in case driver is braking
Opening angle: 60°
y
System action:
Deceleration [m/s²]
Range: 100m
x
dX
Time [s]
Partial Brake 1
Partial Brake 2
- 1g
Full Brake
Group research
Slide 16
Five Steps for the Target Population
1
Target population study (U.S.)
2
Accident level
3
Target population on person level
4
Comparison with target population in Germany
5
Target population in Germany (used for effectiveness calculation)
Group research
Slide 17
Target Population Forward Collision Warning
25%
20%
23%
15%
10%
11%
5%
2%
0%
All crashes
Group research
Nonfatal injury
crashes
Slide 18
Fatal crashes
Source: Farmer 2008, Crash
Avoidance Potential of Five
Vehicle Technologies
Target Population (Persons) Front to Rear / Car to Car in U.S.
90%
80%
70%
60%
50%
Not injured+
40%
Minor injuries+
Severely injured+
30%
20%
10%
0%
First harmful event:
collision with vehicle
Group research
+ Two vehicles
involved, no twowheelers
+ Front to rear collision
Slide 19
+ No avoidance
maneuver
Target Population (Persons) Front to Rear / Car to Car
30%
25%
24%
20%
U.S.
Not injured+
Minor injuries+
15%
Severely injured+
Fatally injured
10%
9%
Germany
7%
5%
2%
0%
Not injured+
Group research
Severely injured+
5.4%
Minor injuries+
Severely injured+
Slide 20
Fatally injured
Target Population Front to Rear / Car to Car Germany
Target population severely injured (MAIS2+) persons for Germany
80%
70%
70,8%
60%
50%
49,2%
40%
39,6%
1830 simulated accidents
30%
City-AEB:
Stationary targets
only below 30 km/h
ego velocity
20%
19,0%
17,6%
10%
5,4%
0%
Vehicle involved + vehicle frontal
deformation
Group research
+ no skidding,
visibility
obstruction,
Kickdown
+ opponent
vehicle
Slide 21
+ opponent
velocity
+ front - rear
collision
1,9%
+ velocity
restriction for
stationary targets
Four Different Variations of the Safety System
1 Basic Configuration: Brake Assist (BAS)
2 “…” + Autonomous Brake
BAS: If driver is braking above 6 m/s² in the
original accident, this is increased to
maximum braking
Three braking levels based on criticality
Driver braking in original accident:
increased to maximum braking
System can detect stationary vehicles up to
ego velocity of 30 km/h (city AEB)
3 “…” + Warning + Driver‘s Reaction
Three braking levels based on criticality
Autonomous braking triggers warning
function, driver reacts with 1s reaction time
4 “…” + always detect stationary vehicles
System can detect stationary vehicles
regardless of ego vehicle velocity
Driver reaction is always braking. This is
increased to maximum braking by the
system
Group research
Slide 22
Effectiveness Evaluation via Reduction of Injury Risk (MAIS2+)
Performance figures
Ø dV original accident data (1830):
15,64 m/s
Ø dV altered accidents (961):
13,82 m/s
Ø dV reduction altered accidents (961): 7,07 m/s
Casualties per year
Injury risk (MAIS2+)
1830 accidents simulated
(target population)
Vehicle - vehicle
0,81%
5.4%
Target population
Adjusted to meet current
safety level
System effectiveness
Difference between reference
and estimated scenario
-396
Without System
Without
system:
2638
Forecast
2242
Severely injured+
Adjusted to meet current
safety level
System
Difference between reference
and estimated scenario
Configuration 3: Brake Assist + Autonomous Brake + Warning + Driver‘s Reaction
Group research
Slide 23
Summary of the Evaluation: Altered Accidents
1 Basic Configuration: Emergency Brake System
2 BAS + Autonomous Brake
3 “…” + Warning + Driver‘s Reaction
4 “…” + always detect stationary vehicles
Group research
Slide 24
Summary of the Evaluation: System Effectiveness
Injury risk (MAIS2+)
0,34%
5.4%
5.06%
≙
6.3 % of
Target
Population
3 “…” + Warning + Driver‘s Reaction
Injury risk (MAIS2+)
2 BAS + Autonomous Brake
0,81%
5.4%
Group research
4.59%
≙
0,68%
5.4%
4.72%
≙
12.6 % of
Target
Population
4 “…” + always detect stationary vehicles
Injury risk (MAIS2+)
Injury risk (MAIS2+)
1 Basic Configuration: Emergency Brake System
15 % of
Target
Population
Slide 25
1,9%
5.4%
4.59%
≙
35.2 % of
Target
Population
Summary of the Evaluation: Injuries and Fatalities
2 BAS + Autonomous Brake
1 Brake Assist (BAS)
-166
-166
-166
Without System
2638
Adjusted to meet current
safety level
2638
2472
2472
2306
System effectiveness
Difference between reference
and estimated scenario
Casualties
3 “…” + Warning + Driver‘s Reaction
-166
2638
2472
1
2
4 “…” + always detect stationary vehicles
-166
-166
-166
-64
2638
2306
2242
2472
2306
-64
-532
2242
1710
1
Group research
2
3
1
Slide 26
2
3
4
rateEFFECT Analysis of the Potential of a Fictitious Pedestrian
Protection System
System description:
Detectable:
PreCrash system with distance sensor to
avoid or mitigate frontal impacts with
pedestrians
Not detectable:
Velocity range: 0 – 60 km/h
Reacts only when pedestrian enters the
road
System action:
Sensor properties:
Driver braking in original accident is increased to
max braking
y
Deceleration [m/s²]
Opening angle: 40°
Emergency braking with 0.8g
Range: 40m
x
dX
Group research
Slide 27
Time [s]
- 1g
0.8g brake
Severely injured (MAIS2+) persons
Target population vehicle-pedestrian accidents Germany
Accidents involving
pedestrians
Group research
Passenger car –
pedestrian accidents
Slide 28
Accidents
altered by
system
330
523
Accidents
not altered
by system
Performance figures
Ø Vc original accident data:
30.35 km/h
Ø Vc altered accidents (961):
24.13 km/h
Casualties per year
853 accidents (target
population)
Injury risk (MAIS2+)
Comparison with Pedestrian Protection
Vehicle - vehicle
2.1%
12.6%
Slide 29
Adjusted to meet current
safety level
System effectiveness
Difference between reference
and estimated scenario
-1026
Without System
Without
system:
6154
Forecast
5128
Severely injured+
Group research
Target population
Adjusted to meet current
safety level
System
Difference between reference
and estimated scenario
Evaluation Conclusions
System effectiveness strongly depends on system design
Based on the simulation vehicle-to-vehicle forward collision warning / mitigating systems
would benefit if stationary targets could be detected at all ego vehicle speeds (sensor
limiations, detection of gullys etc.)
Vechicle-to-vehicle forward collision systems address scenarios where passive safety works
to the fullest
Technical specifications which are not regarded here may lower system effectiveness
Property damage can currently not be assessed
Question: What is the effectiveness in the U.S.?
Group research
Slide 30
rateEFFECT: Effectiveness Evaluation of Active Safety
Systems
1
Intention and Motivation
2
Evaluation Approach - Introduction to rateEFFECT
3
Evaluation of Crash Avoiding Systems
4
Database – GIDAS-preCrashMatrix
5
Opportunities for US Pre-crash Data
Group research
Slide 31
Projecting VW-GIDAS onto Germany
Statistisches Bundesamt Deutschland
The following parameters are taken into account:
(2009: 401.823 injured or killed occupants)
Injury level
Accident site
Mode of traffic
participation
DD / H → new / old
states
VW-GIDAS *
54.332 individuals
BAB …
∙ 6,4
…
fatally inj. occupants
sev. inj. occupants
…
rural area
…
urban area
…
∙ 4,7
fatally inj. pedestrians
severely inj. pedestrians
slightly injured pedestrians
* VW-GIDAS, 06/2010
Group research
Slide 32
∙ 10,6
∙ 27,0
Three Different Databases are Available
Internal database
Self-generated database
with more than 70
variables
Around 4200 data sets
available
preCrash matrices VUFO
Standard of the GIDASpool
Single cases
Arbitrary accident cases
Reenacted scene
Generated by VUFO
GmbH
Single Cases of interest
Currently 2D Design, 3D
planned
Source: VUFO GmbH
Group research
Slide 33
Possibilities and limitations
Possibilities
+ Effectiveness with respect to severely and
fatally injured persons
Current Limitations
- No traffic flow (e.g. lane change accidents)
- No property damage
+ Minor injuries evaluation possible as well,
but definition is a little vague
- Germany only (Czech Republic and China(?)
in preparation)
+ Vehicle – vehicle
vehicle – pedestrian
vehicle – two-wheeler
Group research
Slide 34
Phases of a Road Accident
A Crash can be Divided into Five Phases
„Traditional“ safety research focuses on the in-crash phase and its effects on the vehicle as
well as occupants
Active safety research focuses on the pre-crash phase
The pre-crash phase can be divided into sequences
Pre-crash phase
Phase 1:
Normal driving
Phase 2:
Danger phase
Phase 3:
Crash unavoidable
Phase 4:
In crash
Impact
Group research
Slide 35
Phase 5:
Post-crash (2nd crash?)
time
Requirements for the Simulation – Vehicle & Road Parameters
Vehicle Parameters:
Length
Width
Breite
Width
Wheelbase
Track width
Maximum steering angle
…
Länge
Length
Road Parameters:
Kurvenradius
Curve radius
Curve Radius
Slope
Surface (asphalt, paved, …)
Weather conditions (wet, icy, …)
…
Group research
Kurvenwinkel
Curve
Angle
Slide 36
Requirements for the Simulation – Environment & Psychology
Environment parameters like
Light conditions, fog etc.
Traffic lights, road signs
Visibility obstructions
Vicinity (tall buildings?)
Traffic situation, flow, density (hard to
obtain)
The actual field of view
t + 0,8
Drivers reaction
t + 0,6
t + 0,4
How did the driver react in a certain
accident?
What is an average reaction time?
t
Age, physical conditions, …
t + 0,2
t + 0,2
t
t + 0,4
t + 0,6
t + 0,8
Group research
Slide 37
Requirements for the Simulation – Numerical Accident Data
Numerical information about pre-crash
phase through reliable accident
reconstruction
Initial velocity v0
Velocity after sequence
b(t) – braking as a function
of time
Minimum: Mean braking
deceleration / distance travelled
Information about steering
Skidding parameters
Roadway departure with angle
…
Group research
Slide 38
Example Crash Examined by VW Accident Research
On-scene investigation
and measurements
Rectified, scaled image
of crash site
Group research
Slide 39
Minimum Dataset: Minimal Environmental Data
visibility
obstruction
pedestrian
vehicle
Group research
trajectory of vehicle
Slide 40
trajectory of
pedestrian
Current Data Set Including More Detailed Environmental Data
curb
road markings
trajectories of vehicles
visibility
obstruction
Group research
Slide 41
rateEFFECT: Effectiveness Evaluation of Active Safety
Systems
1
Intention and Motivation
2
Evaluation Approach - Introduction to rateEFFECT
3
Evaluation of Crash Avoiding Systems
4
Database – GIDAS-preCrashMatrix
5
Opportunities for US Pre-crash Data
Group research
Slide 42
Our Perspective on US Data
CIREN
Detailed medical
data
NASS-GES
Statistically
representative
NASS-CDS
Many cases per
year
~300 cases per
year
Many cases per
year
Detailed passive
safety information
“tip of the iceberg”,
MAIS3+
All severities
including property
damage
Pre-crash phase
less detailed
Not statistically
representative
Group research
Less detailed
information
Selection criterion
(tow-away)
Slide 43
GIDAS
2000 cases per year
Statistically
representative (for
severely injured
persons)
Detailed pre-crash
phase
No property
damage
Conclusions
A lot of information is required to simulate existing accidents in order to estimate ADAS
effects
This particularly includes numerical values for the pre-crash and in-crash phase
GIDAS provides a required minimum number of these parameters
How to obtain these data from U.S. databases?
Group research
Slide 44
Variation of the safety system
Original case without system + BAS
+ autonomous brake
+ warning + driver reaction
+ always detect stationary vehicles
Group research
Ø dV
original
accidents
[km/h]
Ø dV
altered
accidents
[km/h]
Ø dV
reduction
altered
accidents
[km/h]
Altered
Accidents
System
effectiveness
BAS
15,64
17,99
6,14
580
0.34%
BAS + Autonomous Brake
15,64
13,82
7,07
961
0.68%
BAS + Autonomous Brake +
Warning + Driver‘s Reaction
15,64
13,82
8,95
961
0.81%
„…“ + always detect
stationary vehicles
15,64
16,70
12,98
1636
1.9%
Slide 45