Active Safety Solutions
Renesas Electronics America Inc.
© 2012 Renesas Electronics America Inc. All rights reserved.
Renesas Technology & Solution Portfolio
2
© 2012 Renesas Electronics America Inc. All rights reserved.
‘Enabling The Smart Society’
 Challenge:
“In the smart society, vehicles are incorporating more
intelligence to detect threats, and then warn the driver
and/or automatically respond (i.e. emergency braking). The
computing solutions behind these active safety systems
must balance high performance against requirements for
integration and low power consumption.”
 Solution:
“Renesas focuses on adding specialized hardware to active
safety solutions to add performance with lower power,
allowing for higher integration.”
3
© 2012 Renesas Electronics America Inc. All rights reserved.
Microcontroller and Microprocessor Line-up
2010
2013
*
Up to 30K DMIPS,
Performance
32-bit
 Cortex A7 or A15-based
 Automotive, 28nm
1200 DMIPS, Superscalar
 Automotive & Industrial, 65nm
 600µA/MHz, 1.5µA standby
500 DMIPS, Low Power
1800 DMIPS, Performance
 Automotive, 40nm
 500µA/MHz, 35µA deep standby
 Automotive & Industrial, 90nm
 600µA/MHz, 1.5µA standby
8/16-bit
25 DMIPS, Low Power
 Industrial & Automotive, 150nm
 190µA/MHz, 0.3µA standby
44 DMIPS, True Low Power
10 DMIPS, Capacitive Touch
 Industrial & Automotive, 130nm
 144µA/MHz, 0.2µA standby
 Industrial & Automotive, 130nm
Wide
Format LCDs
 350µA/MHz, 1µA standby
*tentative naming
4
© 2012 Renesas Electronics America Inc. All rights reserved.
Active Safety Solutions
by Renesas
MP Start
2009
2011
*
2013
Performance
SH7766
w/ Signal Processing HW
Dot-Matrix
SoC
System-on-Chip
External Memory
2015
System-on-Chip
ARM-based
2017
Performance
requirements continue
to grow at high end
IMAPCAR2
w/ Signal Processing HW
SH7766-L
w/ Signal Processing HW
MCU
Internal Memory
Microcontroller
RH850-based
SH7455/6
200MHz
DSP by FPU
As process technology
advances, more
performance can be
packaged with internal
memory
*tentative naming
5
© 2012 Renesas Electronics America Inc. All rights reserved.
Agenda
 Introduction to “ADAS”
 ADAS Systems Overview
 Renesas Solutions
 SH7766 / TopView
 Sensor Fusion
 Smart Sensor
 Ecosystem
 Ethernet AVB in ADAS
 Functional Safety
6
© 2012 Renesas Electronics America Inc. All rights reserved.
Introduction
7
© 2012 Renesas Electronics America Inc. All rights reserved.
ADAS Definition
 Key Term: ADAS
“Advanced Driver Assistance Systems”
 High level description of ADAS:
 Monitor the vehicle and its environment
 If a threat or other notable event is detected
– Alert the driver
– And/or take action
 Goal: Provide the vehicle and driver greater
ability to avoid accidents
8
© 2012 Renesas Electronics America Inc. All rights reserved.
ADAS Application Examples
 Object detection and collision avoidance
 Parking assistance
 Back-up camera
 Surround-view
 Self-parking






9
Traffic Sign Recognition
Lane departure warning / lane keeping
Night Vision
Augmented reality
Headlight control (Automatic dimming of high beams)
Driver monitoring (distracted/drowsy driver)
© 2012 Renesas Electronics America Inc. All rights reserved.
ADAS Application Architecture
Today
Flash
Sensor Data
Complex PCB
for high-speed
RAM access
DDR RAM
DSP
Host
Control
Car
Network
 Major steps
 Collect sensor data on DSP (radar or camera)
 Analyze data (object detection)
 Communicate to other vehicle systems to take appropriate
actions, as needed
– Steering
– Braking
– Audio system (to warn driver)
– etc.
10
© 2012 Renesas Electronics America Inc. All rights reserved.
ADAS in the Vehicle
High-End Vehicle Platforms
Multiple Cameras
Multiple Radars
Low-End Vehicle Platforms
Some ADAS content still
required to meet regulations
and insurance requirements
11
© 2012 Renesas Electronics America Inc. All rights reserved.
High-Level ADAS Trends
 Strong growth is anticipated for ADAS applications
 ADAS expertise is maturing across the Automotive industry
 Technology costs are decreasing
 Government/Insurance regulations increasing
 …but adding sensors, networking, and processing capability
adds weight and power consumption.
 Reduces range of Electric Vehicles
 Reduces mpg when government regulation pushes manufacturers
to higher standards
 Renesas goal: Offer solutions that enable higher integration
and lower power
12
© 2012 Renesas Electronics America Inc. All rights reserved.
ADAS Application Growth
30.00
2011
2015
2019
25.00
20.00
15.00
10.00
5.00
0.00
Sensor
Fusion
13
Radar
Front
camera
© 2012 Renesas Electronics America Inc. All rights reserved.
Rear
camera
Ethernet Surround
camera
view
Driver
monitor
Autonomous Driving as Long-term Target
 Manufacturers and researchers are working on
autonomous vehicle technology
 Most of the building blocks are emerging for taking
R&D to production
 Where to go: Input by driver
 How to get there: From the Navigation Unit
 How to negotiate traffic, interchanges and road
conditions: From ADAS
– Understand position and obstacles through radar &
cameras
– Take action through steering, brakes, and throttle
14
© 2012 Renesas Electronics America Inc. All rights reserved.
Renesas Solutions for ADAS
15
© 2012 Renesas Electronics America Inc. All rights reserved.
Integration Target
Today
Target
Flash
Sensor Data
DSP
Complex PCB
for high-speed
RAM access
RAM
Host
Control
Integrate DSP,
MCU & Memory
RAM
Simple PCB
Lower cost
Car
Network
Sensor Data
16
Flash
© 2012 Renesas Electronics America Inc. All rights reserved.
DSP
Host
Control
Car
Network
Computational Requirements
for Image Recognition
MIPS
10,000
Pedestrian
Pedestrian Detection （Control）
Camera Only
（Warning）
1,000
Eye Focus
Detection
Driver Monitoring
Sign Recognition
100
Lane
Recognition
Detect
Rain
Fog
etc.
Vehicle
Detection
（Control）
Vehicle
Detection
（Comfort） Multi Sensors
Multi Sensors
10
 Requirements go up as camera technology advances in
resolution and frame rates
Source: Hitachi Research Lab.
17
© 2012 Renesas Electronics America Inc. All rights reserved.
Power Strategy: Specialized Hardware
 Power constraints impact how much performance can fit into
a package
 Strategy: Implement dedicated hardware for signal
processing
CPU 1
CPU 1
CPU 0
CPU 0
General
Purpose
CPU
18
Power
Performance
 Add more performance for less power versus faster CPU
Dedicated
Image
Processor
© 2012 Renesas Electronics America Inc. All rights reserved.
General
Purpose
CPU
Dedicated
Image
Processor
*
ADAS Solution Evolution
MP Start
2009
2011
2013
Performance
SH7766
w/ Signal Processing HW
Dot-Matrix
SoC
System-on-Chip
External Memory
2015
R-Car/V
ARM CPU’s
w/ Signal Processing HW
2017
Performance
requirements continue
to grow at high end
IMAPCAR2
w/ Signal Processing HW
SH7766-L
w/ Signal Processing HW
RH850/V
RH850 CPU’s
w/ Signal Processing HW
MCU
Internal Memory
SH7455/6
200MHz
DSP by FPU
Power limits
performance with
integrated memory.
New processes improve
capability.
*tentative naming
19
© 2012 Renesas Electronics America Inc. All rights reserved.
*
Building on ADAS Legacy
 The best technologies from pre-merger products are
combined, with added support for new technology
2013
Performance
2011
SH7766
Key Technologies
w/ Signal Processing HW
Signal Processing HW
Image Distortion Correction HW
Functional Safety
Etc.
R-Car/V
ARM CPU’s
w/ Signal Processing HW
IMAPCAR2
w/ Signal Processing HW
SH7766-L
w/ Signal Processing HW
Key Technologies
Signal Processing HW
Functional Safety
High-performance Flash
Etc.
RH850/V
RH850 CPU’s
w/ Signal Processing HW
SH7455/6
200MHz
DSP by FPU
*tentative naming
20
© 2012 Renesas Electronics America Inc. All rights reserved.
SH7766 Solution Example
21
© 2012 Renesas Electronics America Inc. All rights reserved.
SH7766 – SoC for Top-View Park Assist
System
Package FCBGA 21x21 0.8bp
DMA
CPU
Interfaces
SH4A
2 x HSCIF
533MHz
<1.0V + 1.8V + 3.3V
Ta =-40°C to +85°C
2 x CAN
4 x SPI
FPU
Timers
3x Timer Unit
2 x HSPI
L1- I$ 32k / L1-D$ 32kB
2x IIC
1x Watchdog Timer
4x PWM Unit
Vision/Graphics IP subsystem
Video interface
4x NTSC IN
Memory interface
SRAM I/F
2D Gfx engine
16-bit @ 44MHz
Image Recognition
2x Digital IN
4x IMR-LSX / 1x IMR
Image Correction Distortion
1x Video OUT
RGB666
22
© 2012 Renesas Electronics America Inc. All rights reserved.
IMP-X2
NOR Flash I/F tbc
4-bit @ 66MHz
DDR3 I/F
712 / 16-bit
Approach for Top-view System
1st step
2nd step
3rd step
• Make top view picture in real time
• Parking lane, Pedestrian
recognition
• Improve picture quality
• Highlight concerns
Front camera
Right camera
!
Left camera
Rear camera
4x IMR-LSX can do
perspective correction in
real-time
23
Parking lane, pedestrian
recognition etc, works at real
time with combination with
CPU and IMP
© 2012 Renesas Electronics America Inc. All rights reserved.
Improve picture by
DRC (Dynamic Range Control)
and draw graphics/icons with
the 2D Graphic Engine
IMR: For Camera Perspective Correction
IMR: for Image data already in RAM
IMR-LSX: for Image data on Vin using line memory
SH-4A
VIN
533MHz
（6ch :
4 Analog
2 digital）
IMP-X2
IMR
（1ch）
Translated
Image
IMR-LSX
IMR-LSX
IMR-LSX
（4ch）
DU
DDR3
IMR-LSX
IMR-LSX
Peripheral I/F and
modules
IMP-X2
IMR
 Key Design Points:
 IMR-LSX is in-line with video input, saving memory bandwidth
 4-channel distortion correction is a ~10K MIPS operation. IMRLSX is substantially more efficient than a general-purpose CPU
24
© 2012 Renesas Electronics America Inc. All rights reserved.
IMP: For Image Recognition Tasks
Computation volume of Normal image recognition applications
HIGH LEVEL
PROCESSING
10～30%
Pattern Recognition
Tasks
LOW LEVEL
PROCESSING
70～90%
Fundamental
Image Processing:
Filtering on a whole image
Binarization, Histogram, etc
General
Purpose
Processor
Dedicated
Processor (IMP)
General Purpose
Processor
25
© 2012 Renesas Electronics America Inc. All rights reserved.
General Purpose Processor
+
Dedicated Image Processor
from 10 to 20 times
Computation power
compared with CPU
DRC: Dynamic Range Control
2.
1. Dynamic range control can recover highlights and boost dark areas of images
Dynamic Range Control is implemented in hardware with support for multiple cameras
(No CPU needed!!)
(b)
(a)
Before
Luminance
distribution
Before
After
Before
After
level
(a) Dark area control
26
© 2012 Renesas Electronics America Inc. All rights reserved.
(b) Bright area control
After
Next-gen Top-View: Moving to ARM
 Similar concepts as SH7766
 Focus on Digital Cameras, higher resolutions
 Support Ethernet AVB Camera Networks or Direct Input
 Use dedicated HW
– IMR-LSX2  next-gen
– H.264 Decode
1Gb
EtherAVB
GMII
2D Graphics
Gbit
Ether AVB
H.264/MJPEG
decoder
OR
1280x1024
Camera Input
x5
VIN
0-4
27
© 2012 Renesas Electronics America Inc. All rights reserved.
IMR-LSX2
0-4
Main Bus 260MHz x 256bit x 2
CPUCPU
CA15
TBD CA15
MHz x2
1.5GHz
– IMP  next-gen
IMP
*
Next-gen Top View Example
 4 to 5 Camera Support
 Megapixel cameras
 More processing throughput needed
 improved IMP + faster CPU
 Compress images as camera
 H.264  Decode needed at ECU
 Ethernet AVB to transport
 Gbps Ethernet to receive multiple
camera streams
Top
View
ECU
*tentative naming
28
© 2012 Renesas Electronics America Inc. All rights reserved.
Sensor Fusion & Smart Sensors
29
© 2012 Renesas Electronics America Inc. All rights reserved.
Sensor Fusion
 Collect data from multiple sensors for a unified view of
vehicle environment
 Make decision based on the “big picture”
Fusion ECU
Compile
Sensor
Data
30
© 2012 Renesas Electronics America Inc. All rights reserved.
Analysis
&
Action
Sensor Fusion
 For a vehicle platform with multiple sensors:
 Use a distributed architecture
– Process raw data in separate ECU’s
Object Lists
Front Radar
ECU
Rear Radar
ECU
Side Radar
ECU
31
© 2012 Renesas Electronics America Inc. All rights reserved.
Location
Direction
Velocity
Etc.
Steering
Fusion ECU
Action
Warning
CAN/Flexray
Front Cam
ECU
Ethernet/CAN/Flexray
Top-View
ECU
Braking
Throttle
HMI
*
Sensor Fusion Solutions
 Update rate and number of sensors can drive CPU
performance requirements for fusion
 Break up processing tasks for more sensors
 If output is to steering/braking/throttle, safety requirements
are increased
 Enable higher ASIL levels for decisions & communication
Compile
Analysis
Compile
Analysis
RH850/V
RH850 CPU’s
w/ Signal Processing HW
ASIL B
Compile
Analysis
Decision
Communicate
32
Decision
Communicate
R-Car/V
ARM CPU’s
w/ Signal Processing HW
ASIL B
Decision
Communicate
RH850/V
RH850/V
RH850/V
RH850 CPU’s
w/ Signal Processing HW
ASIL D
RH850 CPU’s
w/ Signal Processing HW
ASIL D
RH850 CPU’s
w/ Signal Processing HW
ASIL D
Low End
Mid End
Few Sensors
More Sensors
© 2012 Renesas Electronics America Inc. All rights reserved.
High End
Many Sensors
*tentative naming
Smart Sensor Solutions
 Sensor fusion may require processing of raw radar or
camera data “somewhere else”, outside of the fusion
controller
 Highly integrated solutions such as the RH850/V series make
it easier to package the processing with the sensor
33
© 2012 Renesas Electronics America Inc. All rights reserved.
Smart Sensor – Camera Example
3.3V
1.25V
Power supply
RH850/V
Video Input
Flash
34
Image
Processor
CPU
RH850
CPU
perf
Flexray
CAN
PHYs
Ethernet
Large SRAM
High performance
Specialized image processing engine and
MCU domains merged into single chip
Embedded memories
Local SRAM (2.5MB) and Flash (4MB)
integration
© 2012 Renesas Electronics America Inc. All rights reserved.
Ecosystem
35
© 2012 Renesas Electronics America Inc. All rights reserved.
Customer Deployment Needs
Solutions
• Lane Departure
• Object Detection,
• …etc.
Optimized Libraries
• OpenCV
• OpenVL
Custom Development
• Compilers, etc.
36
© 2012 Renesas Electronics America Inc. All rights reserved.
Renesas is working with
3rd parties that can be
solution providers or
contractors.
Libraries for common
functions, including
support for open API’s
Tools for HW Accelerators:
e.g. Smarter Compilers
Ecosystem Overview
Carmaker functionality
Infrastructure
Customer/Tier1 Application
System integration
Car
Dvpt. Tools
Hardware
Party
network
AUTOSAR
RTE
RTOS
Dvpt. Tools
Software
3rd
Vision application SW
MCAL
Middleware
ADTF – EB toolchain
ECU
Video IF
Open std.
OpenCV
Vision
SDK
OpenCL
OpenGL
OpenVG
3rd party
Demo
PoC
network
Customization
& porting
Operating Systems
IDE : Eclipse
Debugger
Simulator
Hardware
debugger
Hardware
Renesas
standard
Compiler
PRISM for
MCU tools Multi-core
linker
Application
board
Device
Intermediate
devices
Companion chips
PHY
bin
37
© 2012 Renesas Electronics America Inc. All rights reserved.
Power
Supply
Ethernet AVB in ADAS
38
© 2012 Renesas Electronics America Inc. All rights reserved.
Ethernet AVB in ADAS
 Ethernet is gaining traction in Automotive
 Number of ECU’s and amount of data being transferred
increasing
 Infrastructure already exists in IT space, making it easier to
adopt for Automotive versus developing new technology
 Ethernet AVB adds guaranteed service and determinism for
time-critical data
 The bandwidth is necessary to move sensor data and results in
ADAS applications
 The determinism is critical: making a steering or braking decision
based on out-of-date sensor data can be fatal!
39
© 2012 Renesas Electronics America Inc. All rights reserved.
Key Benefits for Ethernet / Ethernet AVB
 Fast
 100Mbps, 1Gbps, 10Gbps: in use. 40Gbps. 100Gbps: standardized
– Fast, not expensive
 Translate moving pictures at real time
 Configurable/Controllable
 Easy to add/update/remove equipments and sensors
 Controllable including add/update/remove equipment and sensors
 Configurable systems constructed with above equipment and sensors
 Interoperability with external network
 Can be connected with internet and cloud
 Deploy enhanced Security concept
 Ethernet driving down automotive cost of ownership
 System Integration
MLB
INIC
CPU
Ether
40
AV
Network
Debug
Diag
© 2012 Renesas Electronics America Inc. All rights reserved.
CPU
Ether
AVB
Debug
Diag
AV
Network
Functional Safety
41
© 2012 Renesas Electronics America Inc. All rights reserved.
Functional Safety
 As ADAS systems impact operation of critical vehicle
systems, more effort is needed to verify that data is correct,
decisions are acceptable, and commands are delivered
 The amount of driving hours across all vehicles means that
even relatively unlikely events can happen
 Functional Safety defines design practices and device
requirements to meet elevated reliability requirements
 See the Functional Safety Session at DevCon for more detail
42
© 2012 Renesas Electronics America Inc. All rights reserved.
Renesas Functional Safety
RH850/P1x
RH850/V1x
V850E2/Px4
ISO26262 Assessment
by TÜV Süd
"fit for purpose ASIL D"
V850E/PHO2
V850E/PHO3
V850E/PHC3
Safety Features
•
•
•
•
•
ECC on Flash
ECC/Parity on RAM
Clock Monitor
CRC hardware support
Core self test software
IEC61508 Assessment
by TUEV Süd
„fit for purpose SIL3“
Integr. Safety Concept
•
•
•
•
Redundant core in lockstep
HW BIST
Safety Guardian
ßIC measures
2008
43
ISO26262
Compliant
Development Flow &
Safety Concept
© 2012 Renesas Electronics America Inc. All rights reserved.
2011
Integr. Safety Concept
•
•
•
•
•
•
Redundant core in lockstep
Error Control Module
Multicore support
Guard concept
Dependent failure analysis
HW BIST (P-series only)
2015
Roadmap for Safety apps
LSDC: Lock step dual core
RH850/V1x
 Autonomous Drive
 Sensor Fusion
 Smart camera
 Radar
Up to 4 MB Flash
Up to 2.5MB SRAM
ASIL D/B
LSDC + Perf
(w/ SIMD, FPU)
DSP engine
1800DMIPS
+ SIMD
+ DSP engine
1350 DMIPS
 Integrated Safety Unit
ASIL D
 High-End Chassis Controller
 Stability Control
 EPS
 Airbag with integrated sensors
RH850/P1x
Up to 8 MB Flash
Up to 1MB SRAM
670 DMIPS
Up to 2x LSDC
(w/ FPU)
450 DMIPS
ASIL D
225 DMIPS
 ABS
 Airbag
44
© 2012 Renesas Electronics America Inc. All rights reserved.
RH850/R1x
ASIL B
512 KB, 80 MHz,
Single Core
225 DMIPS
Questions?
45
© 2012 Renesas Electronics America Inc. All rights reserved.
‘Enabling The Smart Society’ in Review…
 This slide closes the loop on the Smart Society Challenge…
 Challenge:
“In the smart society, vehicles are incorporating more
intelligence to detect threats, and then warn the driver
and/or automatically respond (i.e. emergency braking). The
computing solutions behind these active safety systems
must balance high performance against requirements for
integration and low power consumption.”
 “Renesas focuses on adding specialized hardware to active
safety solutions to add performance with lower power,
allowing for higher integration.”
 Do you agree that we accomplished the above statement?
46
© 2012 Renesas Electronics America Inc. All rights reserved.
Renesas Electronics America Inc.
© 2012 Renesas Electronics America Inc. All rights reserved.