Multimodal sensors &
digital interfaces
Credits
The original Multimodal Project
developed by and credit for:
Zhigang Zhu and Weihong Li
(Integration of Laser Vibrometry with
Infrared Video for Multimedia
Surveillance Display)
Outline
• Multimodal System Overview
• Multimedia Sensors
• Infrared camera
• The LDV sensor
• PTZ camera
•
•
•
•
Multimodal System Components
System Design Concept
Design Issues
Integration Issues
Multimodal System Overview
• The object of this system is to
provide a multimodal integration of
audio, visible, thermal for human
signature detection.
• The goal is to use the sensing
technologies for perimeter
surveillance.
– Sensors, alarm, response.
• Multimodal system interface
– The environment, the sensors, and the
events.
Multimedia Sensors – Infrared Camera
Infrared camera
– FLIR ThermoVision A40M IR camera
– Temp Range of -20º to 500ºC, accuracy (% of reading)
± 2ºC or ± 2%
– 320x240 Focal Plane Array
– 24º FOV Lens
– Firewire Output IEEE 1394
– Video output – RS170 EIA/NTSC or CCIR/PAL
composite video for monitoring on a TV screen
– ThermoVision System Developers Kit (C++)
– Each thermal image is built from 76,800 individual
picture elements that are sampled 60 times per second
by the camera's on-board electronics.
Multimedia Sensors – Infrared Camera
Samples
Figure 1. A person sitting in dark room can be clearly seen in
the IR image. The temperature at Sp1 on the face is 33.1ºC
Figure 2. Two IR images before and after a person standing at
about 200 feet. The reading of the temperature at Sp1 changes
from 11C to 27C.
Multimedia Sensors – Infrared Camera
Thermographic measurement techniques
• An infrared camera measures and images the emitted
infrared radiation from an object.
• The radiation measured by the camera does not only
depend on the temperature of the objects but is also
a function of the emissivity.
• Radiation also originates from the surroundings and
is reflected in the object
• Radiation from the object and reflected radiation will
also be influenced by the absorption of the
atmosphere
• Parameters need to take care:
–
–
–
–
The emissivity of the object
The reflected temperature
The distance between the object and the camera
The relative humidity
Multimedia Sensors – Infrared Camera
Emissivity:
-
How much radiation is emitted from the object
Object materials and surface treatments exhibit
emissivity ranging from approximately 0.1 – 0.95
Highly polished (mirror) surface < 0.1
Human skin exhibits an emissivity close to 1
Metal: low, increase with temperature
Non-metal: high, decrease with temperature
Multimedia Sensors – Infrared Camera
Reflected ambient temperature:
-
-
To compensate for the radiation reflected in the
object and the radiation emitted from the
atmosphere between the camera and the object.
If the emissivity is low, the distance very long and
the object temperature relatively close to that of the
ambient it will be important to set and compensated
for the ambient.
Distance
-
The distance between object and the front lens of
the camera.
Relative Humidity
-
Normally, default 50%
Multimedia Sensors – Infrared Camera
History of Infrared Technology
- Sir William Herschel (1738-1822)
- Discover of infrared spectrum
- Marsilio Landriani (1746-1815)
- As the blackened thermometer was moved slowly
along the colors of the spectrum, the temperature
readings showed a steady increase from the violet
end to the red end.
- Macedonio Melloni (1798-1854)
- Rock salt (NaCl) (to be made into lenses and
prisms) is remarkably transparent to the infrared.
- Sir John Herschel
- The first ‘heat-picture’ in 1840, thermograph
- Samuel P. Langley (1834-1906)
- Inventor of the bolometer (1880)
Multimedia Sensors – LDV sensor
Vibrometer types:
- Single Point Vibrometers:
- Measure the vibration of an object in the direction
of laser beam
- Differential Vibrometers: (dual beam)
- Allow vibration measurement between two points
vibrating relative to each other.
- Rotational Vibrometers:
- Measure angular vibrations on rotating structures.
- In-plane Vibrometers:
- measure continuous (DC) velocity and
superimposed variable (AC) components
perpendicular to the central axis of two converging
laser beams.
- 3D Vibrometers
Multimedia Sensors – LDV sensor
Laser Doppler Vibrometer (LDV)
-
Optical instruments for accurately measuring velocity
and displacement of vibrating structures completely
without contact.
-
Sensor head OFV-505
- HeNe laser,  = 633.8 nm.
- OFV-SLR lens (f=30mm) 1.8m – 200+m, auto focus
-
Controller OFV-5000 with a digital velocity decode card
VD-6
- RS232 interface for computer control
-
Telescope VIB-A-P05
- ±1º vertical tilt and ± 1.5º horizontal tilt
Multimedia Sensors – LDV sensor
Measurement Principle
S is the light source
f is frequency
P is the moving with velocity v and
reflects the light
O is the receiver (f + fD)
Resultant
frequency shift
For vibrometers: S=O ("backscatter") 1= - 2, therefore,
Multimedia Sensors – LDV sensor
LDV schema
- Velocity is directly obtained by demodulation:
 = 2fm
- Voice frequency f: 300Hz ~ 3000Hz
- LDV can detect vibration at a magnitude as
low as m = /2f = 1/(2*3.14*300) = 0.5µm
Multimedia Sensors – PTZ camera
Pan/tilt/zoom (PTZ) camera
- Human and other target detection at a large
distance
- Canon PTZ
-
26X optical zoom lens & 12X digital zoom
Pan: ±100º, Tilt: +90º/-30º
Built-in IR light (effective up to 9 feet)
BNC video output
RS-232 computer control interface
Multimedia Sensors – PTZ camera
PTZ Samples
Two images of a person at a distance of about 200 feet, captured by
changing the zoom factors of the PTZ camera.
Multimodal System Components
• Three components
– The IR/EO imaging video surveillance
component
• Human motion tracking, human face detection
• Thermal Camera for daytime and nighttime
• Visible camcorder (sony), Web cam (logitech)
– The LDV audio surveillance component
• Audio signal capture, voice recognition
– The human-computer interaction
component
• Cognitive understanding of the environment, the
sensors, and the events
System Design Concept
• The overall goal the project is to design a human
computer interface for human-centered multimodal
(MM) surveillance.
HCI
Sensor Registration and Modeling
HCI
VE
navigation
IR Imaging
LDV Pointing
LDV points
Visualization
AR:
VE
+video
Human/motion
detection
Audio signal
detection
MM:
video
+audio
Target
visualization
LDV point
select/track
Voice Extract
Analysis
voice
synthesis
Face ID
display
Face Detection
Recognition
Voice
Recognition
Voice ID
diaplay
Human ID and Event Understanding
Design Issues
Issues need to be considered
– how to use EO camera tracking human
motion;
– how to incorporate IR imaging with
existing EO captured image;
– how to use IR imaging to help the laser
Doppler vibrometer to select the
appropriate targets;
– how to select optimal viewpoint from
audio detection.
Integration Issues
- Target detection and localization via
IR/EO imaging
- Set up an IR/EO imaging system with an IR
camera and a PTZ camera for finding
vibration targets for LDV listening
- Registration between the IR/EO
imaging system and the LDV system.
- Two types of sensors need to be precisely
aligned so that we can point the laser beam
of the LDV to the target that the IR/EO
imaging system has detected
- Future research on automated
targeting and focusing.
References
- Main multimodal system technical report
- http://wwwcs.ccny.cuny.edu/~zhu/LDV/FinalReportsHTML/CCNYLDV-Tech-Report-html.htm
- Polytec Laser Vibrometer
- http://www.polytec.com/
- FLIR Systems Security ThermoVision
Cameras
- http://www.flir.com/
- Paper:
- Z Zhu, W Li, “Integrating LDV Audio and IR Video for
Remote Multimodal Surveillance”
- Others…
(END)