L ESSON I: I NTRODUCTION
David L. Hall
L ESSON O BJECTIVES



Introduce this course and instructor
Provide an understanding of the course
logistics, requirements, grading, assignments,
and ground rules
Introduce the topic of data and information
fusion
C OURSE O BJECTIVES

To provide an introduction to the field of data
and information fusion







Models of multisensor data fusion
The JDL Data Fusion Process Model
Techniques for data fusion ranging from estimation to pattern
recognition and automated reasoning
Guide you through a team exercise involving design of a data
fusion system to address a selected application
Present a balanced view of the advantages and limitations of
fusion
Understand the role of the human in the loop analyst/decision
maker
Provide a basis for further study and specialization
R EVEALING MY PEDAGOGICAL HAND





Brief presentations
In-class (on-line) exercises
Humor and stories
On-line discussion &
presentations
Planned time for group meetings
& work
It is important that you:
1) Participate every week
2) Focus on and complete
on-line materials, lecture material,
readings, assignments.
O N -L INE M ATERIALS
On-line Lessons
• Site navigation on LHS
• Each lesson summarizes
• Video pre-view
• Introduction
• Lesson objectives
• Commentary &
discussion
• Activities &
assignments
• Links to readings (via
electronic library reserve)
• Electronic copy of Lecture
materials
T EXT AND R EADINGS
Assigned Text
• D. Hall and S. A. McMullen, Mathematical
Techniques in Multisensor Data Fusion,
Artech House, 2004
Selected Readings
• D. Hall and J. Llinas, editors, Handbook of
Multisensor Data Fusion, CRC Press,
2001
• Excerpts from selected textbooks
• Selected technical papers
• One science fiction story
C OURSE L ESSONS
1.
2.
3.
4.
5.
6.
7.
8.
9.
Introduction
JDL Model
Project initiation
Sensor processing
Level 0 processing
Level 1 – Correlation
Level 1 – Estimation
Level 1 – Target ID
Systems Engineering
10.
11.
12.
13.
14.
15.
16.
17.
10. Project design
Level 2 (situation refinement)
Level 3 (Consequence refinement)
Level 4 – Process refinement
Level 5 – Cognitive refinement
Project detailed design
Data Fusion state of the art
Final Presentation
A SSIGNMENTS & W EIGHTS
Individual Assignments and weights (70 %)
• Ten (10) low stakes quizzes (20 %)
• Six (6) on-line discussion participation (15 %)
• Eight (8) Individual writing assignments (24 %)
• Peer evaluation (11 %)
Group project and weights (30 %)
• Final technical report (20 %)
• Final presentation (10 %)
G ROUND R ULES





Attendance/participation & preparation
Plagiarism (cheating)
Academic integrity
Affirmative Action & Sexual Harassment
Americans with Disabilities Act
You have one week to question or dispute grades,
missed assignments, or missed classes:
Note – I dislike “wheedling” for extra credit
T HE ORIGIN OF MULTISENSOR DATA
FUSION
“I say fifty, maybe a
hundred horses . . .
What do you say,
Red Eagle?”
B IOLOGICAL ORIGINS OF SENSOR
FUSION
Smell
Sight
Touch
Sound
Chemical detection
Sonar
A UGMENTATION OF SINGLE SENSES
A long history of single sense augmentation has
included eyeglasses, hearing aids, telescopes,
microscopes (and more recently electronic noses,
chemical detectors and many others
A UGMENTATION OF COGNITION


Similar to the augmentation
of our senses, a long history
of effort has sought to
augment out cognition
Data fusion seeks to
support the augmentation
& automation of the multisensing, cognition process
for improved awareness
and understanding of the
world
JDL D ATA F USION M ODEL
• Course organized using
the JDL Model “Levels”
• Hall and McMullen text
organized around JDL
model
• Lessons in on-line site
focus on JDL levels
• Lessons systematically
“walk through” the levels
• Project focus on
designing a data fusion
system for selected
application using the JDL
framework
D ATA F USION F UNCTIONAL M ODEL
The JDL model (1987-91) and the draft revised model (1997)
• Level 0 — Sub-Object Data Association and Estimation: pixel/signal
level data association and characterization
• Level 1 — Object Refinement: observation-to-track association,
continuous state estimation (e.g. kinematics) and discrete state
estimation (e.g. target type and ID) and prediction
• Level 2 — Situation Refinement: object clustering and relational
analysis, to include force structure and cross force relations,
communications, physical context, etc.
• Level 3 — Significance Estimation [Threat Refinement]: threat intent
estimation, [event prediction], consequence prediction, susceptibility
and vulnerability assessment
• Level 4 — Process Refinement: adaptive search and processing (an
element of resource management)
Adapted from A. Steinberg
J OINT D IRECTORS OF L ABORATORIES D ATA
F USION S UBPANEL :
DEFINITION OF DATA FUSION:
A continuous process dealing with the
association, correlation, and combination of
data and information from multiple sources to
achieve refined entity position and identity
estimates, and complete and timely
assessments of resulting situations and
threats, and their significance.
D EFINITIONS . . .
• Sensor Fusion = Data Fusion from Multiple Sensors
(same or different sensor types)
• Data Fusion = Combining information to estimate
or predict the state of some aspect of the world
• Data Fusion Functions:
– Data Alignment
(etc.)
(spatio-temporal,
Traditional
data normalization,
Focus
evidence conditioning)
Platform
– Data Association
(hypothesize entities)
– State Estimation
Reports
& Prediction
Situation
Cross-Force
Relations
Force Structure
Unit
(etc.)
R EPRESENTATIVE D ATA F USION
A PPLICATIONS FOR D EFENSE S YSTEMS
SPECIFIC
APPLICATIONS
Ocean Surveillance
Air-to-Air and
Surface-to-Air
Defense
Battlefield
Intelligence,
Surveillance and
Target Acquisition
Strategic Warning
and Defense





INFERENCES
BY DF PROCESS
Detection, Tracking,
Identification of
Targets/Events
Detection, Tracking
Identification of
Aircraft
Detection and
Identification of
Potential Ground
Targets
Detection of
Indications of
Impending Strategic
Actions
Detection/Tracking
of Ballistic Missiles
and Warheads





PRIMARY
OBSERVABLE DATA
EM Signals
Acoustic Signals
Nuclear Related
Derived Observations
(wake)
EM Radiation

EM Radiation


EM Radiation
Nuclear Related
SURVEILLANCE
VOLUME
 Hundreds of
Nautical Miles
 Air/Surface/SubSurface



Hundreds of Miles
(Strategic)
Miles (Tactical)
Tens to Hundreds
of Miles about a
Battlefield

Global







SENSOR
PLATFORMS
Ships
Aircraft
Submarines
Ground-based
Ocean-based
Ground-based
Aircraft


Ground-based
Aircraft


Satellites
Aircraft
R EPRESENTATIVE D ATA F USION
A PPLICATIONS (N ON -D O D S YSTEMS )
SPECIFIC
APPLICATIONS
Condition-based
Maintenance


Robotics


Medical Diagnosis

Environmental
Monitoring

INFERENCES
BY DF PROCESS
Detection, characterization
of system faults
Recommendations for
maintenance/corrective
actions
Object location,
recognition
Guide the locomotion of
robot hands, feet, etc.
Location, identification of
tumors. abnormalities, and
disease
Identification, location of
natural phenomena
(earthquakes, weather)
PRIMARY
OBSERVABLE DATA
 EM Signals
 Acoustic Signals
 Magnetic
 Temperature
 X-rays
 TV
 Acoustic Signals
 EM Signals
 X-rays
 X-rays
 NMR
 Temperature
 IR
 Visual Inspection
 Chemical/Biological
Data
 SAR
 Seismic
 EM Radiation
 Core Samples
 Chemical/Biological
Data
SURVEILLANCE
VOLUME
 Microscopic
inspection to
hundreds of feet




Microscopic to
tens of feet
about the robot

SENSOR
PLATFORMS
Ships
Aircraft
Groundbased (e.g.,
factory)
Robot Body

Human body
volume

Laboratory

Hundreds of
miles
miles (site
monitoring)



Satellites
Aircraft
Groundbased
Underground
Samples


W HAT DOES DATA FUSION DO ?
OPERATORS &
SUPPORT SYSTEMS
SENSORS
MISSION
ENVIRONMENT
SOURCES
MISSION
EQUIPMENT &
WEAPONRY
TO SUPPORT
OPERATES ON:
• sensor data
• processed data
• reference data
DATA FUSION FUNCTIONS
• ASSOCIATION
• ESTIMATION
• PREDICTION
• INFERENCING
• ANALYSIS
• ASSESSMENT
• positional, identity, and
attribute estimates about
objects and events
• situation refinement
• refinement of enemy threats,
vulnerabilities, opportunities
H IERARCHY OF I NFERENCE
T ECHNIQUES
Type of Inference
Applicable Techniques
High
- Situation Assessment
- Behavior/Relationships of
Entities
- Identity, Attributes and
Location of an Entity
- Existence and Measurable
Features of an Entity
LEVEL
INFERENCELEVEL
INFERENCE
- Threat Analysis
Low
- Knowledge-Based Techniques
- Expert Systems
- Scripts, Frames, Templating
- Case-Based Reasoning
- Genetic Algorithms
- Decision-Level Techniques
- Neural Nets
- Cluster Algorithms
- Fuzzy Logic
- Estimation Techniques
- Bayesian Nets
- Maximum A Posteriori
Probability (e.g. Kalman
Filters, Bayesian)
- Evidential Reasoning
- Signal Processing Techniques
M ULTI -L EVEL /M ULTI -P ERSPECTIVE
I NFERENCING
where
Level O:
Signal
Refinement
what
when
Level 1:
Positional, Identity &
Attribute Refinement
who
why
Level 2:
Situation
Refinement
how
Level 3:
Threat
Refinement
Level 4:
Process
Refinement
DATA FUSION PROCESSING
PHYSICAL OBJECTS
individual
organizations
EVENTS
specific
aggregated
TERRAIN & ENEMY TACTICS
local
global
ENEMY DOCTRINE & OBJECTIVES
specific
global
FRIENDLY VULNERABILITIES & MISSION
options
needs
FRIENDLY ASSETS
specific & global
PN
PN
 PN
=
(PN + 2) - PN
B ENEFITS OF D ATA F USION : M ARGINAL
G AIN OF A DDED S ENSORS
0.13
0.12
0.11
0.10
0.09
0.08
0.07
0.06
0.05
0.04
0.03
0.02
0.01
N=1(13 Sensors)
N=3(35 Sensors)
N=5(57 Sensors)
0.5
0.6
0.7
0.8
0.9
1.0
PN Single Sensor Probability of Correct Classification
Nahin and Pokokski, IEEE AES, 16 May 1980.
B ENEFITS OF D ATA F USION :
E NHANCED S PATIAL R ESOLUTION
FLIR and Radar Sensor Data Correlation
FLIR
RADAR
TARGET REPORT
LOS
TARGET REPORT
LOS
SLANT RANGE
UNCERTAINTY
AZIMUTH
UNCERTAINTY
COMBINED
SLANT RANGE
UNCERTAINTY
TARGET
REPORT
ELEVATION
UNCERTAINTY
TARGET
REPORT
AZIMUTH
UNCERTAINTY
RADAR ABSOLUTE
UNCERTAINTY
REGION
ABSOLUTE UNCERTAINTY
REGION INTERSECTION
FLIR ABSOLUTE
UNCERTAINTY
REGION
ELEVATION
UNCERTAINTY
Adapted from W.G. Pemberton, M.S. Dotterweich, and L.B. Hawkins, “An Overview of Fusion Techniques”,
Proc. of the 1987 Tri-Service Data Fusion Symposium, vol. 1, 9-11 June 1987, pp. 115-123.
T HE D O D L EGACY:
E XTENSIVE R ESEARCH I NVESTMENTS
•
•
•
•
•
•
•
•
•
JDL Process model
Taxonomy of Algorithms
Lexicon
Engineering Guidelines
– Architecture
Selection
– Algorithm Selection
Evolving Tool-kits
Extensive Legacy of
technical papers, books
Training Materials
Test-beds
Numerous prototypes
C HALLENGES IN D ATA F USION . . .



Robust sensors:

no perfect sensors available

difficult to predict sensor performance

unable to effectively task geographically distributed non-commensurate sensors
Image and non-image fusion:

no true fusion of imagery and non-imagery data

unable to optimally translate image in time-series data into meaningful symbols

no requisite models for coherent fusion of non-commensurate sensor data
Robust target identification:

insufficient training data

unable to perform automated feature extraction

no unified method for incorporating implicit and explicit information for
identification (e.g., information learned from exemplars, model information, and
cognitive-based contextual information)
C HALLENGES IN D ATA F USION . . .


Unified calculus of uncertainty (e.g., random set theory):

do not know how to effectively use these techniques

limited experience in trade-offs and use of fuzzy logic, rules probability,
Dempster-Shafer’s method, etc.

unsure how to select the best uncertainty method
Pathetic cognitive models for Level 2 and 3:

unknown how to select the appropriate knowledge representation
techniques

argue about competing methods

do not know how to use hybrid methods

unable to perform knowledge engineering
C HALLENGES IN D ATA F USION . . .

Non-commensurate sensors:

uncertainty as how to optimize use of wildly non-commensurate sensors

inability to know how to link decision needs to sensor management

unable to effectively use 10N sensors

no consensus on MOE/MOP

Human computer interface (HCI):

trendy and driven by technology and not cognitive needs of user

suffer from the Gutenberg Bible syndrome

no effective tools to overcome cognitive deficiencies

unable to capitalize on built-in human pattern recognition (e.g., recognition of faces,
concepts of harmony)
D ATA F USION I SSUES . . .







What algorithms or techniques are appropriate and optimal for a particular
application?
What architecture should be used (i.e., where in the processing flow should data be
fused (viz. at the data, feature, or decision levels)?
How should individual sensor data be processed to extract the maximum amount of
information?
What accuracy can realistically be achieved by a data fusion process?
How can the fusion process be optimized in a dynamic sense?
How does the data collection environment (i.e., signal propagation, target
characteristics, etc.) affect the processing?
Under what conditions does multi-sensor data fusion improve system operation
(under what conditions does it impede or degrade performance)?
L ESSON 1 A SSIGNMENTS
Review the on-line introduction material (lesson 1)
 Read chapter 1 of Hall and McMullen
 Writing assignment 1: write a brief biographical sketch of
yourself (to share with the class)
 Writing assignment 2: write a paragraph describing the
occurrence of data fusion in a natural setting
 Team Assignment (T-1) - Meet with your assigned team to
discuss the semester collaboration

D ATA F USION T IP OF THE W EEK
“Here’s where we plan to use
data fusion.”
Despite enormous amounts of funding for data fusion research – there
is still no magic data fusion system or techniques!