Cognitive Computation: A Case Study in
Cognitive Control of Autonomous
Systems and Some Future Directions
Professor Amir Hussain, Dr Andrew Abel
1 Division of Computing Science and Mathematics
University of Stirling, Scotland
Work reported here is part of an ongoing UK EPSRC funded project, with:
Dr Erfu Yang1 (RF) & Prof Kevin Gurney2 (CI)
2Adaptive
Behaviours Research Group (ABRG)
Department of Psychology University of Sheffield, UK
The International Joint Conference on Neural Networks (IJCNN)
Dallas, Texas, August 4-9, 2013
1
Introduction
• Why Cognitive Computation?
• Why Cognitive Machines?
• Taylor’s Proposal on Cognitive Machines
• Cognitively Inspired Control of Autonomous Systems
• Towards a more generalised cognitive framework
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
2
Introduction
•
•
Cognitive computation
•
an emerging discipline linking together neurobiology, cognitive psychology and
artificial intelligence;
•
Springer’s journal Cognitive Computation publishing biologically inspired
theoretical, computational, experimental and integrative accounts of all aspects of
natural and artificial cognitive systems.
Professor John Taylor
•
founding Advisory Board Chair of Cognitive Computation;
•
proposed on how to create a cognitive machine equipped with multi-modal
cognitive capabilities.
• This keynote
•
first presents a novel modular cognitive control framework for autonomous
systems - potentially realizes the required cognitive action-selection and learning
capabilities in Professor Taylor's envisaged cognitive machine.
•
Possible future avenues for improving this work in a cognitively inspired manner
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
3
Why Cognitive Computation?
• Promote a more comprehensive and unified understanding of diverse topics
•
perception, action, and attention;
•
learning and memory;
•
decision making and reasoning;
•
language processing and communication;
•
problem solving and consciousness aspects of cognition.
• Industry, commerce, robotics and many other areas are increasingly calling for the
creation of cognitive machines, with ‘cognitive’ powers similar to those of
ourselves:
•
are able to ‘think’ for themselves;
•
reach decisions on actions in a variety of ways;
•
are flexible, adaptive and able to learn from both their own previous experience and
that of others around them
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
4
Why Cognitive Machines?
• A multi-disciplinary research challenge
• Understanding our own cognitive powers:
• how they are created and fostered;
• how they can go wrong due to brain malfunction;
• the modelling of the cognitive brain is an important step in
developing such understanding.
•
Creating autonomous robots and vehicles able to ‘think’ and ‘act’
cognitively and ethically:
• support us in our daily lives.
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
5
Taylor’s Proposal on Cognitive Machines
•
It was published at J.G. Taylor, “Cognitive computation,” Cogn. Comput, vol.1, pp.4–16 (2009).
•
•
Based on ideas published in many places
Taylor raised a number of very interesting points in his attempts to construct an
artificial being empowered with its own cognitive powers:
•
a range of key questions relevant to the creation of such a machine;
•
made detailed and methodical attempts to answer these questions;
•
providing convincing evidence from national and international research projects he
had led over the years.
•
Taylor’s proposal is one of very few attempts to construct a global brain
theory of cognition and consciousness.
•
It is based on a unique multi-modal approach that takes into consideration
vision and attention, motor action, language and emotion.
•
Conventional studies in cognition and consciousness have mostly focussed
on single modalities such as vision.
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
6
Taylor’s Proposal on Cognitive Machines
• Taylor asked a number of questions
•
•
What is human cognition in general, and how can it be modelled?
•
What are the powers of animal cognition, and how can they be
modelled?
•
How important is language in achieving a cognitive machine, and how
might it be developed in such a machine?
•
What are the benchmark problems that should be able to be solved by a
cognitive machine?
•
Does a cognitive machine have to be built in hardware?
•
How can hybridisation help in developing truly cognitive machines
•
Is consciousness crucial?
•
How are the internal mental states of others to be discerned?
Discussed notion of attention control
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
7
Taylor’s Proposal on Cognitive Machines
•
This approach to attention control relevant to our interests
• Will link to a case study that uses this as a basis for a new approach to
autonomous vehicle control
• Initially focus on control and decision making
• Ongoing work!
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
8
Cognitive Control of Autonomous
Systems
A Case Study
9
Two problem domains
Planetary rovers (SciSys)
Smart cars (Google)
10
Challenges in each domain
•
Urban driving in smart cars
• constantly changing trajectories
• moderated speed in urban areas
• ‘sentinel’ awareness of high pedestrian density
•
Planetary rovers
• real-time trajectory planning for feasible path to follow on
• Autonomous navigation
• Intelligent motion control with most optimal controller
• Active and smart obstacle avoidance
• ‘cognitive’ awareness of complex environments
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
11
The problem we tackle:
from partially specified trajectories to cognitive
control
X(t0)
X(t1)
Construct P(t) subject to
smoothness and time constraints
X(t2)
Path following with error correction
Take account of obstacles and challenges
12
The problem we tackle:
from partially specified trajectories to cognitive
control
Vehicle with given
dynamics and kinematics
Drives along P(t)
X(t0)
X(t1)
Construct P(t) subject to
smoothness and time constraints
X(t2)
13
Multiple controller methods
• Historically
• Hard switching
• One controller selected at any one time
• Issue is ‘bumpiness’ when switching between controllers
• Our goal
• ‘Bumpless’ control
• Soft switching
• Select a subset of all controllers
• Mix controller decisions together
• Smoother output
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
14
Existing hard switching control
supervisor
s
reference
input
e(t)
r(t)
+
_
controller 1
bank of candidate
controllers
controller n
1.
2.
disturbance/
noise
switching
signal
w
s
u
Plant Model
measured
output
y
control
signal
Key ideas:
Build a bank of alternative controllers
Switch among them online based on switching condition
15
Compare with the problem of action
selection in animals
Fight, flight or
feeding, but not “do
nothing”
 The animal solution is centred
on a set of sub-cortical brain
nuclei – the basal ganglia, which
act as a central ‘switch’ or selector
 Can we leverage the biological
solutions for use in AVC?
Basal ganglia in brain
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
The biology: Disinhibition gating and action channels
(compare with modular control)
Predisposing conditions
Ctx1: action1
Ctx2: action2
Thalamus
Thalamus
BG
BG
Motor resources
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
Modular control : Challenges
• Meeting multiple performance criteria
– Stability
– Convergence
– Tackling problems of ‘chattering’
– Anti-windup and ‘Bumpless’ switching
– Real-time operation
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory 18
Three-stage modular framework: a bio-inspired
approach
Actual trajectory
Measurements
(sensors, GPS, cameras ,etc)
(`sensing and perception’)

'Planned trajectory'
Actual velocty
Dynamics-based
vehicle control
(engine, drivetrain,etc )
(`action realization’)
vd
Kinematics-based motion
control
(basal ganglia, feedback
controllers, soft switching)
(`action selection’)
Selected
points on a
target path
Motion planning
(`goal selection’)
d
Target velocity and
steering angle
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
19
Using the biomimetic BG model
in a control environment
4-wheel rover – Kinematics-based
motion control and planning
20
Three-stage modular framework: case study
Actual trajectory
Measurements
(sensors, GPS, cameras ,etc)
(`sensing and perception’)

'Planned trajectory'
“Actual trajectory”
Actual velocty
Dynamics-based
vehicle control
(engine, drivetrain,etc )
(`action realization’)
vd
Kinematics-based motion
control
(basal ganglia, feedback
controllers, soft switching)
(`action selection’)
Selected
points on a
target path
Motion planning
(`goal selection’)
d
Target velocity and
steering angle
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
21
Kinematics-based motion control and planning
• The motion control of autonomous vehicles is mostly based on the
vehicle’s kinematics model
• Usually assumed that the vehicle’s internal dynamics can
immediately satisfy the velocity/steering angle requests from the
kinematics-based motion control
• This study:
– BG-based kinematic motion controllers are used for motion planning
and control
– Perfect dynamics assumed
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
22
Kinematics-based motion control and planning
Salience
sx
Fuzzy
Logic

s
s
s
Basal Ganglia
(BG)
Controller
Reference
input
xd
Two trajectory
Components
(input from
motion planner)
x
1
C
Error
+
+
ex
Controller
C2x
Controller
x
3
C
Controller
y
1
C
yd
+
+
ey
Controller
C
y
3
C
sy
Salience
s2y
s3y
x

 3x
x
2
v1x
v2x
gx
Gating
g1x
g 2x
g 3x
v x Feedback
u
linearisation
( x )( v   ( x ))

1
v3x
Output
Autonomous
Vehicle
Kinematics
to path
x
y “actual”
trajectory
v1y
v2y

vy
v3y
 1y
s 1y
Fuzzy
Logic
x
1
y
2
Controller
Controllers
are all Pole
placementbased
Gating function
Selection strength
x
1
x
2
x
3
Basal Ganglia
(BG)
 2y
 3y

y
Selection strength
Gating
gy
g1y
g 2y
g 3y
Gating function
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
23
Action surface for fuzzy salience model
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
24
Simulation Results
A. Circular Trajectory Tracking Control
(b) x − y trajectory comparison for BG-based
switching and a single feedback linearization
motion controller under noise
(a) States in the circular tracking with BG-based switching and
a single feedback linearization motion controller under noises
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
25
C. General Path Tracking – double lane change and roundabout
x-y trajectory under BG-based switching and a single feedback linearization motion controller under noises
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
26
Using the biomimetic BG model
in a control environment
4-wheel rover – B-Spline path planning
and three-stage motion control with
integrated kinematics and dynamics
27
Smooth path planning with B-splines
 The dimension of the knot vector: 24;
 The number of control points: 18;
 The degree of splines: 5
6
5
4
3
2
1
0
0
20
40
60
80
100
120
Control points and smooth path planned with B-spline method
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
28
General Path Tracking – double lane change and roundabout
4
Single fixed
BG switching
desired
3.5
3
y(m)
2.5
2
1.5
1
0.5
0
20
40
60
x(m)
80
100
120
Comparison of BG-based soft switching control and single-fixed controller with noises
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
29
Comparison of Control Performance (MSE: Mean Squared Error)
Perform
ance
BG without
noise
MSE in
x
0.0044
MSE in
y
0.0000016832
MSE in
x-y
0.0031
BG with
noise
Single without
noise
Single with
noise
0.0565
0.0652
0.00090293
0.000014852
0.0020
0.0033
0.04
0.0461
0.0046
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
30
Summary
• BG-based controller selection is bumpless ‘soft-switching’
because it combines outputs of multiple controllers
– We have some evidence that this also helps avoid windup &
chattering
• BG will allow adaptive control by varying internal
parameters which are now better understood from our
neurobiological models
• Based on model of biological decision making
• Attention switching using salience
• Ongoing work
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
31
Autonomous Control Specific Future Work
• Test against traditional switched controller
designs with same controllers
• Adaptive online operations
– learn salience weights to BG controller
– Dynamic allocation of controllers
• Use of more realistic models
• Real experimental test beds
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
32
Cognitive Future Work
• Incorporating vision
– Better able to react to world
– Use of multiple modalities
• Dual process control….
– Automatic behaviour mode
– Process known differently from unknown
– Learning over time, becomes automatic
– Mimics processing in the brain
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
33
Cognitive Computation…
…towards a multimodal framework
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
34
More Cognitive Computation?
• This is a specific case study
– Inspired by work of John Taylor
• Cognitive Computation is very wide ranging field of research
– Can be applied in many different contexts
– Means different things to different people
• Presentation tomorrow
– Discuss cognitive computation in more depth
– Application in more fields
• Want to consider a more general cognitive framework
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
35
Sentic Computing
• Sentiment Analysis
• Common sense computing
• Read emotion and tone from text
• Traditional approaches inadequate
– Machine Learning
– Keyword counting
– May identify topic, but not sentiment
• Concept based approach
– Can assign emotions to concepts
– Relate similar concepts together
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
36
AffectNet Graph
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
37
AffectiveSpace
E. Cambria and A. Hussain. Sentic Computing: Techniques, Tools, and Applications.
Dordrecht, Netherlands: Springer, ISBN: 978-94-007-5069-2 (2012)
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
38
Sentic Computing
• Sentic Activation
• Consider conscious and unconscious level processing
• The two interact
• Can be used for sentiment analysis
• Emotion detection
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
39
Multimodal Speech Processing
• Traditional hearing aids focus on single modality
• This is not the whole story!
• Perception, attention switching
• Multimodality
• McGurk effect
• Lip reading used in noisy environments
– More extensively by those with hearing problems
• Visual information used, but only when appropriate
• Conscious and unconscious processing
– Speech often works on prediction
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
40
Multimodal Speech Processing
• A different direction for listening devices and hearing aids
• Consider how people actually hear
• Lip reading as part of speech filtering
• Cognitively inspired nuanced use of visual information
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
41
General Cognitive Framework
• Taylor discussed the creation of a cognitive being
– Language
– Consciousness
– Decision making
– Memory
– Emotional coding
• Aim is to consider a more general purpose approach
– Basal Ganglia inspired decision making
– Concept based emotion analysis
– Multimodal speech interpretation capabilities
– Dual level processing
• Can they be combined into a multimodal framework?
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
42
General Cognitive Framework
• Multimodality
– More environmentally aware
– Additional sensors to feed into a vehicle control system
– Vision, sound, weather conditions etc.
• Communication
– Communicate with those in the car and outside
– Speech recognition and generation
– Sentiment analysis from passengers
– Able to learn and adapt to wishes of those in car
• Adjust behaviour to suit conditions and emotions
• Multimodal social and cognitive agents
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
43
Sentic Blending: Scalable Multimodal Fusion for the
Continuous Interpretation of Semantics and Sentics
• A general and scalable methodology termed sentic blending, for
interpreting the conceptual and affective information associated
with natural language through different modalities:
•
enables the continuous interpretation of semantics and sentics (i.e., the
conceptual and affective information associated with natural language);
•
based on the integration of an affective common-sense knowledge base with any
multimodal cognitive signal image and control processing module.
•
operates in a multidimensional space that enables the generation of a continuous
stream characterizing user’s semantic and sentic progress over time - despite
the outputs of the unimodal categorical modules having very different timescales and output labels.
•
Uses decision fusion
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
44
A sample schema of continuous multimodal
fusion through sentic blending
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
45
An application example: SenticNet Engine
Ensemble streams obtained when applying sentic blending to the SenticNet engine
(left) and the facial expression analyser (right), without ‘sentic kinematics’ filtering.
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
46
An application example: SenticNet Engine
Ensemble stream obtained when applying sentic blending to the proposed
conversation, with (right) and without (left) using ‘sentic kinematics’ filtering.
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
47
Performance Comparison
Confusion matrix obtained combining the five classifiers. Success rates for neutral, joy, and
surprise are very high, but disgust, anger, and fear tend to be confused
Confusion matrix obtained after human assessment. Success ratios considerably increase,
meaning that the adopted classification strategy is consistent with human classification.
48
General Cognitive Framework
• Considers the emotional states of others
• Considers aspects of human cognition
• Considers the issue of language
• Considers benchmark problems
– Convincing communication
• Could be extended to include vehicle and language control
– Driving, extremely challenging problem
– Dual level processing
– Cognitively inspired use of different modalities
• Dual layer processing is unifying
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
49
Acknowledgements
• Everyone who helped to organise this conference!
• All of the COSIPRA Lab
– http://cosipra.cs.stir.ac.uk
– Dr Erfu Yang, Prof Leslie Smith, Dr Erik Cambria
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
50
• Thanks for listening!
• Questions?
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
51
Appendix
52
Two Modes of Biological Action Selection:
Automatic/Habitual and Controlled/Executive Processing - I
In psychological literature, modes of behavioural control refer to
automatic (or habitual) & controlled (or executive) processing respectively
with their joint use constituting a dual-process theory of behaviour
Controlled processing is under the subject’s direct and active control, is
slow, and requires serial attention to component stimuli or sub-tasks. In
contrast, automatic control is less effortful, less prone to interference from
simultaneous tasks, is driven largely by the current stimulus and does not
necessarily give rise to conscious awareness
Dual-process theory also supposes a dynamic transfer of control under
learning.
The development of automatic processing has close similarities with the
notion of stimulus-response (S-R) learning, or habit learning.
Controlled processing may be likened to goal-directed behaviour in
animals.
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
53
Two Modes of Biological Action Selection:
Automatic/Habitual and Controlled/Executive Processing - II
Habits are supported in closed-loop circuits through BG associated with
sensorimotor cortical areas.
The pre-frontal-cortex (PFC) serves as an ‘executive' or supervisory role
in enabling controlled processing. PFC also forms loops through BG. The
‘supervisory' PFC works to modulate or bias the action selection of the
automatic (sensorimotor) processing system.
Controlled processing dominates in the early acquisition of new skills which
subsequently, when well-practiced, are carried out using automatic processing.
As in dual-process theory, it is supposed that goal-directed (non-habitual)
behaviours governed by PFC can transfer into habits in sensorimotor loops by
learning therein under the influence of the PFC loops
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
54
Kinematics Vehicle model for
Motion Control and Planning:
: heading angle
: steering angle
Yang, Hussain, and Gurney. (BICS 2013) to appear.
nonholonomic constraints
imposes the physical constraint, the steering angle delta is
contrained within a desired (state)
range (to enable a smooth time invariant
control solution)
: Cartesian location
If the steering angle is selected
as one control input, then the
kinematics model can be
further simplified as:
with the inputs chosen as
The State Constraint:
55
Advanced Motion Controller Method: I/O Feedback Linearization Controller Design
Process
Linear Controller Design, e.g.
LQR, zero-pole placement
State-space
Linearization
Push Back
Augmented
System after
Extension
Dynamic Extension
(if needed)
Original Nonlinear
System
Erfu Yang, Amir Hussain, and Kevin Gurney. A basal ganglia inspired soft switching approach to the motion control of a car-like autonomous vehicle.
56
The 2013 International Conference on Brain Inspired Cognitive Systems (BICS 2013),June 9-11, 2013, Beijing, China, to appear.
Fuzzy logic rules for BG-Based soft switching motion control
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
57
Path/Trajectory Planning:
Consider two sixth-order polynomials of time t and their derivatives
the initial and final boundary points are:
58
Thus, solving the following equations
59
Generic Solution
If T=30, the resulting solution is
60
Dynamics vehicle model used (for car-like rover)
Eric N Moret. Dynamic Modeling and Control of a Car-Like Robot. Thesis, Virginia Polytechnic Institute and State University,2003.
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
61
Gain-Scheduling vs BG control
Another important idea formed in this project thus far is to utilize the
reference signal as a priori knowledge of the control system under
consideration to aid the realization of automatic (habitual) mode behaviour.
This shares some similarity with traditional gain-scheduling solution in
which a family of controllers such as PI or PID related to the control
reference signal and desired output are designed (Zhao et al, ).
An engine control model for autonomous vehicle has been employed
initially to illustrate this traditional gain-scheduling approach.
Throttle Ang.
Combustion
u
u
Air charge
Air charge
Engine Speed, N
1
Throttle
perturbation
Air Charge
Engine
Air Charge
N
Torque
Teng
rad/s
Speed
to rpm
(rpm)
Throttle & Manifold
Induction to
N
N
Power Stroke Delay
-K-
Speed
Tload
Load
Vehicle
Dynamics
Drag Torque
Cognitive Signal Image and Control Processing Research Laboratory
62
Gain-Scheduling vs. Reference-based habits?
11 PI controllers are demonstrated.
So, the action (controller) selection in the ‘automatic mode’ can be
realized by mapping the reference signal (desired engine speed in the case)
to the controllers’ parameters (gains).
In our proposed BG-based soft switching approach, this action selection
can be realised in a more natural way, which will be demonstrated further in
the vehicle’s cognitive cruise control - NEXT
Gain-Scheduling Proportional and Integral Gains
Step Response
0.016
From: engine l/Sum To: Out(1)
o
1.2
0.014
1
Kp
Ki
0.012
0.8
Amplitude
0.6
0.01
0.4
0.008
0.2
0.006
0
-0.2
0
0.5
1
1.5
Time (seconds)
2
2.5
0.004
2000
2200
2400
2600
2800
3000
3200
3400
3600
3800
4000
Speed (rpm)
Cognitive Signal Image and Control Processing Research Laboratory
63
Appendix B
Sentic blending
64
Sentic Blending: Scalable Multimodal Fusion for the
Continuous Interpretation of Semantics and Sentics
 Aimed at extending the modular cognitive framework to incorporate
additional modalities
 by integrating vision, language and emotion;
 for enabling multi-modal social cognitive and affective behavioural capabilities in
autonomous agents.
 A general and scalable methodology termed sentic blending, for
interpreting the conceptual and affective information associated with
natural language through different modalities:
 enables the continuous interpretation of semantics and sentics (i.e., the conceptual and
affective information associated with natural language);
 based on the integration of an affective common-sense knowledge base with any
multimodal cognitive signal image and control processing module.
 operates in a multidimensional space that enables the generation of a continuous stream
characterizing user’s semantic and sentic progress over time - despite the outputs of
the unimodal categorical modules having very different time-scales and output labels.
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
65
A sample schema of continuous multimodal
fusion through sentic blending
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
66
An application example: SenticNet Engine
Ensemble streams obtained when applying sentic blending to the SenticNet engine
(left) and the facial expression analyser (right), without ‘sentic kinematics’ filtering.
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
67
An application example: SenticNet Engine
Ensemble stream obtained when applying sentic blending to the proposed
conversation, with (right) and without (left) using ‘sentic kinematics’ filtering.
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
68
Performance Comparison
Confusion matrix obtained combining the five classifiers. Success rates for neutral, joy, and
surprise are very high, but disgust, anger, and fear tend to be confused
Confusion matrix obtained after human assessment. Success ratios considerably increase,
meaning that the adopted classification strategy is consistent with human classification.
69
Appendix C
Attention control
70
Taylor’s Attention Control
Goal
Module
ATTN
Signal
Creator
Input
Module
Ballistic Attention Control System
ATTN
Signal
Creator
Goals
Input
Module
Attention
Controller
WM cd
ATTN
Copy
Module
Monitor
Buffer
Memory
Attention copy of Attention Control
Wm input
Cortex
Objects/
Features
The corollary discharge of attention model
(CODAM) for consciousness 71
Multiple controller methods
• One promising approach to AVC is to break the task into sub-tasks, each
valid over a restricted range of conditions, and to switch between them
when required, based on sensory and internally generated signals.
• Historically achieved using several approaches such as
• PID+Gain scheduling (Ahmad 09)
• Sliding mode control
• Dynamic feedback linearisation (Oriolo 02; Kulkarn,NASA JPL )
• Fuzzy logic+PID+multiple models (Iagnemma 99, MIT; Narendra,
Yale; Hussain & Gurney et al. 08,09, Stirling)
• Neural approaches (Shumeet 96, Kawato & Wolpert, 2001)
• Decision-theoretic control (Zilberstei,02)
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
72
Appendix D - A biological
interlude
Basal ganglia and action selection
73
BG Functional Model
S1
S2
S3
Feedforward
Off-centre, on-surround
network
Z1
Z2
Z3
(Gurney et al. 2001)
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
Vector inputs: effective salience
• Effective salience s (scalar), with input vector x, and channel weight
vector w, is given by s = f(w, x)
• s = f(w, x) may be simple dot product or arbitrary nonlinear function
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory 75
Appendix E - Using the
biomimetic BG model in a
control environment
4-wheel rover – Kinematics-based
motion control and planning
76
Three-stage modular framework: case study
Actual trajectory
Measurements
(sensors, GPS, cameras ,etc)
(`sensing and perception’)

'Planned trajectory'
“Actual trajectory”
Actual velocty
Dynamics-based
vehicle control
(engine, drivetrain,etc )
(`action realization’)
vd
Kinematics-based motion
control
(basal ganglia, feedback
controllers, soft switching)
(`action selection’)
Selected
points on a
target path
Motion planning
(`goal selection’)
d
Target velocity and
steering angle
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
77
Kinematics-based motion control and planning
• The motion control of autonomous vehicles is mostly based on the
vehicle’s kinematics model
• Usually assumed that the vehicle’s internal dynamics can
immediately satisfy the velocity/steering angle requests from the
kinematics-based motion control
• This study:
– BG-based kinematic motion controllers are used for motion planning
and control
– Perfect dynamics assumed
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
78
Kinematics-based motion control and planning
Salience
sx
Fuzzy
Logic

s
s
s
Basal Ganglia
(BG)
Controller
Reference
input
xd
Two trajectory
Components
(input from
motion planner)
x
1
C
Error
+
+
ex
Controller
C2x
Controller
x
3
C
Controller
y
1
C
yd
+
+
ey
Controller
C
y
3
C
sy
Salience
s2y
s3y
x

 3x
x
2
v1x
v2x
gx
Gating
g1x
g 2x
g 3x
v x Feedback
u
linearisation
( x )( v   ( x ))

1
v3x
Output
Autonomous
Vehicle
Kinematics
to path
x
y “actual”
trajectory
v1y
v2y

vy
v3y
 1y
s 1y
Fuzzy
Logic
x
1
y
2
Controller
Controllers
are all Pole
placementbased
Gating function
Selection strength
x
1
x
2
x
3
Basal Ganglia
(BG)
 2y
 3y

y
Selection strength
Gating
gy
g1y
g 2y
g 3y
Gating function
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
79
•
•
•
•
Each controller has different parameters
One salience, one controller
300 controllers
Sub tasks – following path
80
•
•
Input signals (x,y) separated
Each input fed into all controllers
– Each controller is different
– Outputs a recommended action
•
Signal and error also fed into fuzzy logic
– Determines salience,
– urgency, based on error and reference
•
Apply to basal ganglia model
– Selection strength of each controller
•
Gating function to normalise
– Between 0 and 1
•
Gating function output applied to each controller
– Acts as a weight, could be zero
•
•
•
Outputs summed
Recoupled to determine output
See BICS 2013 paper
81
Action surface for fuzzy salience model
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
82
• Each one represents a different salience
output
• Essentially, each one reacts differently
83
Simulation Results
A. Circular Trajectory Tracking Control
(b) x − y trajectory comparison for BG-based
switching and a single feedback linearization
motion controller under noise
(a) States in the circular tracking with BG-based switching and
a single feedback linearization motion controller under noises
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
84
• Currently only single controller
• Testing against hard controller currently
85
B. Lane Change
2
2
1.5
1.5
y(m)
2.5
x(m)
2.5
1
Single fixed
BG switching
desired
0.5
0
5
10
15
Time (s)
20
25
Single fixed
BG switching
desired
0.5
0
30
1.5
0
5
10
15
Time (s)
y(m)
0
2
1
20
25
30
2.5
1.4
2
1.2

 y(m)
25
30
0.5
0.61
0.4
0.5
0.2
-0.2
Single fixed
BG switching
desired
0
5
10
0
5
10
15
Time
15 (s)
Time (s)
20
20
25
25
Single fixed
BG switching
desired
0.5
Single fixed
BG switching
0.8
00
Single fixed
BG switching
1
1
1.5
gle fixed
switching
red
1
1.5
0
-0.5
0
-0.5
-1
0
Single fixed
BG switching
1
30
30
1.5
2
2.5
3
x(m)
-1.5
0
5
10
15
Time (s)
(a) States under BG-based switching and a single feedback
linearization
motion controller under noises
1.5
1
0.5
20
25
30
(b) x − y trajectory comparison for BG-based
switching and a single feedback linearization
motion controller under noise
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive
Signal Image and Control Processing Research (COSIPRA) Laboratory
0.5
gle fixed
86
C. General Path Tracking – double lane change and roundabout
x-y trajectory under BG-based switching and a single feedback linearization motion controller under noises
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
87
Using the biomimetic BG model
in a control environment
4-wheel rover – B-Spline path planning
and three-stage motion control with
integrated kinematics and dynamics
88
• B spline generates smoother path
89
Smooth path planning with B-splines
 The dimension of the knot vector: 24;
 The number of control points: 18;
 The degree of splines: 5
6
5
4
3
2
1
0
0
20
40
60
80
100
120
Control points and smooth path planned with B-spline method
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
90
What is spline?
What is the knot vector, control parameter,
controlling?
91
General Path Tracking – double lane change and roundabout
4
Single fixed
BG switching
desired
3.5
3
y(m)
2.5
2
1.5
1
0.5
0
20
40
60
x(m)
80
100
120
Comparison of BG-based soft switching control and single-fixed controller with noises
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
92
Comparison of Control Performance (MSE: Mean Squared Error)
Perform
ance
BG without
noise
MSE in
x
0.0044
MSE in
y
0.0000016832
MSE in
x-y
0.0031
BG with
noise
Single without
noise
Single with
noise
0.0565
0.0652
0.00090293
0.000014852
0.0020
0.0033
0.04
0.0461
0.0046
Cognitive Signal Image Processing and Control Systems Research (COSIPRA) Laboratory
Cognitive Signal Image and Control Processing Research (COSIPRA) Laboratory
93