Ad multos annos
Joos Vandewalle
Frameless Wave Computing
Tamás Roska
András Horváth and Miklós Koller
Pázmány P. Catholic University,
Budapest
Outline
• Cellular Wave Computing
• Frameless spatial-temporal computing
• Activation controlled frameless computing
• Delayed template frameless computing
• Outlook
Cellular Wave Computing
Spatial-temporal waves combined:
Input wave
Self wave
Activation wave (e.g. stroboscopic effect)
boundary wave
in a CNN Universal Machine with non-standard
CNN dynamics
The computational model
We have three wave dynamics evolving together:
• the dynamics of the spatial-temporal input flow (u)
• the self-dynamics of the computing cellular array (x defined by F)
• the dynamics of the active light-sources (v defined by G1 G2)
We are interested in their interaction in two cases:
‘independent activation’ case
x  F ( x, f1 ( x), u )


v  const. or v  G1 (v, f 2 (v)) 
‘adaptive activation’ case
x  F ( x, f1 ( x), u ) 

v  G2 (v, f 2 (v), f1 ( x)) 
u: two-dimensional input-flow
x: two-dimensional computation-flow (inner state of the cells)
v: two-dimensional flow defining the activation strength of the
light-sources
Frameless spatial-temporal computing
A. Solving an NP hard problem with a Cellular
Wave Computer with sparse nonlocal connection
in one sigle wave
B. Detecting spatial-temporal events
For A:
M. Ercsey-Ravasz, T. Roska, Z. Néda, „Cellular Neural Networks for NP-hard
optimization”,EURASIP Journal on Advances in Signal Processing, Special
issue: CNN Technology for Spatio-temporal Signal Processing, doi:
10.1155/2009/646975, 2009.
M. Ercsey-Ravasz, Z. Toroczkai, "Optimization Hardness as Transient Chaos in
an Analog Approach to Constraint Satisfaction", Nature Physics 7, 966 (2011)
arxiv:1208.0526
B. Molnár, Z. Toroczkai, M. Ercsey-Ravasz, "Continuous-time Neural Networks Without
Local Traps for Solving Boolean Satisfiability", CNNA 2012, Torino, Italy (2012)
doi:10.1109/CNNA.2012.6331411
Jedlik Laboratories, Pazmany University, Budapest
Problem statement for an NP complete problem
Solution of the K-SAT problem
The KSAT problem is NP complete and (widely used in the field of
optimization)
For our prototype problem we have 10 state variables
(xi) and 35 constraints (Ci) each of them containing
three state varaibles.
A constraint can be writen in the following form:
The problem is solved if each of the constarints are
satisfied in the formula.
Jedlik Laboratories, Pazmany University, Budapest
Problem statement
The example problem we have investigated can be written in the
following form:
Jedlik Laboratories, Pazmany University, Budapest
The Dynamics
This heterogenous CNN network contains two type of
cells (one for the state and one for the constraints) with
state variables s(t) and a(t)
Where:
Jedlik Laboratories, Pazmany University, Budapest
The architecture of the network
Jedlik Laboratories, Pazmany University, Budapest
Transient behaviour
The only fixed point of this system is the solution of the logical formula
The system converges to the solution from every initial state
Jedlik Laboratories, Pazmany University, Budapest
Transient behaviour
The only fixed point of this system is the solution of the logical formula
The system converges to the solution from every initial state
Jedlik Laboratories, Pazmany University, Budapest
Transient behaviour
The state transition of all 10 state variables 1.5 means true and -1.5 means
false
Jedlik Laboratories, Pazmany University, Budapest
B. Spatial-temporal event detection
• No frames in biology – multichannel
visual „computing” – starting in the retina
• Dynamic spatial-temporal motifs
• Examples:looming, horizontal and vertical
speed „calculated already in the retina,
like an optical flow
• Combining a few wave channels
• Registration of three modalities in
superior colliculus (vision, audio, touch)
Jedlik Laboratories, Pazmany University, Budapest
Activation controlled frameless computing
Use an unstable spatial-temporal self
wave
Use a constant activation dynamics
Apply the reflected wave as an input
The output dynamics becomes stable
in time and codes the terrain property
Jedlik Laboratories, Pazmany University, Budapest
The general scope:
the aim: to detect spatio-temporal features or events
the computational environment:
a Cellular Wave Computer architecture, where the
computations are done by locally propagating waves.
The active light of the sensors can be adaptively tuned in
spatial-temporal rule.
system setup:
computational method: software simulation
hardware framework: infrared lighting and sensor array
spatia-temporal algorithms
measurement and simulation results
Jedlik Laboratories, Pazmany University, Budapest
System setup
Sensor array:
to collect the input-data
from the scene
• A) 8x8 active LED array
with receiver photo sensors
• B) control- and readoutcircuits
Simulator:
to process the raw measurement data in the afore mentioned
computational model
• state-equations: both explicit Euler and RK-45 methods to
approximate
•software framework: c++, MATLAB
Jedlik Laboratories, Pazmany University, Budapest
The particular example:
The task: to detect a specific terrain feature (a bump or a
valley) which has bigger size than the sensorarray itself.
The key step: to apply the whole image flow on the input,
instead of the separately captured frames (frameless
detection).
Jedlik Laboratories, Pazmany University, Budapest
The computational model
We have three wave dynamics evolving together:
• the dynamics of the spatial-temporal input flow (u)
• the self-dynamics of the computing cellular array (x defined by F)
• the dynamics of the active light-sources (v defined by G1 G2)
We are interested in their interaction in two cases:
‘independent activation’ case
x  F ( x, f1 ( x), u )


v  const. or v  G1 (v, f 2 (v)) 
‘adaptive activation’ case
x  F ( x, f1 ( x), u ) 

v  G2 (v, f 2 (v), f1 ( x)) 
u: two-dimensional input-flow
x: two-dimensional computation-flow (inner state of the cells)
v: two-dimensional flow defining the activation strength of the
light-sources
Jedlik Laboratories, Pazmany University, Budapest
Template-program of the computing array
An asymmetric template with few non-zero element:
0 0 0
A s p s
0 r 0
0 0 0
B0 b 0
0 0 0
zz
• boundary condition: zero-flux
• size of the computational
array: 8 x 8 cells
• computational model:
Chua-Yang
Please consider the qualitative
effect of the vertical coupling
(from I. Petrás; size: 41 x 23;
FSR-model):
s  1.1,
T:
b  1.0,
p  1.0, r  0.6,
z  0.0
Jedlik Laboratories, Pazmany University, Budapest
Jedlik Laboratories, Pazmany University, Budapest
Delayed template frameless computing
Motivation
• Delays-time constant
differences in single synapses
• Drastic delay differences
between electrical and chemical
synapses
• Delay differences between
channels
Jedlik Laboratories, Pazmany University, Budapest
Detection of different spatial frequencies
Grayscale input image
Binary output image representing the
different structures
The CNN Universal Machine architecture is
capable of detecting structures (spatial
characteristic) by simple templates (operations)
in a simple and elegant way
Jedlik Laboratories, Pazmany University, Budapest
Detection of spatial frequencies in
practice
Periodic Pattern Formation and Its Applications in Cellular Neural Networks
Taisuke Nishio, Yoshifumi Nishio
Jedlik Laboratories, Pazmany University, Budapest
Frameless detection
Nyquist-Shanon sampling theorem:
If a function x(t) contains no frequencies higher than B hertz, it is completely
determined by giving its ordinates at a series of points spaced 1/(2B) seconds
apart.
Temporal detection: almost always frame based
temporal changes are the differences between the frames, not the real dynamics.
The detection of a spatial-temporal event can be easier in the continuous timedomain if the criteria above are not fulfilled.
Jedlik Laboratories, Pazmany University, Budapest
Frameless detection
It is difficult to identify the highest frequency in some dynamics: Tsunami
If the event is fast the (sampling and processing) detection has to be two times
faster.
Jedlik Laboratories, Pazmany University, Budapest
Spatial-temporal detection in the retina
Continuous analogue processing in the retina
Our retina (brain) handles dynamics, not image sequences:
Low frame-rate movies, animations
Jedlik Laboratories, Pazmany University, Budapest
Example: Looming detection
-Complex task
-Computationally expensive with regular architectures
-Simply done in the retina
- Done in an analogue, continuous way
Jedlik Laboratories, Pazmany University, Budapest
Looming
Modeling the response of the ganglion cells with a CNN chip
Jedlik Laboratories, Pazmany University, Budapest
Delay type CNN template
Not only the coupling strengths,
but also coupling delays are defined.
Extension of regular CNN dynamics, the delay is defined as the delay between the elements
CNN with implicit memory
B and W templates design
Jedlik Laboratories, Pazmany University, Budapest
Diagonal movement
detection
Input video
Output video
Jedlik Laboratories, Pazmany University, Budapest
Diagonal movement
detection
Excites the cells temporarily: the time of excitation is
controlled by the template
Jedlik Laboratories, Pazmany University, Budapest
Diagonal movement
detection
The excited cells remain excited (in this case black).
Detects the trajectory of an object.
Jedlik Laboratories, Pazmany University, Budapest
Detection of a given
trajectory
The aim is to identify the object moving
up in the input-flow
This task can be solved by a single
delayed-cnn template
Input Video
Output video
Jedlik Laboratories, Pazmany University, Budapest
Detection of a given
trajectory
The previous result can be extended to identify objects
moving along a given trajectory with a given speed
Input Video
Output video
Jedlik Laboratories, Pazmany University, Budapest
Delayed edge detection:
Identification of movement speed and
direction
The dark edge will appear where we can detect an edge on the current input flow, while
the bright edge will appear where the edge was τe time ago. This can be used to detect
the speed and the direction of the moving object.
Input Video
Output video
Jedlik Laboratories, Pazmany University, Budapest
Outlook
• Develop a design methodology for
spatial-temporal computing without
frames
• Develop a special physical mplementation
framework
• Towards a 3-layer vertically integrated
system
• Learning from neurobiological prototypes