Hierarchical Temporal Memory (HTM)
A new computational paradigm
based on cortical theory
Jeff Hawkins
May 10, 2006
IBM
Today’s PDA
Market Indicator
Pipe Dream
Driven By
Greed
Mother
Of All
Markets
Today’s Cognitive
Computing Indicator
Not in our
Lifetime
Any
Moment
Now

Not in our lifetime
-
Decades of effort
-
-
-
Not much success
-
-
AI
neural networks
fuzzy logic
5th generation project
decade of the brain
vision, language, robotics
Brain is very complex

Not in our lifetime

Any moment now
-
Decades of effort
-
Neocortex:
-
-
-
-
-
vision, language, robotics
Brain is very complex
-
-
Fast
Flexible
Robust
100 years of data
-
Not much success
-
-
AI
neural networks
fuzzy logic
5th generation project
decade of the brain
Anatomical, physiological
Mathematics
Common cortical algorithm
Cortical Theory (HTM)
People
Cars
Buildings
Words
Songs
Ideas
World
patterns
Senses
HTM/Cortex
“Causes”
“Beliefs”
People
Cars
Buildings
Words
Songs
Ideas
World
patterns
Senses
cause1
cause2
cause3
cause4
cause5
cause6
0.22
0.07
0.00
0.63
0.00
0.08
HTM/Cortex
HTM
Representations
of Causes
Causes
What does an HTM do?
1
2
3
4
Discover causes in the world
Infer causes of novel input
Predict future
Direct motor behavior
HTMs use a hierarchy of memory nodes
Belief
Sensory data
HTMs use a hierarchy of memory nodes
Beliefs
Sensory data
Each node:
Discovers causes (of its input)
Passes beliefs up
Passes predictions down
HTMs use a hierarchy of memory nodes
Beliefs
Sensory data
Each node:
Discovers causes (of its input)
Passes beliefs up
Passes predictions down
Each node:
Stores common sequences
Changing sensory data forms stable beliefs at top
Stable beliefs at top form changing sensory predictions
1) Why does hierarchy make a difference?
2) How does each node discover and infer causes?
Why does hierarchy make a difference?
1)
Shared representations lead to generalization and efficiency
Why does hierarchy make a difference?
1)
Shared representations lead to generalization and efficiency
2)
HTM hierarchy matches spatial and temporal hierarchy of
causes in world
Why does hierarchy make a difference?
1)
Shared representations lead to generalization and efficiency
2)
HTM hierarchy matches spatial and temporal hierarchy of
causes in world
3)
Belief propagation techniques ensure all nodes quickly reach
mutually compatible beliefs
Belief Propagation
90% cat
CPT
80% woof
20% meow
70% pig image
30% cat image
Why does hierarchy make a difference?
1)
Shared representations lead to generalization and efficiency
2)
HTM hierarchy matches spatial and temporal hierarchy of
causes in world
3)
Belief propagation techniques ensure all nodes quickly reach
mutually compatible beliefs
4)
Affords mechanism for attention
How does each node discover causes?
How does each node discover causes?
1)
2)
Learn common spatial patterns
Learn common sequences of spatial patterns
How does each node discover causes?
1)
Learn common spatial patterns
(things that happen at the same time are likely to have a
common cause)
How does each node discover causes?
1)
Learn common spatial patterns
Common patterns:
remember
Uncommon patterns:
ignore
How does each node discover causes?
1)
2)
Learn common spatial patterns
Learn common sequences of spatial patterns
How does each node discover causes?
1)
2)
Learn common spatial patterns
Learn common sequences of spatial patterns
Common sequence:
assign to cause
Common sequence:
assign to cause
Uncommon sequence:
ignore
time
How does each node discover causes?
1)
2)
3)
Learn common spatial patterns
Learn common sequences
Use context from above in hierarchy
Do HTMs really work?
Simple HTM vision system
Level 3
Level 2
Level 1
4 pixels
(32x32 pixel)
Training images
Training
images
Correct
Incorrect
Correctly recognized images
Numenta Plan
1)
Develop a detailed computational theory of
neocortical function (HTM)



On Intelligence (Times Books, 2004)
HTM white paper, www.numenta.com
Biological mapping paper, August 2006
Numenta Plan
1)
Develop a detailed computational theory of
neocortical function (HTM)
2)
Develop a software platform for HTM
applications
Numenta Platform
Run time environment
Dev Tools
Trainer
Debugger
Supervisor
API
Supervisor
Net list
Gigabit switch
Configurator
Node
Processor
Node Processor 2
:
Node Processor N
Fileserver
Numenta Plan
1)
Develop a detailed computational theory of
neocortical function (HTM)
2)
Develop a software platform for HTM
applications





Multiple processor/server architecture
Optimized C++ routines
Developer toolset with flexible scripting using Python
Supports Linux + MacOS. Windows to come.
Build a community of developers


Early access partners, 2nd meeting end of May 2006
Beta release early 2007
Numenta Plan
1)
Develop a detailed computational theory of
neocortical function (HTM)
2)
Develop a software platform for HTM
applications
3)
Test HTM with a machine vision system
Numenta Machine Vision System



Robust Object Recognition From Natural Images
Recognition Task Defined
Data collection in process



Highly realistic 3D models and textures used to generate sequences
90,000 images and 102 sequences collected to date
Each image has accurate alpha channel for programmatic 2D modifications
HTM Applications
1)
What humans find easy and computers hard
-
2)
vision, language, robotics
many apps from security to self-driving cars
extend with new senses, IR, sonar, radar…
Discovering causes in unusual worlds
-
geology, markets, weather, physics, genetics
HTM Capabilities
1)
2)
3)
4)
Discover causes
Inference
Prediction
Behavior
Beyond biology
1)
2)
3)
Faster
Larger
Exotic senses
www.OnIntelligence.org
www.Numenta.com
(white paper posted this week)
Today’s Cognitive
Computing Indicator
Not in our
Lifetime
Any
Moment
Now
Thank _ _ _
world
HTM
HTM models world,
including hardwired
motor behaviors
motor
world
HTM
motor
Representations of
motor behavior are
auto-associatively
paired with motor
generators
Hierarchical Temporal Memory
1)
2)
Powerful, flexible, robust
Can be applied to many problems
-
vision
language
robotics
manufacturing
business modeling
market modeling
network modeling
resource exploration
weather prediction
math, physics
Beliefs
(of causes)
?
Sensory data
Discovering and inferring causes has proven to be
very difficult, e.g.
- visual pattern recognition
- language understanding
- machine learning
 “What
is conspicuously
lacking is a broad framework
of ideas within which to
interpret these different
approaches.”

Francis Crick, 1979
Belief Propagation
Belief Propagation
“maybe diagonal line,
maybe vertical line”
Belief Propagation
“maybe diagonal line,
maybe vertical line”
“maybe diagonal line,
maybe vertical line”