Lifelong Machine Learning
and Reasoning
Daniel L. Silver
Acadia University,
Wolfville, NS, Canada
TAMALE Seminar Series
University of Ottawa - February 5, 2015
1
Intelligent Information Technology Research Lab, Acadia University, Canada
Introduction
2
Intelligent Information Technology Research Lab, Acadia University, Canada
Talk Outline





Position and Motivation
Lifelong Machine Learning
Deep Learning and LML
Learning to Reason
Summary
3
Intelligent Information Technology Research Lab, Acadia University, Canada
Position


It is now appropriate to seriously consider the
nature of systems that learn and reason over
a lifetime
Advocate a systems approach in the context
of an agent that can:



Acquire new knowledge through learning
Retain and consolidate that knowledge
Use it in future learning, reasoning
and other aspects of AI
4
Intelligent Information Technology Research Lab, Acadia University, Canada
Moving Beyond Learning
Algorithms - Rationale
1. Strong foundation in prior work
(Q. Yang, D. Silver 2013)
2. Inductive bias is essential to learning
(Mitchell, Utgoff 1983; Wolpert 1996)
 Learning systems should retain and use prior
knowledge as a source of inductive bias
 Many real-world problems are non-stationary;
have drift; ; inductive bias shifts
5
Intelligent Information Technology Research Lab, Acadia University, Canada
Moving Beyond Learning
Algorithms - Rationale
3. Practical Agents/Robots Require LML


Advances in autonomous robotics and intelligent
agents that run on the web or in mobile devices
present opportunities for employing LML systems.
The ability to retain and use learned knowledge is
very attractive to the researchers designing these
systems.
6
Intelligent Information Technology Research Lab, Acadia University, Canada
Moving Beyond Learning
Algorithms - Rationale
4. Increasing Capacity of Computers

Advances in modern computers provide the
computational power for implementing and testing
practical LML systems

IBMs Watson (2011)




90 IBM Power-7 servers
Each with four 8-core processors
15 TB (220M text pages) of RAM
Tasks divided into thousands of stand-alone
jobs distributed among 80 teraflops (1 trillion ops/sec)
7
Intelligent Information Technology Research Lab, Acadia University, Canada
Moving Beyond Learning
Algorithms - Rationale
5. Theoretical advances in AI: ML  KR


“The acquisition, representation and transfer of
domain knowledge are the key scientific concerns
that arise in lifelong learning.” (Thrun 1997)
KR plays an important a role in LML



Interaction between knowledge retention & transfer
LML has the potential to make advances on the
learning of common background knowledge
Leads to questions about learning to reason
8
Intelligent Information Technology Research Lab, Acadia University, Canada
Lifelong Machine Learning



Considers systems that can learn many tasks over a
lifetime from one or more domains
Concerned with methods of retaining and using learned
knowledge to improve the effectiveness and efficiency of
future learning
We investigate systems that must learn:




From impoverished training sets
For diverse domains of tasks
Where practice of the same task happens
Applications:

Agents, Robotics, Data Mining, User Modeling
10
Intelligent Information Technology Research Lab, Acadia University, Canada
Lifelong Machine Learning
Framework
Testing
Examples
Instance Space
X
Domain
Knowledge
Retention
(xi, y =f(xi))
Knowledge
Transfer
Training
Examples
Inductive
Bias, BD
Knowledge
Selection
Inductive
Learning System
short-term memory
S
Model of
Classifier
h
Prediction/Action = h(x)
11
Intelligent Information Technology Research Lab, Acadia University, Canada
Essential Ingredients of LML

The retention (or consolidation) of learned
task knowledge


Knowledge Representation perspective
Effective and Efficient Retention




Resists the accumulation of erroneous knowledge
Maintains or improves model performance
Mitigates redundant representation
Allows the practice of tasks
12
Intelligent Information Technology Research Lab, Acadia University, Canada
Essential Ingredients of LML

The selective transfer of prior knowledge
when learning new tasks


Machine Learning perspective
More Effective and Efficient Learning



More rapidly produce models
That perform better
Selection of appropriate
inductive bias to guide search
13
Intelligent Information Technology Research Lab, Acadia University, Canada
Essential Ingredients of LML

A systems approach


Ensures the effective and efficient interaction of
the retention and transfer components
Much to be learned from the writings of early
cognitive scientists, AI researchers and
neuroscientists such as Albus, Holland, Newel,
Langley, Johnson-Laird and Minsky
14
Intelligent Information Technology Research Lab, Acadia University, Canada
Overview of LML Work




Supervised Learning
Unsupervised Learning
Hybrids (semi-supervised, self-taught, cotraining, etc)
Reinforcement Learning


Mark Ring, Rich Sutton,
Tanaka and Yamamura
15
Intelligent Information Technology Research Lab, Acadia University, Canada
LML via context sensitve csMTL
Task Rehearsal
Functional transfer
(virtual examples) for
slow consolidation
f1(c,x)
Short-term
Learning
Network
f’(c,x)
Long-term
Consolidated
Domain
Knowledge
Network
Representational
transfer from CDK
for rapid learning
c1
One output
for all tasks
ck x1
Task Context
Silver, Poirier, Currie (also Tu, Fowler)
Inductive transfer with context-sensitive neural networks
Mach.Learning (2008) 73: 313–336
Intelligent Information Technology Research Lab, Acadia University, Canada
xn
Standard Inputs
17
An Environmental Example
16
MAE (m^3/s)
15
14
13
12
11
0
No Transfer
1
2
3
4
Years of Data Transfered
Wilmot
Sharpe
Sharpe & Wilmot
5
6
Shubenacadie
x = weather data
Stream flow rate prediction [Lisa Gaudette, 2006]
f(x) = flow rate
19
Intelligent Information Technology Research Lab, Acadia University, Canada
csMTL and
Tasks with Multiple Outputs

Liangliang Tu (2010)



Image Morphing:
Inductive transfer
between tasks that
have multiple outputs
Transforms 30x30
grey scale images
using inductive
transfer
Three mapping tasks
NA
NH
NS
20
Intelligent Information Technology Research Lab, Acadia University, Canada
csMTL and
Tasks with Multiple Outputs
Demo
Intelligent Information Technology Research Lab, Acadia University, Canada
22
Two more Morphed Images
Passport
Angry
Filtered
Passport
Sad
Filtered
23
Intelligent Information Technology Research Lab, Acadia University, Canada
Unsupervised LML
Deep Learning Architectures

Consider the problem of trying to classify
these hand-written digits.
Hinton, G. E., Osindero, S. and Teh, Y. (2006) A
Layered networks of unsupervised
auto-encoders efficiently develop
hierarchies of features that capture
regularities in their respective inputs
Intelligent Information Technology Research Lab, Acadia University, Canada
Deep Learning Architectures
2
2000 top-level artificial neurons
3
0
1
2
3
4
5
6
7
8
9
DLA Neural Network:
- Unsupervised training, followed
by back-fitting
- 40,000 examples
- Learns to:
* recognize digits using labels
* reconstruct digits given a label
- Stochastic in nature
1
500 neurons
(higher level features)
500 neurons
(low level features)
Images of
digits 0-9
(28 x 28 pixels)
Intelligent Information Technology Research Lab, Acadia University, Canada
Deep Learning Architectures

Develop common features from unlabelled
examples using unsupervised algorithms
Courtesy of http://youqianhaozhe.com/research.htm
Intelligent Information Technology Research Lab, Acadia University, Canada
26
Deep Learning Architectures



Stimulates new ideas about how knowledge
of the world is learned, consolidated, and
then used for future learning and reasoning
Learning and representation of common
background knowledge
Important to Big AI problem solving
29
Intelligent Information Technology Research Lab, Acadia University, Canada
LML and Reasoning



ML  KR … a very interesting area
Knowledge consolidation provides insights into how best
to represent common knowledge for use in learning and
reasoning
A survey of learning / reasoning paradigms has identified
two additional promising bodies of work:
 NSI - Neural-Symbolic Integration
 L2R - Learning to Reason
30
Intelligent Information Technology Research Lab, Acadia University, Canada
Neural-Symbolic Integration


Considers hybrid systems that integrate
neural networks and symbolic logic
Takes advantage of:


Learning capacity of
connectionist networks
Transparency and
reasoning capacity of
logic
[Garcez09,Lamb08]
31
Intelligent Information Technology Research Lab, Acadia University, Canada
Learning to Reason (L2R)




Takes a probabilistic perspective on learning and
reasoning [Kardon and Roth 97]
Agent need not answer all possible knowledge queries
Only those that are relevant to the environment of a
learner in a probably approximately correct (PAC)
sense (w.r.t. some prob. dist.) [Valiant 08, Juba 12&13 ]
Assertions can be learned to a desired level of
accuracy and confidence using training examples of the
assertions
39
Intelligent Information Technology Research Lab, Acadia University, Canada
Learning to Reason (L2R)

We are working on a LMLR approach that uses:
Concepts from Multiple Task Learning
 Primed by Unsupervised Deep Belief Network learning
PAC-learns multiple logical assertions expressed as binary
examples of Boolean functions
Reasoning is done by querying the trained network using similar
Boolean examples and looking for sufficient agreement on T/F




Uses a combination of:


DLA used to create hierarchies of abstract DNF-like features
Consolidation is used to integrate new assertions with prior
knowledge and to share abstract features across a domain
knowledge model
40
Intelligent Information Technology Research Lab, Acadia University, Canada
Learning to Reason (L2R)

Example: To learn the assertions


The L2R system would be provided with examples of
the Boolean functions equivalent to the assertion and
subject to a distribution D over the examples :






(A ∧ B) ∨ C, and (A ∨ C) ∧ D
abcdT
000*0
001*1
010*1
011*1
abcdT
100*1
101*1
110*1
111*1
abcdT
0*000
0*010
0*100
0*111
abcdT
1*000
1*011
1*100
1*111
To query the L2R system about an assertion:

A ∨ ~C – use examples of this function to test the system fro
agreement
41
Intelligent Information Technology Research Lab, Acadia University, Canada
Learning to Reason (L2R)
Early Experiments (Jane Gomes)
f’(c,x)
20-10-10-1 network
c1
ck x1
Task Context
xn
Standard Inputs
42
Intelligent Information Technology Research Lab, Acadia University, Canada
Learning to Reason (L2R)
Early Experiments (Jane Gomes)
Learning the Law of Syllogism:
KB: (A  B) ^ (B  C)
Q: (A  C)
f’(c,x)
When trained on the 8 examples of
(A  B) and (B  C)
And queried with 4 examples
(A  C)
Con A B C T/F
101 00 0 1
101 00 1 1
c1
ck x1
xn
101 10 0 0
101 10 1 1
Task Context
Standard Inputs
100% correct
43
Intelligent Information Technology Research Lab, Acadia University, Canada
Summary




Propose that the AI community move to systems that are
capable of learning, retaining and using knowledge over a
lifetime
Opportunities for advances in AI lie at the locus of machine
learning and knowledge representation
Consider the acquisition of knowledge in a form that can be
used for more general AI, such as Learning to Reason (L2R)
 Methods of knowledge consolidation will provide insights
into how to best represent common knowledge –
fundamental to intelligent systems
46
Intelligent Information Technology Research Lab, Acadia University, Canada
Thank You!
QUESTONS?



danny.silver@acadiau.ca
http://tinyurl/dsilver
http://ml3.acadiau.ca
48
Intelligent Information Technology Research Lab, Acadia University, Canada