Announcements
CS Ice Cream Social
•
9/5 3:30-4:30, ECCR 265
•
includes poster session, student group presentations
Concept Learning

Examples
 Word meanings
 Edible foods
glorch
 Abstract structures (e.g., irony)
not
glorch
glorch
not
glorch
Supervised Approach To Concept Learning


Both positive and negative examples provided
Typical models (both in ML and Cog Sci) circa 2000
required both positive and negative examples
Contrast With Human Learning Abiliites


Learning from positive examples only
Learning from a small number of examples
 E.g., word meanings
 E.g., learning appropriate social behavior
 E.g., instruction on some skill

What would it mean to learn from a
small number of positive examples?
+ +
+
Tenenbaum (1999)
Two dimensional continuous feature space

Concepts defined by axis-parallel rectangles

e.g., feature dimensions

 cholesterol level
 insulin level
e.g., concept

healthy
Learning Problem



Given a set of given a set of n examples,
X = {x1, x2, x3, …, xn}, which are instances of the
concept…
Will some unknown example Y also be an
instance of the concept?
Problem of generalization
+ +
1
2
3
+
Hypothesis (Model) Space

H: all rectangles on the plane,
parameterized by (l1, l2, s1, s2)
h: one particular hypothesis

Note: |H| = ∞
Consider all hypotheses in parallel

In contrast to non-Bayesian approach of
maintaining only the best hypothesis
at any point in time.
Prediction Via Model Averaging
Will some unknown input y be in the concept
given examples X = {x1, x2, x3, …, xn}?
Q: y is a positive example of the concept (T,F)

P(Q | X) = ⌠h p(Q & h | X) dh
Marginalization

P(Q & h | X) = p(Q | h, X) p(h | X)
Chain rule

P(Q | h, X) = P(Q | h) = 1 if y is in h
Conditional
independence and
deterministic concepts

p(h | X) ~ P(X | h) p(h)
Bayes rule
likelihood
prior
Priors and Likelihood Functions
Priors, p(h)
 Location invariant
 Uninformative prior
(prior depends only on area of rectangle)
x
 Expected size prior

Likelihood function, p(X|h)
 X = set of n examples
 Size principle
Expected size prior
Generalization Gradients
MIN:
smallest hypothesis consistent with data
weak
Bayes: instead of using size principle, assumes
examples are produced by process independent of the true
class
Dark line =
50% prob.
Experimental Design
Subjects shown n dots on screen that are
“randomly chosen examples from some rectangle
of healthy levels”

 n drawn from {2, 3, 4, 6, 10, 50}

Dots varied in horizontal and vertical range
 r drawn from {.25, .5, 1, 2, 4, 8} units in a 24 unit
window

Task
 draw the ‘true’ rectangle around the dots
Experimental Results
Number Game

Experimenter picks integer arithmetic concept C
 E.g., prime number
 E.g., number between 10 and 20
 E.g., multiple of 5


Experimenter presents positive examples drawn at
random from C, say, in range [1, 100]
Participant asked whether some new test case
belongs in C
Empirical Predictive Distributions
E
x
a
m
p
le
s
1
6
1
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
6
0
1
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
1
6826
41
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
1
62
31
92
01
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
6
08
01
03
01
0
.5
Hypothesis Space
 Even numbers
 Odd numbers
 Squares
 Multiples of n
 Ends in n
 Powers of n
 All numbers
 Intervals [n, m] for n>0, m<101
 Powers of 2, plus 37
 Powers of 2, except for 32
•
•
Observation = 16
Likelihood function
 Size principle
•
Prior
 Intuition
data = 16
even
odd
squares
mult of 3
mult of 4
mult of 5
mult of 6
mult of 7
mult of 8
mult of 9
mult of 10
ends in 1
ends in 2
ends in 3
ends in 4
ends in 5
ends in 6
ends in 7
ends in 8
ends in 9
powers of 2
powers of 3
powers of 4
powers of 5
powers of 6
powers of 7
powers of 8
powers of 9
powers of 10
all
powers of 2 + {37}
powers of 2 − {32}
0
0.1
prior
35
35
30
30
25
25
20
20
15
15
10
10
5
5
0
0.2 0
0.2
lik
0
0.4 0
0.2
post
0.4
Observation =
16 8 2 64
data = 16 8 2 64
•
•
Likelihood function
 Size principle
•
Prior
 Intuition
even
odd
squares
mult of 3
mult of 4
mult of 5
mult of 6
mult of 7
mult of 8
mult of 9
mult of 10
ends in 1
ends in 2
ends in 3
ends in 4
ends in 5
ends in 6
ends in 7
ends in 8
ends in 9
powers of 2
powers of 3
powers of 4
powers of 5
powers of 6
powers of 7
powers of 8
powers of 9
powers of 10
all
powers of 2 + {37}
powers of 2 − {32}
0
0.1
prior
35
35
30
30
25
25
20
20
15
15
10
10
5
5
0
0.2 0
1
lik
0
2 0
−3
x 10
0.5
post
1
Posterior Distribution After Observing 16
E
x
a
m
p
le
s
1
6
Model Vs. Human Data
1
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
6
0
1
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
1
6826
41
0
.5
MODEL
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
1
62
31
92
01
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
6
08
01
03
01
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
6
05
25
75
51
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
HUMAN
DATA
8
12
543
61
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
8
19
88
69
31
0
.5
0
4 81
21
62
02
42
83
23
64
04
44
85
25
66
06
46
87
27
68
08
48
89
29
6
1
0
0
Summary of Tenenbaum (1999)

Method
 Pick prior distribution (includes hypothesis space)
 Pick likelihood function (size principle)
 leads to predictions for generalization as a function of r
(range) and n (number of examples)
Claims people generalize optimally given
assumptions about priors and likelihood
Bayesian approach provides best description of
how people generalize on rectangle task.
Explains how people can learn from a small
number of examples, and only positive examples.
Important Ideas in Bayesian Models
Generative models
 Likelihood function
Consideration of multiple models in parallel
 Potentially infinite model space
Inference
 prediction via model averaging
 role of priors diminishes with amount of evidence
Learning
 trade off between model simplicity and fit to data 
Bayesian Occam’s Razor
Ockham's Razor
medieval philosopher
and monk
tool for cutting
(metaphorical)
If two hypotheses are equally consistent with the data,
prefer the simpler one.
Simplicity
•
•
•
•
can accommodate fewer observations
smoother
fewer parameters
restricts predictions more
(“sharper” predictions)
Examples
1st vs. 4th order polynomial
small rectangle vs. large rectangle
in Tenenbaum model
H1
H0
H0 H1
Motivating Ockham's Razor
PRIORS
Aesthetic considerations

A theory with mathematical beauty is more likely to
be right (or believed) than an ugly one, given that
both fit the same data.
Past empirical success of the principle

Coherent inference, as embodied by Bayesian
reasoning, automatically incorporates
Ockham's razor

Two theories H1 and H2

LIKELIHOODS
Ockham's Razor with Priors
Jeffreys (1939) probabililty text
more complex hypotheses should have lower priors
Requires a numerical rule for assessing
complexity
e.g., number of free parameters
e.g., Vapnik-Chervonenkis (VC) dimension
Subjective vs. Objective Priors
subjective or informative prior
specific, definite information about a random variable
objective or uninformative prior
vague, general information
Philosophical arguments for certain priors as
uninformative
Maximum entropy / least committment
e.g., interval [a b]: uniform
e.g., interval [0, ∞) with mean 1/λ:
exponential distribution
e.g., mean μ and std deviation σ: Gaussian
Independence of measurement scale
e.g., Jeffrey’s prior 1/(θ(1-θ)) for θ in [0,1]
expresses same belief whether we talk
about θ or logθ
Ockham’s Razor Via Likelihoods
Coin flipping example

H1: coin has two heads
H2: coin has a head and a tail
Consider 5 flips producing HHHHH

H1 could produce only this sequence
H2 could produce HHHHH, but also HHHHT, HHHTH,
... TTTTT
P(HHHHH | H1) = 1, P(HHHHH | H2) = 1/32
H2 pays the price of having a lower likelihood
via the fact it can accommodate a greater
range of observations

H1 is more readily rejected by observations

Simple and Complex Hypotheses
H2
H1
Bayes Factor
BIC is approximation to Bayes factor

A.k.a. likelihood ratio

Hypothesis Classes Varying In Complexity
E.g., 1st, 2nd, and 3d order polynomials

Hypothesis class is parameterized by w

v
Rissanen (1976)
Minimum Description Length
Prefer models that can communicate the data
in the smallest number of bits.

The preferred hypothesis H for explaining
data D minimizes:

(1) length of the description of the hypothesis
(2) length of the description of the data with the help
of the chosen theory
L: length
MDL & Bayes
L: some measure of length (complexity)

MDL: prefer hypothesis that min. L(H) + L(D|H)

Bayes rule implies MDL principle

P(H|D) = P(D|H)P(H) / P(D)
–log P(H|D) = –log P(D|H) – log P(H) + log P(D)
= L(D|H) + L(H) + const
Relativity Example
Explain deviation in Mercury's orbit at
perihelion with respect to prevailing theory

E: Einstein's theory
F: fudged Newtonian theory
deviation
α = true deviation
a = observed
Relativity Example (Continued)
Subjective Ockham's razor

result depends on one's belief about P(α|F)
Objective Ockham's razor

for Mercury example, RHS is 15.04
Applies to generic situation
