Local factors in a graphical model

First, a familiar example

Conditional Random Field (CRF) for POS tagging
Possible tagging (i.e., assignment to remaining variables)
…
v
v
v
…
preferred
find
tags
Observed input sentence (shaded)
1
Local factors in a graphical model

First, a familiar example

Conditional Random Field (CRF) for POS tagging
Possible tagging (i.e., assignment to remaining variables)
Another possible tagging
…
v
a
n
…
preferred
find
tags
Observed input sentence (shaded)
2
Local factors in a graphical model

First, a familiar example

Conditional Random Field (CRF) for POS tagging
”Binary” factor
that measures
compatibility of 2
adjacent tags
v
v 0
n 2
a 0
n
2
1
3
a
1
0
1
v
v 0
n 2
a 0
n
2
1
3
a
1
0
1
Model reuses
same parameters
at this position
…
find
…
preferred
tags
3
Local factors in a graphical model

First, a familiar example

Conditional Random Field (CRF) for POS tagging
“Unary” factor evaluates this tag
Its values depend on corresponding word
…
…
v 0.2
n 0.2
a 0
find
preferred
tags
can’t be adj
4
Local factors in a graphical model

First, a familiar example

Conditional Random Field (CRF) for POS tagging
“Unary” factor evaluates this tag
Its values depend on corresponding word
…
…
v 0.2
n 0.2
a 0
find
preferred
tags
(could be made to depend on
entire observed sentence)
5
Local factors in a graphical model

First, a familiar example

Conditional Random Field (CRF) for POS tagging
“Unary” factor evaluates this tag
Different unary factor at each position
…
…
v 0.3
n 0.02
a 0
find
v 0.3
n 0
a 0.1
preferred
v 0.2
n 0.2
a 0
tags
6
Local factors in a graphical model

First, a familiar example

Conditional Random Field (CRF) for POS tagging
p(v a n) is proportional
to the product of all
factors’ values on v a n
…
v
v 0
n 2
a 0
v
a
1
0
1
v
v 0
n 2
a 0
a
v 0.3
n 0.02
a 0
find
n
2
1
3
n
2
1
3
a
1
0
1
…
n
v 0.3
n 0
a 0.1
preferred
v 0.2
n 0.2
a 0
tags
7
Local factors in a graphical model

First, a familiar example

Conditional Random Field (CRF) for POS tagging
p(v a n) is proportional
to the product of all
factors’ values on v a n
…
v
v 0
n 2
a 0
v
a
1
0
1
v
v 0
n 2
a 0
a
v 0.3
n 0.02
a 0
find
n
2
1
3
n
2
1
3
a
1
0
1
= … 1*3*0.3*0.1*0.2 …
…
n
v 0.3
n 0
a 0.1
preferred
v 0.2
n 0.2
a 0
tags
8
Great Ideas in ML: Message Passing
Count the soldiers
there’s
1 of me
1
before
you
2
before
you
3
before
you
4
before
you
5
behind
you
4
behind
you
3
behind
you
2
behind
you
adapted from MacKay (2003) textbook
5
before
you
1
behind
you
9
Great Ideas in ML: Message Passing
Count the soldiers
there’s
1 of me
2
before
you
Belief:
Must be
22 +11 +33 =
6 of us
3
only see
my incoming behind
you
messages
adapted from MacKay (2003) textbook
10
Great Ideas in ML: Message Passing
Count the soldiers
there’s
1 of me
1
before
you
Belief:
Belief:
Must be
Must be
1 1 +11 +44 = 22 +11 +33 =
6 of us
6 of us
4
only see
my incoming behind
you
messages
adapted from MacKay (2003) textbook
11
Great Ideas in ML: Message Passing
Each soldier receives reports from all branches of tree
3 here
7 here
1 of me
11 here
(= 7+3+1)
adapted from MacKay (2003) textbook
12
Great Ideas in ML: Message Passing
Each soldier receives reports from all branches of tree
3 here
7 here
(= 3+3+1)
3 here
adapted from MacKay (2003) textbook
13
Great Ideas in ML: Message Passing
Each soldier receives reports from all branches of tree
11 here
(= 7+3+1)
7 here
3 here
adapted from MacKay (2003) textbook
14
Great Ideas in ML: Message Passing
Each soldier receives reports from all branches of tree
3 here
7 here
3 here
adapted from MacKay (2003) textbook
Belief:
Must be
14 of us
15
Great Ideas in ML: Message Passing
Each soldier receives reports from all branches of tree
3 here
7 here
3 here
adapted from MacKay (2003) textbook
Belief:
Must be
14 of us
16
Great ideas in ML: Forward-Backward

In the CRF, message passing = forward-backward
belief
message
α
…
v
v 0
n 2
a 0
v 7
n 2
a 1
n
2
1
3
α
v 1.8
n 0
a 4.2
av 3
1n 1
0a 6
1
β
message
v 2v
n v1 0
a n7 2
a 0
n
2
1
3
β
a
1
0
1
v 3
n 6
a 1
…
v 0.3
n 0
a 0.1
find
preferred
tags
17
Great ideas in ML: Forward-Backward

Extend CRF to “skip chain” to capture non-local factor

More influences on belief 
α
v 5.4
n 0
a 25.2
β
v 3
n 1
a 6
…
v 3
n 1
a 6
find
v 2
n 1
a 7
…
v 0.3
n 0
a 0.1
preferred
tags
18
Great ideas in ML: Forward-Backward

Extend CRF to “skip chain” to capture non-local factor


More influences on belief 
Red messages not independent?
v 5.4`
Graph becomes loopy 
Pretend they are!
α
n 0
a 25.2`
β
v 3
n 1
a 6
…
v 3
n 1
a 6
find
v 2
n 1
a 7
…
v 0.3
n 0
a 0.1
preferred
tags
19