UNDERSTANDING SCENES WITH SHADOWS
VISION FLASH 21
David L. Wal t z
Massachusetts Institute of Technology
Artificial Intel 1 igence Laboratory
Vision Group
November 1971
ABSTRACT
The basic problem of this research i s t o find methods which will enable a
program to construct a three dimensional interpretation from the line
drawing of a scene, where the scene may have shadows and various degeneracies. These methods differ from those used in earlier related programs
in that they use region information extensively, and include formal isms
for eye and lighting position. The eventual result of t h i s research will
be a program which should be able t o successfully t r e a t scenes with far
fewer restrictions than present programs wi 11 to1 erate.
Work reported herein was conducted a t the Artificial Intelligence Laboratory, a Massachusetts Institute of Te,chnology research program supported
by the Advanced Research Projects Agency of the Department of Defense,
and was monitored by the Office of NavaJ Research under Contract Number
N00014-70-A-0362-0002.
Understanding scenes w i t h shadows;
Introduction. 2
l NTRODUCT l ON
I n t h e body o f t h i s paper,
I deal p r i m a r i l y w i t h t h e
c o n c r e t e r e s u l t s I have o b t a i n e d so f a r ,
but I t h i n k that the
s i g n i f i c a n c e o f t h e r e s u l t s may not be immediately apparent.
Therefore I be1 l e v e t h a t i t w i 11 be u s e f u l t o say something about
how I approached t h i s research,
s o r t t h e y are,
about why t h e r e s u l t s a r e of t h e
and about what relevance t h i s work has t o human
p e r c e p t ion,
Why does It seem so easy f o r us t o apprehend r a p i d l y t h e
t h r e e - d l m e n s l o n a l n a t u r e o f scenes,
even when t h e y a r e presented
t o us I n t h e f o r m o f b a d l y exposed and b a d l y focussed photographs
of b a d l y 1 I t and u n f a m l l t a r o b j e c t s ? While i t i s p o s s i b l e t o
demonstrate a l a r g e number o f optical
illuslons,
I think that the
u s u a l r e l l a b i l l t y o f my p e r c e p t i o n s I s a f a r more s t r i k t n g
phenomenon, Because we can p e r f o r m as r e 1 1 a b l y as we can,
I began
w 1 t h t h e b a s i c assumption t h a t scenes p r o v i d e many redundant
pieces of l n f o r m a t l o n ~ What k i n d s o f i n f o r m a t i o n and r u l e s c o u l d
s e r v e t o e.xpl a i n t h e observed performance?
Some cues and ru1,es had a1 ready been demonstrated b y
Huffman,
Clowes and others.
T h e i r work has shown f i r s t t h a t t h e
s e t s o f edge types and j u n c t i o n types which can occur i n a scene
made up o f o b j e c t s w l th p l a n a r surfaces and t r i hedral j u n c t i o n s
; J n d e r s t a n d l n g scenes w i t h shadows;
i n t r o d u c t ion, 3
can be e a s i l y enu~neratcfl, and t i i e n t h a t we cat1 o n l y i n t e r r ~ r c t s
l i n e as a p a r t i c u l a r t y p e i f we can l a b e l t h e e n t i r e scene f r o m
t h e s e t o f a l l o w a b l e j u n c t i o n s i n a manner c o n s i s t e n t w i t h t h i s
1 i n e assignment.
C l e a r l y t h i s approach cannot p r o v i d e a
s u f f i c i e n t explanation o f v i s u a l perception,
s i n c e i n g e n e r a l any
scene can be l a b e l e d i n s e v e r a l d i f f e r e n t ways,
and t h e r e i s no
o b v i o u s way t o s e l e c t t h e i n t e r p r e t a t i o n we would c a l l c o r r e c t .
i d e a was t o e x t e n d t h e s e t o f a1 l o w a b l e j u n c t i o n s t o
My f i r s t
i n c l u d e shadow j u n c t i o n s ,
and I f o u n d t h a t I n s i m p l e scenes w i t h
shadows we can sometilnes p a r s e u n i q u e l y because shadows i n some
sense p r o v i d e a d d i t i o n a l i n f o r m a t i o n . However i n more complex
(
scenes,
t h e l a r g e r s e t o f l a b e l s m e r e l y compounded t h e problem o f
m u l t i p l e i n t e r p r e t a t i o n s . T h i s l e d me t o t h i n k more about t h e
n a t u r e o f o u r p e r c e p t i o n o f scenes w i t h shadows.
Why does i t seem easy t o i n t e r p r e t scenes r e g a r d l e s s o f
1i g h t i n g ,
shadows,
and i n p a r t i c u l a r ,
how a r e we a b l e t o " f a c t o r o u t "
even i n cases where t h e y a r e t h e most p r o m i n e n t f e a t u r e s
i n terms o f c o n t r a s t ? My s e a r c h f o r r u l e s began w i t h t h e
o b s e r v a t i o n t h a t i f a scene i s 1 i t w i t h a s i n g l e 1 i g h t source we
can p a r t i t i o n t h e s e t o f j u n c t i o n s i n t o t h o s e which can o c c u r
w i t h r e s p e c t t o a p a r t i c u l a r p l a n e and t h o s e w h i c h cannot o c c u r .
T h i s i n t r o d u c e d a new p r o b l e m because we must know t h e
o r i e n t a t i o n o f a s u r f a c e and l i g h t source w i t h r e s p e c t t o t h e
v l e w e r b e f o r e we can say f o r c e r t a i n whether o r n o t p a r t i c u l a r
k.
Understandl ng scenes w f t h shadows;
i n t roduct I o n . 4
j u n c t i o n s can occur.. Nevertheless I had found an i n t e r e s t i n g k i n d
o f i n t e r r e l a t f o n between these v a r i a b l e s i n t h a t i f we know
o r i e n t a t i o n s we can p a r t i t i o n j u n c t i o n s and i f we know the
j u n c t i o n l a b e l s we can t e l l something about surface,
l i g h t and
eye placement. T h i s suggested t h a t I c o u l d use t h i s type o f f a c t
b o t h t o check f o r l a b e l l n g v a l i d i t y ( a l l l a b e l i n g s must be
c o n s i s t e n t w l t h a s i n g l e ' i n t e r p r e t a t i o n of l i g h t and eye
positions) b u t even more important i t provided a p o t e n t i a l a c t i v e
t o o l i f I c o u l d o b t a l n p a r t i a l f n f o r m a t f o n i n a scene. Perhaps
i t p o i n t e d o u t the importance of f l n d t n g
most i m p o r t a n t o f a l l ,
surface orientations;
j u s t as a l i n e l a b e l c o n s t r a i n s t h e
p o s s i b l e l a b e l i n g o f the j u n c t l o n s a t both ends o f t h e l i n e ,
so a
reglon o r i e n t a t i o n l a b e l could constrain the possible junctions
and 1 i n e s whl ch bound it.
As soon as I began t h i n k i n g i n terms o f regions I came up
w l t h some powerful r e s u l t s . The f i r s t r e g i o n I considered was the
i l l u m i n a t e d p o r t i o n o f t h e s u r f a c e (assumed t o be a plane) which
supports a scene.
Along the boundary of t h i s r e g i o n and the
scene,
t h e o n l y 1 lne assignments which are p o s s i b l e are shadow
edges,
concave edges (when t h e scene o b j e c t s r e s t on the surface)
and o b s c u r i n g edges. Therefore we can (1) p a r t i t i o n the set o f
p o s s f b l e j u n c t i o n s i n t o those which can occur along the boundary
and those which cannot,
( 2 ) we can say how these j u n c t i o n s must
be o r i e n t e d i f t h e y occur on t h e boundary (no convex edges can
\
Understand1ng scenes w i t h s.ha?.ow,s; In t r o d u c t ion. 5
occur a l o n g t h e boundary f o r example),
and ( 3 ) we can f u r t h e r
p a r t i t i o n these j u n c t i o n s according t o l i g h t i n g d i r e c t i o n ,
since
they a1 1 appear i n r e l a t i o n t o t h e same surface.
Once I enbumerated these j u n c t i o n s another f a c t emerged:
c e r t a i n j u n c t i o n s on t h e scene/background boundary rnust always be
p a r t i a l l y l a b e l e d i n a unique manner.
What t h i s means I s t h a t
c e r t a i n l o c a l f e a t u r e s always have a unique meaning I n a
p a r t i c u l a r context.
I have s i n c e been a b l e t o f i n d a number o f
s i m i l a r examples which occur i n o t h e r contexts. These
o b s e r v a t i o n s have a1 lowed me t o f i n d r u l e s which enable us t o
p a r t i a l 1 y l a b e l v i r t u a l l y any scene;
g i v e n a , p a r t i a l l a b e l ing,
can o f t e n f i n d addl t i o n a l " f o r c e d choice" labels,
t o determine l l g h t l n g position,.
we
use a1 1 t h e s e
and.use t h i s i n - t u r n t o .do
further forced labeling.
As I show In t h e paper, we can a l s o f o r m a l i z e l a b e l s f o r
r e g i o n o r i e n t a t i o n and I 1 1umina.t i o n and l o o k .for scene cues t o
l a b e l these j u s t as we do l i n e s . Some o f t h e cues i n c l u d e l i n e
slopes,
j u n c t i o n types,
and placement I n t h e scene (e.g.
scene/background boundary o r i n t h e . i n t e r i o r ) .
on t h e
I have a l s o
developed some o f t h e r u l e s which r e l a t e 1 i.ne l a b e l in^, j u n c t i o n
labeling,
and r e g i o n 1abel.lng t o each o t h e r and t o themselves as
w e l l as t o l i g h t i n g and eye p o s i t i o n s .
To i l l u s t r a t e t h e i n h e r e n t redundancy o f t h e scene features,
t h e types o f i n f o r m a t l o n I have 1tste.d a r e sufficient t o s o l v e a
Understanding scene's w i t h shadows;
number o f shadow scenes,
introduction. 6
y e t I have n o t even mentioned one o f t h e
most obvious f e a t u r e s o f shadows- t h e i r darkness. A t one time I
had f e l t t h a t t h i s was perhaps t h e most i m p o r t a n t shadow cue,
and
t h a t i t was a l s o p o s s i b l e t h a t we were a b l e t o d e t e c t a
p a r t i c u l a r k i n d o f m i c r o s t r u c t u r e i n shadow edges which we c o u l d
use t o distinguish them frorn o t h e r edges. C l e a r l y darkness and
edge t y p e a r e n o t enough t o alone i d e n t i f y shadows,
n o t o f t e n confuse p a i n t e d areas w i t h shadows.
s i n c e we do
I n order t o
r e c o g n i z e shadows we need t o know about t h e cons i s t e n t
r e l a t i o n s h i p s which shadows have t o l i g h t i n g and t o t h e o b j e c t s
whl ch l'causel'
Finally,
t h e shadows.
a l t h o u g h I d i d n o t s e t o u t t o s o l v e t h e problem o f
degeneracies i n scene$ I have been a b l e t o make some progress
toward hand1 i n g them. Roughly,
degeneracies a r e p o i n t s a t whi ch
separated v e r t i c e s and l i n e s appear t o be p a r t o f the same
vertex,
so t h a t t h e r e s u l t a n t v e r t e x i s n o t i n t h e a1 lowable s e t
o f junctions,
o r where 1 i g h t i n g i s such t h a t shadow 1 ines are
p r o j e c t e d on j u n c t i o n s o r o t h e r w i s e l i n e up w i t h edges.
c o u l d move t h e 1 i g h t source and eye s l i g h t l y ,
I f we
these degeneracies
would disappear (though we m i g h t form new ones). However I have
been a b l e t o f i n d some methods o f decomposing o r o t h e r w i s e
u n d e r s t a n d i n g such j u n c t i o n s w i t h o u t resorting t o maklng physlcal
changes,
l a r g e l y as a byproduct o f t h e use o f a l a r g e r number c f
scene cues and b e t t e r general r u l e s .
Understanding scenes w i t h shadows;
Rather than conslder any more d e t a l l s ,
introduction. 7
I would now l i k e t o
r e t u r n t o t h e general problem, What e x . a c t l y does t h i s research
add t o our understandlng o f t h e general probletns o'f v i s i o n
programs and hurnan v i s i o n ?
F i r s t I have shown t h a t we can. f o r m a l i z e and make good use
o f scene i n f o r m a t i o n o t h e r than l i n e s and j u n c t i o n s .
Next,
a l t h o u g h I have c h o s e n . t o use o n l y t h a t i n f o r m a t i o n
which I know how t o e x t r a c t f rorn scenes,
I have provided a f a r
more general f rarnework than . e x i s t e d . before.
For example,
a1 ready
having a f o r m a l i s m f o r r e g i o n o r i e n t . a t i o n a n d . l i g h t l n g
i n f o r m a t i o n a l l o w s us t o add c o l o r s o r textur.es o f r e g i o n s as
properties.
I have a1 so shown t h a t when we a s s i g n values t o scene
v a r i a b l e s we can put e x p l l c i t condi.tions on o t h e r scene
v a r i a b l e s . T h i s leads t o a g r e a t e r understanding o f t h e a c t i v e
n a t u r e o f v i s u a l processing, which we can express i n the form o f
theorems i n a program, T h i s approach has several advantages over
p r e v i o u s l a b e l i n g procedures.
(1) We can check a t each s t e p t o
see whether i t i s ' reasonable t o proceed i n a p a r t i c u l a r manner,
and do n o t need t o d e f e r c o n f i r i n a t i o n o r c o n t r a d i c t i o n u n t i l
l a t e r i n t h e o p e r a t i o n o f t h e program.
conditions which must be s a t i s f i e d ,
p a r s i n g s f o r each scene;
( 2 ) We have more
so t h a t we should get fewer
i n o r d e r t o o b t a i n these,
we a l s o do n o t
have t o s t a r t w i t h a completely u n l a b e l e d 1 i n e drawing f o r every
Understanding scenes w i t h shadows;
Introduction. 8
p a r s i n g , ( 3 ) We can handle degeneracies more e a s i l y s i n c e we do
n o t have. t o depend on t h e s i n g l e o b s e r v a t i o n t h a t a l l t h e l i n e
l a b e l s must match.
I n a1 1 t h e s e senses,
I have made a s t a r t toward
u n d e r s t a n d i n g - v i s i o n i n a way which w i l l a l l o w me t o w r i t e a
program u s i n g procedures as. i n Winograd's program r a t h e r than
b l i n d t r e e search techniques. L i k e Wlnograd I have t r l e d t o make
t h e meaning of v a r i o u s scene f e a t u r e s c l e a r I n terms o f t h e
consequences o f i n t e r p r e t i n g them i n each possi b l e manner.
F l n a l l y I be1 I e v e t h a t my research w i l l p r o v i d e a more s o l i d
bas1 s f o r u n d e r s t a n d i n g va.rious phenomena i n human v i st on, Whll e
I would c e r t a i n l y n o t contend t h a t t h e processes i n human v i s i o n
a r e 1 ike t h e .ones I descr ibe,
we shout-d a t l e a s t 'be a b l e t o g a i n
a b e t t e r I d e a o.f t h e necessary comp1exit.y o f a t h e o r y t o e x p l a i n
human .vi s.lon,
and some perspect l v e on what k.lnds o f l n f o r r n a t l o n
such a t h e o r y must c o n t a i n .
Understandl ng Scenes W i t h Shadows. Page 2
I. Ground r u l e s
I w i l l assume t h a t I am g i v e n o n l y t h e f o l l o w i n g i n f o r m a t i o n
which comprises a l i n e drawing o f a scene:
1. Lines.
Lines a r e j o i n e d c o l l e c t i o n s o f f e a t u r e p o i n t s
d e f i n e d t o be on t h e r e t i n a , where a f e a t u r e p o i n t I s a r e t i n a l
p o i n t a t which the spacial d e r i v a t i v e o f l i g h t i n t e n s i t y i s
I w i l l assume t h a t a l i n e
g r e a t e r than a c e r t a i n threshold.
drawlno has o n l y s t r a ' i g h t l i n e s ,
and t h a t we are g i v e n no
lnformat i o n about 'which 1 i.nes are 1 1 k e l y t o be cracks,
shadows,
o r edges by t h e 1 i n e - f l n d i n g programs.
2. .Junctions. P o i n t s on t h e r e t i n a where two o r more l i n e s
intersect.
3.
I n t e n s i t i e s . R e l a t i v e l i g h t i n c i d e n t on any p o i n t on the
r e t lna.
4. Regions, Areas o f t h e r e t l n a bounded by l i n e s , The set o f
a l l regions I n t h e l i n e .drawing f i l l s t h e r e t i n a .
T h i s i n f o r m a t i o n can be coded i n t o t h e f o l l o w i n g form which
I w i l l assume I s d i r e c t l y a c c e s s i b l e t o the program.
1. For each j u n c t i o n ,
a l l t h e j u n c t i o n s t h a t a r e connected
t o i t by s i n g l e l i n e segments.
2.
For each junction, a type,
such as e l 1,
I w i 11 have more t o say about these l a t e r .
fork,
arrow,
etc.
Understan.dlng Scenes W i t h Shadows. Page 3
3.
For each j u n c t l o n ,
a l l the r e g i o n s surround.ing i t .
4.
For each j u n c t i o n ,
i t s coordinates.
5.
For each region,
6.
For t h e background region,
i t s Intensity.
a s p e c i a l name t o d i s t i . n g u i s h
i t from a l l o t h e r r e g i o n s I n t h e l l n e drawing.
The o b j e c t o f my work i s t o a r r i v e a t a s i n g l e parsing,
or
assignment o f l i n e l a b e l s t o l i n e segments i n t h e l i n e drawing.
There are seven l l n e l a b e l s : concave,
convex,
crack,
I w i l l a l l o w the
o r i e n t a t i o n s each f o r obscure and shadow.
scene (which corresponds t o t h e 1 i n e drawfng)
f r o m a s i n g l e l i g h t source,
t o have shadows
coincidences o f l l n e segments and
j u n c t i o n s f r o m t h e p o l n t o f view of t h e eye,
have faces w i t h a r b i t r a r y r e f l e c t i v i t y .
e s s e n t i a l l y ambiguous;
and o b j e c t s whlch
Some scenes a r e
b y t h i s I mean t h a t humans can a r r i v e a t
more than one i n t e r p r e t a t i o n f o r such a scene.
goal
p l u s two
w i l l be t o f i n d these i n t e r p r e t a t i o n s .
I n these cases
my
These a r e
c o n s i d e r a b l y more ambi t ious goal s than have been a t t e r l p t e d
before,
b u t I b e l i e v e t h a t t h e r e i s s u f f i c i e n t i n f o r v a t i o n i n the
r e p r e s e n t a t i o n s t h a t I have d e s c r i b e d t o come c l o s e t o t h e s e
goal s.
I
Understandlng Scenes Wlth Shadows.
Page 4
I n t h e s e c t l o n s which f o l l o w I w l l l f i r s t discuss t h e
programs which have been suggested t o s o l v e slrnl l a r problems,
t h e i r l i m l t a t l o n s and then new methods f o r overcoming t h e s e
1 iml tations.
Understanding Scenes W f t h Shadows. Page 5
I I. Huffman,
Clowes,
and Dowson's Programs
The programs w r i t t e n by o r suggested by these men a1 1 share
t h e f o l l o w i n g assumptlons which a r e b u i l t i n t o the methods:
1. Every j u n c t i o n which can appear i n a l i n e drawing under
p a r t i c u l a r assumptions (such as o n l y t r i h e d r a l j u n c t i o n s ) i s
listed;
each j u n c t i o n which occurs I n a l i n e drawfng Is assumed
t o be i n t h i s
1is.t.
2. The o n l y c o n d l t l o n s f o r 1;be.llng
a l i n e are a
' t h a t the
j u n c t t o n s which r e s u l t be from the a1 l d w a b l e ' s e t and (b) t h a t a1 1
j u n c t i o n s i n the 1 i n e draw'ing can be laba'led ' I n a manner which i s
c o n s i s t e n t w i t h t h i s l a b e l f o r t h e 1.ine.
3.. Thus these programs r e q u i r e o n l y t'hat 1 l n e l a b e l s agree
and p u t no e x p l i c i t c o n d i t i o n s a t a l l on regions.
I n general each program o f t h i s type w i l l produce several
parsings., and indeed,
as long as a l l j u n c t i o n s i n t h e l i n e
drawing appear i n the s e t o f allowab'le j u n c t i o n s and as l o n g as
t h e r e a r e no degeneracies,
t h e p a r s i n g we would c a l l llcorrectlt
w i l l be among those produced.
However, t h i s s t i l l leaves us w i t h
t h e problem o f f l n d l n g the llcorrectll
parsing
from among those
U.nderstanding Scenes W i t h Shadows. Page 6
produced.
Even more s e r i o u s l y ,
these programs
produce a c o r r e c t
p a r s i n g o n l y i f t h e needed j u n c t i o n s a r e i n t h e a l l o w a h l e . s e t .
There i s no way t o handle degenerate j u n c t i o n s except t o l i s t
them i n t h e a l l o w a b l e s e t ,
I n general d o i n g t h i s w l l l lead t o
many more p a r s i n g s from which t o s e l e c t t h e c o r r e c t one,
t h e r e w i l l be more p o s s i b i l i t i e s f o r e a c h j u n c t i o n type.
FORK j u n c t i o n s i n s t e a d o f 10,)
since
(i.e.
20
An obvious method one might
suggest f o r t h i s problem would then be t o somehow o r d e r t h e
j u n c t i o n types so t h a t t h e ones most 1 i k e l y t o be encountered
would be t r i e d f i r s t .
Then we c o u l d suggest as the c o r r e c t
p a r s i n g Che one which had t h e n no st 1 ikely." s e t of c o n s i s t e n t
junctions.
The problem w i t h t h i s approach i s t h a t t h e r e l a t i v e
l i k e l i h o o d o f v a r i o u s j u n c t i o n s depends on l i g h t i n g and eye
position,
on p o s i t i o n and o r . i e n t a t i o n w i t h i n t h e l i n e drawing,
and o t h e r f a c t o r s ;
i n many i n s t a n c e s a j u n c t i o n l a b e l i n g which I s
e x t r e m e l y l i k e l y I n one c o n t e x t w i l l be impossfble i n anoth-er
context,
I n t h e methods I suggest,
I w i l l t r y a t each p o i n t t o make
t h e c o n t e x t o f each l i n e segment, ,junction,
drawing as e x p l i c l t as possfble,
region,
and l i n e
and l a b e l 1 l n e segments and
r e g i o n s o n t h e b a s i s of t h i s c o n t e x t i n f o r m a t i o n . By c o n t e x t I
mean a l l t h e p a r t i a l l n f o r r n a t i o n e x t r a c t e d up t o t h e p a r t i c u l a r
p o i n t i n t h e program. As w f 11 be shown,
p a r t l a 1 l a b e l fngs o f
Understanding Scenes W i t h Shadows. Page 7
l i n e s o r regions,
eye p o s i t i o n ,
l i g h t position,
t h e types of o b j e c t s i n t h e scene,
knowledge about
o r i e n t a t i o n o f l i n e s ahd
j u n c t i o n types adjacent t o a j u n c t i o n can a l l f u n c t i o n as c o n t e x t
i n f o r m a t i o n . The b u l k o f t h i s paper i s devoted t o showing ways i n
which c o n t e x t a f f e c t s t h e i n t e r p r e t a t i o n of l o c a l f e a t u r e s i n a
1 i n e drawing,
and t o showing ways t o o b t a i n c o n t e x t inforrnat i o n
from l o c a l f e a t u r e s .
I f t h e c o n t e x t I s u n c e r t a i n ( a s a t t h e s t a r t o f t-he
program) I w i l 1 , d e m o n s t r a t e t h a t we can make i n i t i a l hypotheses
and then work on t h e b a s i s o f these.
L a t e r I w i l l show how t o
generate hypotheses as we proceed through t h e program. The
r e s u l t s of t h i s approach a r e methods which a l l o w a l a r g e p o r t i o n
of t h e l a b e l ing t o be done w i t h 1 it t l e o r no back-up,
greater
i n s i g h t i n t o t h e i n t e r a c t i o n s o f d i f f e r e n t l e v e l s o f information,
and programs w h i c h ( I hope t o convince you) w i l l be a b l e t o
s u c c e s s f u l l y f t n d t h e " c o r r e c t parsings'l f o r scenes
under t h e
limitations I have l i s t e d ,
Bot h Huffman and Clowes t r e a t a t some l e n g t h an a l t e r n a t e
scene representatton,
c a l l e d t h e dual-graph o f a scene.
I n dual
space each r e g l o n maps i n t o a p o i n t i n such a way t h a t p a r a l l e l
surfaces map i n t o t h e same p o i n t . For a treatment o f t h i s
representation,
r e f e r t o t h e i r papers. T h e i r f e e l i n g seems t o be
r
Understanding Scenes With Shadows. Page 8
t h a t i t would be u s e f u l t o develop t h e dual r e p r e s e n t a t i o n and
t h e normal r e p r e s e n t a t i o n of. a scene i n para1 l e l ,
whereas ' , I f e e l
t h a t i t i s more n a t u r a l t o c a s t dual-space ideas and r e s u l t s i n t o
a form t h a t can be u.sed i n the normal r e p r e s e n t a t i o n .
However I
am s t i 1 1 open on t h e matt-er and n o t prepared t o defend my view.
F i g u r e 1 1 i s t s a1 1 t h e p o s s i b l e t r i h e d r a l ,
non-degenerate
j u n c t i o n s c l a s s i f i e d by j u n c t i o n t y p e and w l t h l n each t y p e by
l i n e l a b e l s . These j u n c t i o n s were o b t a i n e d b y methods dlscusscd
i n Dowson and Waltz.
Understanding Scenes With Shadows. Page 1 3
I I I. General plan f o r a progratn
Basically,
t h e methods I w i l l d e s c r i b e depend on t h e
f o l l o w i n g observat ions:
1. We can generate a complete s e t o f non-degenerate j u n c t i o n
l a b e l ings under p a r t ic u l a r a-ssumptions.
2. For each j u n c t l o n type,
we can a l s o s p e c i f y a f i n i t e s e t
of r e g i o n l a b e l i n g s which g i v e values f o r o r i e n t a t i o n and
illurninatlon,
3.
one and o n l y one o f which must always be t r u e .
J u s t as two j u n c t i o n s j o i n e d by a l i n e segment must be
l a b e l e d so t h a t t h e l i n e connecting them has a s i n g l e value,
so
must each j u n c t i o n around a r e g i o n be l a b e l e d so t h a t t h e r e g i o n
has a s i n g l e o r l e n t a t i o n and i l l u m i n a t i o n value.
4. Whenever we assign a r e g i o n a p a r t i c u q a r value we a l s o
p l a c e e x p l i c i t c o n s t r a i n t s on the p o s s i b l e p o s i t i o n o f the eye
and l i g h t source,
5. A l l reglons and j u n c t i o n s w i t h i n a scene must agree w i t h
a s i n g l e value f o r l i g h t source p o s i t i o n and a s i n g l e value f o r
eye p o s i t i o n .
6.
Junctions which a r e degenerate because of l i g h t source
~ o s i t i o n l n g( u s u a l l y when t h e l i g h t source l i e s I n t h e plane o f a
surface) can on1 y occur under
very r e s t r i c t i v e c o n d i t ions on t h e
regions surrounding t h e j u n c t l o n and on t h e l i g h t and eye;
Under s t and ing Scenes W 1 t h Shadows. Page 1 5
t h e r e f o r e these j u n c t i o n s can be i n c l u d e d i n t h e complete s e t
w i t h o u t danger o f causing l a r g e numbers o f s p u r i o u s parsings.
F i g u r e 2 shows t h i s s e t o f j u n c t i o n s .
7. We can i d e n t i f y o t h e r degenerate j u n c t f ons as those which
e i t h e r ( a ) do n o t have t h e form o f a known j u n c t i o n o r ( b ) cannot
be i n t e r p r e t e d as members o f the s e t o f a l l o w a b l e j u n c t l o n s i n a
manner c o n s i s t e n t w i t h t h e r e s t o f t h e scene.
8. While i t I s i m p r a c t i c a l t o l i s t a l l these o t h e r types o f
degenerate j u n c t i o n s which can occur,
as l o n g as these j u n c t i o n s
comprise o n l y a small percentage o f t h e t o t a l number o f j u n c t i o n s
i n a scene we w i l l g e n e r a l l y be a b l e t o o b t a i n a t l e a s t a p a r t i a l
l a b e l i n g imposed by j u n c t i o n s and r e g i o n s surrounding them,
On
t h e b a s i s of p a r t i a l l a b e l i n g and knowledge about t h e ways t h a t
degeneracies can occur, we can suggest ways o f s e p a r a t i n g t h e
t r u e j u n c t i o n components.
9. We can use l i n e o r i e n t a t i o n s t o suggest r e g i o n and
j u n c t i o n l a b e l s , Given t h r e e unknown v a r i a b l e s ,
name1 y t h e r e g l o n
typ-es on b o t h s i d e s o f a l i n e segment and t h e l i n e segment l a b e l ,
p l u s one known,
the 1 l n e o r i e n t a t i o n ,
we can t e l l v e r y 1 It . t l e .
However Sf any of t h e t h r e e v a r i a b l e s have known values,
be a b l e t o s o l v e f o r t h e o t h e r two,
we may
and i f two o f t h e t h r e e a r e
known we u s u a l l y w i l l be a b l e t o deduce t h e t h i r d .
I n e i t h e r case
we can a t l e a s t l i m i t t h e p o s s i b i l i t i e s .
10. Region i n t e n s i t y i s n o t always a v e r y good i n d i c a t o r o f
Understanding Scenes With Shadows, Page 1 6
r e g i o n type,
s i n c e o b j e c t s can have a r b i t r a r y r e f l e c t i v i t y . We
can however say t h a t whenever we l a b e l a l i n e as a shadow 1 ine,
t h e r e g i o n on t h e s i d e we have l a b e l e d as a shadow must be darker
t h a n t h e o t h e r s i d e of t h e l i n e . S i m i l a r l y ,
I f two regions
separated b y a 1 tne a r e approxtrnatel y equal i n Intensl t y ,
It i s
u n l i k e l y t h a t t h e l i n e s e p a r a t i n g them i s a shadow l i n e .
In
a d d i t i o n we can say t h a t t h e d a r k e s t r e g i o n s are l i k e l y t o be
shadow regions,
whereas t h e b r i g h t e s t ones a r e extremely u n l i k e l y
t o be shadow regions.
Understanding Scenes With Shadows. Page 1 7
IV.
I V.A.
Region n o t a t i o n
. J u n c t i o n segment n o t a t i o n
As w i l l become obvious l a t e r ,
we w i l l have use f o r a
n o t a t i o n which a l l o w s us t o name r e g i o n s around a j u n c t i o n .
convent i o n s shown i n F i g u r e 3,
By
we can code a r e g i o n i n t h e form
o f a c i r c u l a r l i s t o f sequenti.al j u n c t i o n s ,
l i s t e d by c o n v e n t i o n
i n a c l o c k w i s e d i r e c t i o n when viewed froin i n s i d e t h e r e g i o n .
1V.B.
Illumination
Definition:
Causing j u n c t i o n s a r e ones i n which (1) one o f t h e
l i n e s o f t h e j u n c t i o n i s a shadow l i n e and ( 2 ) t h i s shadow l i n e
i s t h e p r o j e c t i o n o f one o f t h e o t h e r l i n e s i n t h e f u n c t i o n .
A r e g i o n may be one o f t h e f o l l o w i n g typesr
I = illuminated
S = shadowed;
i f ' t y p e S t h e n i t must be e i t h e r
SAP a p r o j e c t e d shadow r e g i o n w i t h one o r more "causing
j u n c t i o n s " an i t s boundary.
SB= a p r o j e c t e d shadow r e g i o n w i t h no I1causfng j u n c t i o n s " cil
i t s boundary whicli cause i t . ( A t y p e S!3 r e g i o n can be bounded b y
causing j u n c t i o n s which obscure i t . See f i g u r e 4 A f o r t h e
d'ist inction.
Understand1 ng Scene-s W i t h Shadows, Page 2 1
SAB = a p r o j e c t e d shadow r e g i o n formed b y an o v e r l a p p i n g of
a t y p e . S A and a t y p e SB shadow r e g i o n .
a r e g i o n which i s shadowed because i t i s o r i e n t e d away
SC
f r o m t h e l i g h t source.
The background r e g i o n i s d e f i n e d as t h a t p o r t i o n o f t h e
s u p p o r t s u r f a c e which i s i 1 l u m i n a t e d and which shares a boundary
w i t h t h e edge o f t h e r e t i n a . The s u p p o r t s u r f a c e I s t h e u n i o n o f
t h e background reglon,
any I11 umlnated r e g i o n s o f t h e s u p p d r t
s u r f a c e n o t connected t o t h e background reg.ion, and shadows
p r o j e c t e d o n t o t h e s u p p o r t plane.
Th'e s'upport 'surface can thus
have r e g i o n s which a r e l a b e l e d i n any manner except as t y p e S C o r
t y p e SAB.
1V.C.
Some f a c t s about r e g i o n s and' i l l ' u m i n a t i o n
Definition:
A shadow causi,ng edge i s one which c a s t s a
shadow.
I w i l l appeal t o your knowledge o f t h e w o r l d t o s t a t e t h e
f o l l o w i n g f a c t s w i t h o u t p r o o f o r w i t h mln.lma1 p r o o f .
1. A shadow c a u s i n g edge must always be c6nvex.
2.
Every convex edge must;always be l a b e l e d as "+" o r
obscure.
3.
Every'
11+"
o r o b s c u r i n g Ii n e segment i s convex.
Understanding Scenes .With Shadows. Page 2 2
4.
Every shadow c a u s i n g edge has a t y p e I r e g i o n on t h e s i d e
o r i e n t e d toward t h e 1 l g h t source and a t y p e S C regFon on t h e
o t h e r side.
Proof: Because an edge I l l u m i n a t e d on b o t h s t d e s cannot
cause a shadow,
nor can an edge shadowed on b o t h s i d e s cause one.
I f t h e shadowed s i d e I s any o t h e r t y p e than t y p e SC,
t h e n the
shadow i s due t o t h e p r o j e c t i o n o f a n o t h e r o b j e c t where t h e
p r o j e c t e d shadow c o i n c i d e s e x a c t l y w i t h t h e edge i t s e l f ;
b u t then
i f t h e o b j e c t c a u s l n g t h e shadow were removed, b o t h s i d e s o f t h e
edge would b e illuminated, so t h e edge cannot c a s t a shadow.
Definition:
A r e g i o n bounded by two edges o f a j u n c t i o n
which obscure i t has no necessary r e l a t i o n t o e i t h e r of i t s
a d j o i n i n g regions;
I w i l l c a l l such a r e g i o n an obscured r e g i o n
w i t h r e s p e c t t o such a j u n c t i o n .
5.(a)
, then
I f a shadow causing edge i s l a b e l e d
It+',,
o f t h e edge i s t y p e I and t h e o t h e r i s t y p e SC.
c a u s i n g edge i s l a b e l e d as an o b s c u r i n g edge,
a shadow c a u s l n g j u n c t i o n (1.e.
one s i d e
I f t h e shadow
and t h e j u n c t i o n i s
t h e shadow I s v i s i b l e ) then ( b )
one s i d e o f t h e edge I s l a b e l e d t y p e S A and t h e o t h e r i s t y p e I,
o r e l s e ( c ) one s i d e I s l a b e l e d t y p e S C and t h e o t h e r i s l a b e l e d
I, except when t h e r e g i o n a d j a c e n t t o t h e t y p e S C r e g i o n I s a
r e g i o n obscured w i t h r e p e c t t o t h e shadow causlng j u n c t i o n .
Proof:
(a) i s t r i v i a l .
I n ( b ) and ( c ) I f we can see t h e edge
t h e n we can see e i t h e r t h e t y p e I r e g i o n ( b ) o r t h e t y p e S C
Understanding Scenes W i t h Shadows. Page 2 3
region (c).
I f we can see t h e t y p e I r e g i o n and we can see t h e
t y p e S A shadow region,
then these two r e g i o n s must a d j o i n one
another a t t h e j u n c t i o n - what c o u l d be between thern? I f we can
see t h e t y p e SC region,
and t h e j u n c t i o n i t s e l f ,
then the
a d j a c e n t r e g i o n must be I o r e l s e t h e shadow causing edge c o u l d
n o t c a s t a shadow a t t h e j u n c t i o n ,
except i n t h e case where t h e
a d j a c e n t r e g i o n i s an obscured region,
w i t h respect t o t h e shadow
causing j u n c t ion.
6,
A t y p e S C r e g i o n can have no shadow edges bounding i t .
7,
I n non-degenerate cases a r e g i o n X separated from a
r e g i o n Y by a c r a c k o r concave edge i s t y p e I i f f r e g i o n Y i s
t y p e I; o t h e r w i s e b o t h a r e t y p e S .
Proof: The o n l y way t h i s could n o t be t r u e i s i f a shadow
edge c o i n c i d e d e x a c t l y w i t h t h e concave o r c r a c k edge.
i f we can i d e n t i f y a j u n c t i o n as a shadow causing
8.
junction,
then we can u n i q u e l y l a b e l a l l o f I t s r e g i o n s f o r
I 1 l u m i n a t t o n except those which a r e obscured w i t h respect t o t h e
junction,
i f any,
Proof: Consider t h e s e t o f a l l shadow c a u s i n g j u n c t i o n s
( f i g u r e 4B).
We can always by d e f i n i t i o n l a b e l the Pegions
bounded b y t h e shadow l i n e as types SA on one s i d e and I on t h e
other.
edge,
I f t h e SA r e g i o n I s bounded on t h e o t h e r s i d e by a concave
l a b e l t h e n e x t r e g i o n S.C, .ofherwise l a b e l i t t y p e I (us1 ng
f a c t 5 above),
u n l e s s t h a t r e g i o n i s obscured w i t h respect t o t h e
Understanding Scenes W l t h Shadows. Page 25
junction.
Using f a c t 7,
we can then l a b e l a l l t h e o t h e r regions.
Exact r e l a t i o n s between eye,
1V.D.
light,
and scene;
v i s i b i l i t y and i l l u m i n a t i o n of surfaces
As a f i r s t approximation,
consider the l i g h t source and eye
t o be i n f i n i t e l y d i s t a n t from t h e scene,
supported by a p l a n e w i t h no holes,
t h e scene t o be
and t h e 1 l g h t source t o be a t
an e l e v a t i o n g r e a t enough .sat h a t a1 1 shadaws c a s t i n the, scene
a r e e n t i r e l y surrounded by i l l u m i n a t e d backgound on t h e r e t i n a l
p r o j e c t i o n of t h e scene.
Def i n e a c o o r d i n a t e systern C w i t h respect t o t h e eye.
a r i g h t hand c o o r d i n a t e system,
scene,
C is
o r j g i n i n t h e center o f the
o r i e n t e d w i t h I t s x and z axes on t h e s u p p o r t p l a n e and y
a x l s normal t o t h e support plane.
The z a x i s coincides w i t h t h e
p r o j e c t i o n of a l i n e from t h e o r i g i n t o t h e eye o n t o t h e support
plane,
o r i n o t h e r words t h e eye 4 s i n t h e y-z p l a n e I n t h e
quadrant between t h e +z and + y axes. The eye e l e v a t i o n ,
& , is
defined as t h e a n g l e between t h e +z a x i s and t h e l i n e from t h e
o r i g i n t o t h e eye. Thus
(PE
i s between 0 and 90 degrees.
Understanding Scenes With Shadows, Page 2 6
The normal t o any surface I n t h e scene can be defined I n
#S
t h i s c o o r d i n a t e system by two angles,
and e d m
9%
i s t h e a n g l e between t h e normal t o a s u r f a c e and t h e p r o j e c t i o n
$
of t h e normal o n t o t h e support plane.
i s t h e angle
between t h e p r o j e c t i o n of t h e normal o n t o t h e support p l a n e and
t h e z axis,
measured i n a counterclockwise d i r e c t i o n when viewed
from the +y axis.
Using these d e f i n i t i o n s ,
the v e c t o r from t h e o r i g i n t o the
eye i n C i s
*
VE=
C
+
COS
9'
The normal v e c t o r t o a s u r f a c e f s
3
vS
=
sin
escos +s
A
x
+ sin
cb, .(: +
cos
$ COs #+s 2.
The c o n d i t i o n which must, then. be s a t i s f i e d I n o r d e r t h a t a
s u r f a c e be v l s l b l e I s
i
Understanding Scenes W i t h Shadows. Page 2 7
we can def i n e a c o o r d i n a t e systern C',
Simi larly,
1 i g h t source l i e s on t h e yl-z'
p l a n e a t an e l e v a t f o n
where t h e
+L
The s u r f a c e o r i e n t a t i o n s a r e g i v e n i n t h l s c o o r d i n a t e system b y
$2
0;.
and
The c o n d i t i o n s f o r v i s i b i l i t y i n C a r e t h e n
analogous t o t h e c o n d i t i o n s f o r i l l u m i n a t i o n i n t h i s c o o r d i n a t e
Therefore l e t
system C ' .
E'
= sin
$:
sin
4'
I
+
cos
cos
4: '6
cos
then
i f E1
>
0,
the surface*i s illuminated
i f E' = 0, t h e n t h e 1 i g h t source 1 i e s i n t h e surface plane,
1.e.
if El
<
t h e s u r f a c e has g l a n c i n g ~ I l u r n I n a t i o n
0,
t h e n t h e s u r f a c e I s shadowed,
and i t s edges c a s t
shadows,
We can t h e n d e f i n e a t r a n s f o r m a t i , o n from C '
i s r e l a t e d t o C b y t h e a n g l e %L.
a x i s o f C t o the z axis o f C',
t o . C,
where C '
e E k l s t h e a n g l e from t h e
r
o r ~ e q u i ~ v a l ' ~ n t l 'tyh, e a n g l e between
t h e p r o j e c t i o n s o f t h e e y e - o r i g i n and l i g h t - o r i g i n l i n e s o n t o t h e
support surface.
I n C,
t h e .Il g h t source normal i s
Understanding Scenes W i t h Shadows. Page 2 8
-3
VL
A
= cos b ' s i n
eL
fx
A
A
+
s.in +L
Y + COS +'COS
Q'Z
so t h e c o n d i t i o n f o r s u r f a c e - i l l u m i . n a t i o n i s
b
OR ( c o s #tL
3
s i n egLsin & c o s
4s
+ sin
$s
sin$L
I n Appendix 1, I show haw we can f i n d a l l these angles under
c e r t a i n assumptions.
1V.E.
Quantized r e g i o n o r i e n t a t i o n
Rather t h a n use t-he f u l l
treatment. developed i n t h e previous
s e c t i o n and App.endix 1, I w - i l l
angles .defined t h e r e .
o f t e n be d i f f i c u l t
I n t r o d u c e a quarvtization o f t h e
One reason f o r doing so i s t h a t i t w i 11
t o f i n d - t h e eye e l e v a t i o n i n cases where t h e r e
a r e o b j e c t s w i t h o u t r i g h t angles,
and another i s t h a t t h e r e i s no
n e c e s s i t y t o b e t h i s prec1s.e f o r t h e t i m e being.
It is
I n t e r e s t i . n g t h a t even a v e r y "rough q u e n t i t a t i o n prov1.des a
u n i f o r m and' e f f e c t i v e .method f o r e l i m i n a t i n g p a r s i n g s whlkh a r e
p h y s i c a l l y tmpossi ble,
r e q u i r e d t o match.
b u t a1 lowabl e when on.1y 1 i n e 1abel.s a r e
Understanding Scenes With Shadows. Page 29
Two r e g i o n s o r i e n t a t i o n s a r e s p e c i a l .
horizontal,
denoted by H,
One o f these Is
which means p a r a l l e l t o t h e support
The o t h e r i s 7 w h i c h i s used t o l a b e l any r e g i o n which
surface.
I s p a r t of t h e v i s i b l e support surface,
A l l o t h e r o r i e n t a t i o n s are q u a n t i z e . l o c ' c o r l i n z t o t h e i r
value,
i.e.
es
according t o t h e a n g l e between t h e p r o j e c t i o n o f
t h e i r normals o n t o t h e support s u r f a c e plane and t h e scene z
axis.
The q u a n t i z a t i o n s are:
F = front,
es : , w i t h i n about
1 0 degrees o f t h e z axis;
the
exact values w i l l v a r y depending on t h e d l s t a n c e o f the eye from
t h e scene,
b u t t h e working d e f i n i t i o n i s t h a t such a surface can
have r i g h t angle v e r t i c e s a t b o t h l e f t and r i g h t extremes such
t h a t f o r each of t h e v e r t i c e s o n l y one j u n c t i o n segment o u t o f a
possible three i s v i s i b l e .
Another way o f p u t t i n g t h l s i s t h a t a
r e g i o n l a b e l e d F can have edges l a b e l e d i'convex"
a t I t s r i g h t and
l e f t extrema o n l y I f t h e o b j e c t does n o t have r e c t a n g u l a r
vertices.
R = rlght,
BR
OS
= back r l g h t ,
B L = back l e f t ,
L = left,
es
between t h e l i m i t of F and 9 0 degrees.
es
ec
between 9 0 and 1 8 0 degrees.
between 1 8 0 and 2 7 0 degrees.
between 2 1 0 degrees and t h e 1 l m l t o f F.
Understanding Scenes With Shadows. Page 3 0
To t h i s w i l l be appended a s u f f i x t o I n d i c a t e the v a l u e o f
, the
a n g l e between the normal t o a surface and t h e
p r o j e c t i o n o f the normal o n t o the s u p p o r t plane.
These s u f f i x e s
are:.
+s
Vsvertical,
D = down.,
=Odegrees.
<
+s
VU = v . e r t i c a l / u p ,
O degrees.
+S
HU = h o r I z o n t a l / u p ,
between 0 and 4 5 degree,^.
between 45 and 9 0 degrees ( 9 0
degrees = hor.1z o n t a l 1.
N o t i c e t h a t I n a scene made up o n l y of o b j e c t s which have
r e c t a n g u l a r corners,
bnother,
LV,
FV,
where none of t h e o b j e c t s l1leansf1on
t h e o n l y p o s s i b l e v f s t b l e s u r f a c e o r l e n t a t l o n s a r e B,
and RV,
r e g a r d l e s s o f eye posi t i o n .
I n addl t i o n ,
s u r f a c e e i t h e r belongs t o t h e s e t o f i l l u m i n a t i o n types
SB,
SAB
o r i s t y p e SC,
l i g h t / e y e angle,
1V.F.
a
I, SA,
and t h i s i s a f u n c t i o n o n l y o f r e l a t i v e
Independent o f l i g h t e l e v a t i o n .
Light posl t i o n -quantization
The r e l a t i v e l i g h t l e y e angle,
a c c o r d i n g t o T a b l e 1.
H,
BL , w i l l
be q u a n t i z e d
There a r e t w e l v e p o s s i b l e values,
which
can be visualized by c o n s i d e r i n g a vantage p o l n t on t h e scene y
Understanding Scenes With Shadows. Page 3 2
axis.
From t h i s p o s i t i o n ,
l o o k i n g down on t h e scene,
F i s a 30
degree wide segment centered on t h e z a x i s ( 6 o ' c l o c k ) .
then centered around 5 o'clock,
FR i s
B around. 1 2 otc:1ock, e t c .
The l i g h t e l e v a t i o n i s quantized i n t o t h r e e values,
a c c o r d i n g t o which angle i s nearest t h e a c t u a l value:
L = low,
3 0 degrees.
M = medium,
H = high,
45 degrees,
6 0 degrees,
The eye e l e v a t l o n angle i s quantized I n f o u r values,
-
three
(
o f which a r e exact1 y t h e same as f o r t h e 1 l g h t source elevatlon,
the
a d d i t i o n a l one being VL
l e s s than 1 5 degrees.
Given t h i s q u a n t f z a t i o n and the methods o f Appendix 1 we can
then generate v a l u e s f o r each combi nat.lon o f 1 i g h t source
posl t i o n ,
s u r f a c e o r i e n t a t ion,
and eye p o s i t i o n which l n d l c a t e
how a g i v e n s u r f a c e 1s i l l u m i n a t e d and whether o r n o t I t i s
v l s l ble.
Understand1ng Scenes W i t h Shadows. Page 35
V.
The background/seene boundary
While I w i l l t a l k e x t e n s i v e l y about t h e scene/background
boundary,
i would l i k e t o p o i n t o u t t h a t a l l o f t h e techniques
developed t o handle t h i s boundary w i l l a l s o be appl i c a b l e t o any
i s o l a t e d s u b p i c t u r e i n t h e scene,
o r f o r most p o r t i o n s of t h e
scene around whlch we can draw a p a t h k h l c h c o n s i s t s o n l y of the
j u n c t i o n s which c h a r a c t e r i z e t h e scene/background boundary,
s p e c i a l treatment o f TEE j u n c t i o n s .
Since I have n o t worked o u t
I w i l l not
t h e d e t a i l s o f i s o l a t i n g such p o r t i o n s o f t h e scene,
make any o v e r o p t i t n i s t l c claims,.
plus
b u t I fntend t o f o l l o w up on t h i s
approach.
You w i l l remember t h a t e a r l i e r I d e f l n e d t h e background
r e g i o n t o be t h a t p o r t i o n o f t h e v l - s i b l e -support s u r f a c e which I s
illuminated.
Every 11-ne whlch bounds t h i s r e g i o n can have o n l y
one o f t h r e e labels,
support s u r f ace),
namely obscure ( a 1 i n e segment o b s c u r i n g t h e
conc'ave ( o r minus),
and shadow,
shadow l i n e can be o r i e n t e d i n o n l y one d i r e c t i o n .
where t h e
We can then
l o o k a t t h e complete l i s t o f p o s s i b l e j u n c t i o n s and p i c k o u t
those whlch have segments bounded by any combination of these
three labels.
F i g u r e 5 A shows t h i s s e t o f j u n c t i o n s ,
where B / I
I n a j u n c t i o n segment i n d i c a t e s t h a t t h i s segment can be p a r t o f
t h e background region.
Understanding Scenes W i t h Shadows. Page 3 6
We can now make t h e r a t h e r remarkable o b s e r v a t i o n t h a t
whenever a j u n c t i o n on t h e scenelbackground boundary has one o f
i t s segments of t y p e T I , T2, S 1 , 52,
M,
o r F as p a r t o f t h e
background region, we can p a r t i a l 1 y l a b e l t h a t j u n c t i o n d i r e c t 1 y.
Frequently,
h a v i n g done t h i s ,
o t h e r j u n c t i o n labelings;
1.e.
t h e r e tnay be f o r c e d choices f o r
i f we check a l l t h e j u n c t i o n s
which a r e p a r t i a l l y labeled, we may f i n d t h a t t h e r e i s o n l y one
p o s s i b i l i t y f o r l a b e l i n g some j u n c t i o n s ,
given that the p a r t i a l
l a b e l i n g must be preserved,
I n any event,
m e r e l y by knowing t h a t one o f t h e segments o f
a j u n c t i o n i s p a r t ~f t h e background region, we can use t h i s
r e s t r i c t e d s e t o f j u n c t i o n s from which t o choose a l a b e l lng.
F i g u r e 5B shows t h e r e s u l t s o f a p p l y i n g the r e s t r i c t e d s e t of
p o s s l b i l i t i e s t o a l l n e drawing ( t a k e n from a photograph).
F i g u r e 5 C shows t h e r e s u l t s o f a p p l y i n g f o r c e d r u l e s t o t h e
p a r t i a l l y l a b e l e d l i n e drawing o f F i g u r e 5B.
The steps I n v o l v e d
i n forced l a b e l i n g are l i s t e d i n order o f t h e i r application
beneath F i g u r e 5 C .
Understanding Scenes With Shadows. Page 3 9
VI.
V1.A.
Scenes w i t h r e c t a n g u l a r v e r t i c e s
Theory
As I mentioned e a r l i e r ,
t h e cases where a scene c o n t a i n s
o n l y r e c t a n g u l a r v e r t i c e s i s a p a r t i c u l a r l y easy one.
In this
s e c t i o n i w i l l show how we can cornplete t h e l a b e l i n g o f t h e l i n e
drawing shown i n F i g u r e s 5B and 5 C assuming r e c t a n g u l a r v e r t i c e s
o n l y . F i r s t r e c a l l t h a t t h e o n l y s u r f a c e o r i e n t a t i o n s which a r e
p o s s i b l e i n t h e r e c t a n g u l a r v e r t i c e s case are B, t i 8 RVO FV,
LV.
I n F i g u r e 6,
1 show a c o m p i l a t i o n o f a l l t h e p o s s i b l e ways
i n which each r e g i o n can be l a b e l e d f o r o r i e n t a t i o n ,
-,
p a r t i c u l a r l i n e labeling f o r a junction.
t h e j u n c t i o n types,
given a
I have n o t r i s t e d a l l
b u t have conf l n e d myself t o those which are
needed t o complete t h i s example: e l l ,
psi,
and
and kay j u n c t i o n s .
arrow,
fork,
tee,
peak,
In p r a c t i c e we need not have t h i s e n t i r e
t a b l e i n lneinory since' we can use theorems t o generate p o s s i b l e
r e g i o n l a b e l i n g s as needed.
1 w i l l g i v e rules f o r generating t h i s
t a b l e i n t h e s e c t i o n on l i n e o r i e n t a t i o n s .
ELL
IPbel'kq
J
e
Allowable wr
m'*&%tiorrs
to
LL
Note: thmykO'Lt t k j s
A w e I htzeomXted
"u" f;.m FV, RV4L.V.
4
AUowdb
labslkq
SUP&
~n'wbtiows$w
Po P l
PP P3
*
KAY
PSI
L&Gau
.fs..
A - b a b l ~su
0 & t & t i ~
$1
s t Sf'
s2'
Understanding Scenes W i t h Shadows. Page 45
I n t h e r e c t a n g u l a r v e r t e x case t h e r e i s a p a r t i c u l a r l y
simple. r e l a t i o n s h i p between o r i e n t a t i o n and p o s s i b l e 1 i g h t i n g .
Table 2 shows t h i s r e l a t i o n s h i p .
and F i g u r e 6,
Using t h e r e s u l t s o f t h i s t a b l e
we can now a s s i g n l i g h t i n g possibilities t o each o f
t h e p o s s i b l e j u n c t i o n s p r e v i o u s l y c l a s s i f l e d by l i n e l a b e l s and
region o r i e n t a t i o n ,
Rather than show t h e complete set,
I have
chosen t o show o n l y t h e p o s s i b i l i t i e s f o r j u n c t i o n s which border
t h e background,
b u t the same methods can be used t o a s s i g n v a l u e s
f o r a1 1 j u n c t i o n s ,
F i g u r e 7,
The r e s u l t s of t h i s process a r e shown I n
Understanding Scenes Wfth Shadows. Page 5 1
Let me review e x a c t l y what we need t o do t o a r r i v e a t these
listings.
F i r s t we must generate a Ful-1 s e t o f j u n c t i o n 1 i n e
l a b e l i n g s f o r each j u n c t i o n type.
and a s s i g n r e g i o n values f o r
w h i c h t h i s l a b e l i n g can occur.
which have cracks,
We can then t a k e each l a b e l fng
a l l the possible orientations i n
( N o t i c e t h a t i n those j u n c t i o n s
I have i n c l u d e d some g r a v i t y information,
on
t h e b a s i s o f t h e o b s e r v a t i o n t h a t c e r t a i n o r i e n t a t i o n s can never
occur i f g r a v l t y i s assumed t o be i n t h e -y d i r e c t i o n ,
normal t o t h e suppport surface and-downward.
say about t h i s l a t e r ,
1.e.
I w i l l have more t o
a l o n g w i t h i n f o r m a t i o n about support which
can be deduced from l o c a l f e a t u r e s . 1
. ,
We haqe. thus d i v i d e d each
p o s s i b l e l a b e l i n g f o r 1 i n e s i n t o se"era1
has t h e same l i n e labels,
l a b e l ings,
each o f which
We can
but d.ifferent region labels.
t h e n t a k e each member o f t h i s extended set and d l v l d e i t i n t o
s e v e r a l p o s s l b l l l t i e s a c c o r d i n g t o 1 i g h t i n g direction.
N o t i c e t h a t i n some cases,
p a r t i c u l a r l y i n those j u n c t i o n s I
have c a l l e d "shadow causing junctions",
t h e r e i s o n l y one
p o s s i b i l i t y f o r region illuminatlon o f the junction.
I n these
cases we can t h e n say t h a t t h e j u n c t i o n as l a b e l e d can o n l y be
. present i n t h e scene i f t h e 11-ghting i s from a p a r t i c u l a r subset
of t h e s e t o f p o s s i b l e p o s i t i o n s .
Thus knowing t h a t t h e l i g h t i n g i s
-from a p a r t i c u l a r p o s i t i o n we can c o n s t r a i n t h e p o s s i b l e
l a b e l i n g s f o r j u n c t l o n s In t h e l i n e dr-awing and 1.n t h e o p p o s i t e
Understandi-ng Scenes W i t h Shadows. Page 53
d i r e c t i o n of Implication,
knowing a p a r t i c u l a r j u n c t i o n labellnq
we' can. c o n s t r a i n t h e possl.bl~e1 l g h t p o s i t i o n s .
V1.B.
An example
To show how these f a c t s h e 1 p . u ~i n l a b e l i n g a scene,
now r e t u r n t h e example s t a r t e d i n t h e p r e v i o u s s e c t i o n ,
t h e p a r t i a l labeling,
Given
o f 5 C and assuming t h a t a l l i t s v e r t i c e s
are r e c t a n g u l a r w i t h . no degeneracies,
l a b e l e d (1) i n F l g u r e 8.
f i g u r e 7,
l e t us
look f i r s t a t t h e j u n c t i o n
Glvbn t h e l i n e . l a b e l s a-nd r e f e r i n g t o -
t h e r e I s o n l y one p o s s i b i l i t y f o r l a b e l i n g t h e regions:
F i g u r e 8A
A t j u n c t i o n ( 2 ) by t h e same reasoning we have:
Understanding Scenes W i t h Shadows.
Page 54
F i g u r e 88
Thus we may deduce t h a t t h e l i g h t i n g must be somewhere I n t h e
i n t e r s e c t i o n o f t h e two i n t e r v a l s s p e c l f l e d by (1) and ( 2 1 ,
namely (RF,
LF)
/\
( F L O BL) = ( F L O L F ) ,
A t ( 3 ) we now have:
Figure 8C
so we must l o o k f o r a j u n c t i o n which agrees by r e g i o n type,
label,
and l i g h t i n g .
l a b e l i n g are:
line
The o n l y two candidates which agree on l i n e
Understanding Scenes W i t h Shadows. Paee 55
..
and c l e a r l y o n l y t h e f i r s t I s p o s s i b l e when r e g i o n l a b e l s are
considered.
Hence. we can l a b e l t h e o t h e r two segments and t h e
o t h e r two l i n e s .
s i n c e (F,
LF)
No f u r t h e r r e s t r i c t i o n on l l g h t i n g I s Implied,
(FL,
LF') = (FL,
LF),
A t (4) we now have:
Figure 8E
and a l l these a r e p o s s i b l e .
1 i n e segments,
yet.
Howevek, we can now l a b e l a l l t h e
even though we a r e n o t a b l e t o l a b e l t h e r e g i o n s
We do know t h a t t h e two regions f o r which we do not yet
have an o r i e n t a t i o n l a b e l must be g i v e n t h e same o r i e n t a t i o n
Understanding Scenes W i t h Shadows.
Page YQ
label :
Figure 8F
At (5) w e can have either:
Figure 8 G
At ( 6 ) w e can thus have, i f @ i s the correct labeling, only:
Figure 8H
U n d e r s t a n d i n g Scenes W i t h Shadows. Page 57
whereas I f
i s c o r r e c t we can have e l t h e r :
F i g u r e 81
However,
i fa w e r e t r u e ,
l i g h t i n g was between (R,
j u n c t i o n ( 6 ) would i m p l y t h a t t h e
F ) and (R,
F ) f ) (FL,
LF) = n i l .
S i m i l a r l y t h e 1 i g h t i n g p o s s l b l l i t i e s g i v e n m a s t h e proper
l a b e l i n g have an empty I n t e r s e c t i o n w i t h (FL,
el-lminate b o t h @ a n d B
LF),
so we can
T h l s now e n a b l e s us t o
leaving only
go back t o (4) and . a s s i g n X = L.
Because (6) must b e l a b e l e d as
ins we can
l a b e l a1 1 on t h e
l l n e s i n j u n c t i o n s ( 7 ) t h r o u g h (11) as shadow l i n e s ,
are the o n l y possible continuations,
s i n c e these
regardless of l i g h t i n g
posi tlon,
J u n c t i o n ( 1 2 ) must be l a b e l e d as:
Understand i n g Scenes W l t h Shadows.
Page
50
F i g u r e 8J
since again t h i s i s the o n l y p o s s i b i l i t y .
J u n c t i o n (13) must be l a b e l e d as:
F l g u r e 8K
s i n c e t h e r e I s no o t h e r l a b e l i n g whlch has one l i n e l a b e l e d as
shown and whlch a l s o i s p o s s i b l e g i v e n t h e l i g h t i n g d i r e c t i o n .
T h i s t h e n a l l o w s us t o l a b e l (14) through (18) as f o r c e d choices
on a l i n e l a b e l i n g
b a s i s alone. N o t i c e t h a t t h i s f o r c e s r e g i o n
R 1 t o b e l a b e l e d llFV1l,
since t h i s i s the o n l y region o r i e n t a t i o n
w h i c h can be bounded by b o t h (14) and ( 1 8 ) .
U n d e r s t a n d i n g Scenes W i t h Shadows.
I w i 1 1 n o t c o n t i n u e t o show each step,
r e s t as an e x e r c i s e ,
but w i l l
Page 5 3
leave the
i f you a r e i n t e r e s t e d enough t o c o m p l e t e I t .
The f i n a l r e s u l t s a r e shown i n F i g u r e 9.
ambiguous 1 i n e segments,
There a r e t h r e e
and p a r t o f my work t o b e done I n t h e
f u t u r e w i l l d e a l w i t h ways o f r e s o l v i n g t h e s e a m b i g u i t i e s .
Understanding Scenes With Shadows. Page 6 1
V I I . L i ne segrnent o r i e n t a t ions
i n general we w i l l n o t be a b l e t o t o t a l l y l a b e l a 1 i n e
preceeding
drawing w i t h o n l y t h e i n f o r m a t i o n we used i n t h e
example. F o r t u n a t e l y ,
t h e r e are ways o f e x t r a c t i n g i n f o r m a t i o n
which we have n o t yet expl.oited,
V I I .A.
Background/scene boundary r e v i s i t e d
We can a s s i g n q u a n t i z e d o r i e n t a t i o n values t o each o f t h e
l i n e segments we encounter w h i l e moving around t h e scene I n o u r
normal ( c l o c k w i s e as seen from t h e background) d i r e c t i o n . We
a1 ready know t h a t each 1 i n e segrnent on t h l s bound"ary must be
l a b e l e d obscure,
concave,
case of b l o c k s only,
exposition,
o r shadow,
and t h a t i n t h e r e s t r i c t e d
which I w i l l c o n t i n u e t o use f o r ease o f
each r e g i o n separated from t h e background by one o f
these 1 i n e segments must be H,
FV,
R.V,
LV, o r B / S A .
I w i 11 assign
1 i n e o r i e n t a t i o n values from a set o f e l g h t compass points,
so
t h a t f o r i n s t a n c e i f we move d i r e c t l y upward along a l i n e
segment,
I w i l l l a b e l t h a t l i n e N,
and so on. N,
E,
S,
and W w i l !
be d e f i n e d t o i n c l u d e l i n e s which f a l l w f t h i n about 5 degrees o f
v e r t i c a l o r horizontal,
denoted by NE,
SE,
SW,
and t h e angles I n between w i l l then b e
and NW. (The w i d t h o f t h e r e g i o n s 1 s a
f u n c t i o n of eye distance;
as the d i s t a n c e goes t o I n f i n i t y ,
the
U n d e r s t a n d i n g Scenes W i t h Shadows.
N,
S,
Page ;2
and W s e c t o r s can becorne a r b i t r a r i l y s m a l l . 1
E,
We can t h e n s t a t e t h a t t h e f o l l o w i n g f a c t s a r e t r u e b y
e x h a u s t i v e l y c o n s i d e r i n g a1 1 p o s s i b i l i t i e s :
1. A shadow l i n e may have any o r i e n t a t i o n .
2.
I f t h e l i n e L i s an o b s c u r i n g l i n e ,
r e g i o n s e p a r a t e d f r o m t h e background by L,
o r i e n t a t i o n o f L,
O(L) = N
+X
= RV, FV, o r H.
O(L) = E ) X = F V
= SE . ) X
O(L) = S
O(L) =
3
and O(L) = t h e
then:
O(L) = NE S X = R V
O(L)
and X denotes the
(*I
(*I
= LV ( * I
X = LV, FV, o r H.
{SW,
W,
NW]
+X
= H.
( * I i n d i c a t e s t h a t t h e o b j e c t o f w h i c h t h i s edge i s a p a r t 1 s
s u p p o r t e d o f f t h e s u r f a c e o r e l s e i s concave.
3.
I f L I s a concave edge then:
O(L) = {N,
S,
SW, W,
edge.
O(L)
= NE .) X = R V / I .
+
N W ~ t h e r e i s no such X w i t h a concave
Understanding Scenes W i t h Shadows.
Using t h e same approach w i t h o'ther l i n e l a b e l s ,
g e n e r a t e t a b l e 3,
which give;
Page 63
we can
the possible region labels f o r
1 i n e s w l t h any l a b e l and o r i e n t a t i o n . T h i s t a b l e can t h e n be used
t o generate f i g u r e 6, which I presented i n s e c t i o n V I . A .
w i t h no
just i f ication,
Interior forks
V I I.B,
.
.
Rather than g i v e a f u l l treatment of l n t e r i , o r j u n c t i o n s ,
w i 11 t r e a t o n l y one p a , r t , i c u l a r l y , u s e f u l ' j u n c t i o n .
I
I..f i n a scene
w i t h o n l y r e c t a n g u l a r v e r t i c e s we .encounter a f o r k i n t h e
i n t e r i o r of a scene ( l.e.
none..o..f I . t s segments i s p a r t o f t h e
i1 lumlnated background) 'then' the' correspon'dinp; v e r t e x I s t y p e I
( a l l l i n e segments convex1 i f and o n l y i f one l i n e segment p o i n t s
"south"
sort,
from t h e j u n c t i o n ;
i f there i s a type I junction. o f t h i s
t h e n we can a l s o l a b e l i t s r e g i o n s :
Figure 17A
Understanding Scenes With Shadows. Page
6s
Rough p r o o f :
The e n t i r e s e t o f p o s s l b l e f o r k s i s :
yyyyyy'y
CI)
(*)
I
(3)
C4 )
.CS)
Figure 168
C6)
(7)
C l e a r l y (1) can be l a b e l e d as shown.
A l l of t h e o t h e r s have a t l e a s t . one concave edge.
concave edge p o f n t s "south";
then t h e r e g i o n t o I t s r i g h t must be
LV and t h e r e g i o n t o I t s l e f t must be RV.
obscured r e g i o n must be H,
Suppose a
Then I n ( 2 ) t h e
b u t we have assumed from t h e b e g i n n i n g
t h a t . t h e eye e l e v a t i o n i s s u f f i c i e n t s.o t h a t any h o r i z o n t a l
surface w i t h normal upward i s v l s i b l e unless obscured.
Likewise
( 3 ) 1 s . Impossible,
but the
s i n c e I t must a l s o have an H region,
normal o f t h i s r e g i o n p o i n t s downward,
so t h e surface c o u l d n o t
be v l s l b l e . S i m l l a r l y ( 4 ) through ( 7 ) a r e impossible w i t h t h e i r
concave edges downward,
eliminate (21,
V I I.C,
(41,
(51,
and s i m i l a r reasoning can be used t o
and ( 7 ) i n t h e i r o t h e r o r i e n t a t i o n s ,
An example w i t h degeneracies
I have p i c k e d t h e scene i n f i g u r e 10A t o i l l u s t r a t e not o n l y
t h e use of l i n e o r i e n t a t i o n s ,
w i t h several degeneracies.
b u t a l s o t h e p a r s i n g of a scene
We can s t a r t by l a b e l i n g the l i n e
Understanding Scenes With Shadows. Page 6 7
segments of j u c t ions w i t h scene/background segments o f T I ,
Sl, 52, M, and F as we d i d i n s e c t i o n V I . 8 ,
r e s u l t i n g "forced"
l a b e l ings,
T2,
and then l a b e l
choices, F i g u r e 1 0 A c o n t a i n s a l l these
p l u s o r i e n t a t ions f o r each 1 i n e segment around t h e
scene/background boundary. Each de.generate j u n c t i o n i s marked
IID
I1
b
i g n o r i n g t h e i n t e r i o r f o r k s I n t h i s example, we can
c o n s t r u c t p o s s i b l e l a b e l i n g s f o r R 1 through R 1 3 .
Notice that f o r
t h e l i n e segments a l r e a d y l a b e l e d ltobscu.ret' we need n o t conslder
shadow and concave edge r e g i o n p o s s i b i l i t i e s . The p o s s i b l e
l a b e l ings f o r each r e g i o n are shown i n t a b l e form i n f i g u r e I O B .
From t h e r e s u l t s of t h e a p p l . i c a t i o n o f t h e l i n e o r i e n t a t i o n r u l e s
we can unambiguously l a b e l t h r e e r e g i o n s and s i x l i n e segments
directly.
We can then proceed w i t h " f o r c e d choices" as we d i d i n t h e
p r e v i o u s example.
10C.
T h i s process i s shown s t e p b y s t e p i n F i g u r e
As t h e r e s u l t o f t h e a p p l i c a t i o n o f these l i n e o r i e n t a t i o n
r u l e s we a r e l e f t w i t h s i x j u n c t i o n s n o t t o t a l l y labeled,
of whlch are degenerate, These a r e shown i n F i g u r e 10D.
three
RI- NW
utqe
RI-s€
G/I
Rl-Sw
K5- S E
RS-S
h e
H
L
B/sA
v(R~)=
8 / 5 ~
Understanding Scenes With Shadows. Page 7 2
About degenerate j u n c t i o n s we can say:
1. C e r t a i n j u n c t l o n s can never be degenerate,
namely e l l and
and p o s s i b l y o t h e r s i n p a r t i c u l a r o r i e n t a t i o n s .
I f ever we
a r e l e f t w l t h such a j u n c t l o n which we cannot l a b e l ,
then t h e r e
tee,
i s a m i s l a b e l i n g somewhere e l s e i n t h e scene.
2. Any j u n c t l o n whlch does n o t f a 1 1 i n t o one o f t h e
a l l o w a b l e j u n c t l o n categories must. be a degenerate j u n c t i o n .
3.
Any j u n c t i o n whlch i s degenerate can be broken up i n t o :
( a ) two real, j u n c t l o n s ,
one o f which must s.eparate a l o n g
1i.nes l a b e l e d e l t h e r -both. o b s c u r i n g o r b o t h concave,
or
.
( b ) a j u n c t i o n w i t h a c o i n c i d e n ' t a l l y a l i g n e d sh.adow edge.
A good h e u r i s t i c t o n o t e I s t h a t any s t r a i g h t l i n e s I n a
degenerate j u n c t i o n whlch appear t o be c o l l l n e a r a r e l i k e l y t o be
t r u l y connected. Some cases which f o l l o w t h i s p a t t e r n a r e showri
i n f i g u r e 1 0 E a l o n g w l t h some cases which do not.
I n any event,
I
have n o t y e t devoted s u f f i c i e n t a t t e n t i o n t o degenerate j u n c t i o n s
t o say an.y more t h a n t h a t I can show methods t o l d e n t i f y whlch
j u n c t I o n s a r e degenerate,
and t h a t I can suggest some promising
c o n s i s t e n c i e s between v a r i o u s p o s s i b l e degenerate j u n c t l o n s .
Understanding Scenes W i t h S h a d o ! ~ ~ ?a:;c
.
Sorne m i s c e l l a n e o u s p o i n t s o f i n t e r e s t
VIII.
V I I I .A.
Support
I f we ever encounter any j u n c t i o n s such t h a t t h e r c q i o n s and
1 i n e s a r e l a b e l e d as i n f i g u r e 11, we can draw c o n c l u s i o n s about
support
listed.
between t h e o b j e c t s which correspond t o t h e r e g i o n s
I have shown c o n f i g u r a t i o n s which i m p l y support,
t h e same token,
b u t by.
c e r t a i n o t h e r c o n f i g u r a t i o n s can be used t o
a s c e r t a i n cases where s u p p o r t between o b j e c t s i s precluded,
Note t h a t i f t h e r e a r e no shadows h t h e scene we w i l l
u s u a l l y n o t be a b l e t o t e l l whether one b l o c k supports a n o t h e t o r
whether i t m e r e l y obscures it, s i n c e shadows a r e the prime means
o f d i s t i n g u i s h i n g between o b s c u r i n g and concave edges,
V1iI.B.
F i n d i n g i s o l a t e d subscenes
I mentioned a t t h e b e g i n n i n g o f s e c t i o n V that we can
i s o l a t e p o r t i o n s o f a scene which i t may be p o s s i b l e t o t r e a t as
though t h e y were surrounded by background. To do t h i s we need
o n l y f i n d p a t h s around a l l s e t s o f r e g i o n s i n t h e l i n e drawing
which c o n s i s t o n l y o f j u n c t i o n s which can occur on t h e
background/scene boundary p l u s TEES which a r e o r i e n t e d so t h a t TO
i s i n t h e i n t e r i o r o f t h e r e g i o n we a r e i s o l a t i n g .
I belleve t h a t
Understanding Scenes W i t h Shadows. Page 7 6
t h i s I s one o f t h e most p r o m i s i n g ' l i n e s o f r e s e a r c h t o b e
f o l l o w e d up.
VII1.C.
Non-rectangular
scenes
I n t h i s a r e a I s t i l l have a great. d e a l o f work t o do.
However,
from p r e l i m i n a r y o b s e r v a t i o n I be1 I e v e t h a t i t may b e
p r o f i t a b l e t o c o n s i d e r e v e r y scene i n i t i a l l y as though i t were a
scene w i t h o n l y r e c t a n g u l a r v e r t i c e s . T h i s w i 11. accomplish a t
l e a s t two ends:
any subport ions. o f t h e scene which c o n s i s t o f
r e c t a n g u l a r j u n c t i o n s o r i n which non-rectangul a r j u n c t i o n s
f o l low t h e same t y p e s o f r u l e s as do r e c t a n g u l a r j u n c t i o n s can be
labeled,
and we can e f f e c t i v e l s y I s o l a t e j u n c t i o n s which do n o t
f o l l o w t h i s pattern,
these junctions w i l l t h e n be e i t h e r non-
r e c t a n g u l a r o r degenerate.
V I I I .D,
Curved surfaces
U s i n g t h e r e s u l t s o f Appendix 1, we can f i n d c o n d i t i o n s on
t h e b o u n d a r i e s of curved surfaces,
s i n c e a t these boundaries t h e
p o i n t s must a l l s a t i s f y t h e c o n d i t i o n s V
t h e o r i g i n - e y e v e c t o r and V
V
= 0, where V
Is
i s t h e s u r f a c e normal. Consequent1 y
we can a l s o c h a r a c t e r i z e t h e q u a n t i z e d r e g i o n v a l u e f o r a curved
s u r f a c e a t each p o i n t a l o n g i t s boundary,
and perhaps can
Understanding Scenes W l t h Shadows. Page 7 7
u s e f u l l y t r e a t such surfaces accordlng t o t h e range of values
through whlch t h e surface normal must pass.
V1II.E.
A grammar o f r e g i o n s
An i n t e r e s t i n g c o r o l l a r y t o t h e o b s e r v a t i o n t h a t each r e g i o n
can have o n l y one l a b e l I s t h a t we can d e s c r i b e a g e n e r a t i v e
grammar f o r surfaces o f particular types. For example,
every
s u r f a c e whlch I s bounded b y a l l t y p e 1 v e r t i c e s ( t y p e 1 v e r t i c e s
a r e ones'whlch occupy o n l y one o c t a n t o f space a t t h e vertex,
or
a l t e r n a t e l y have t h r e e convex edges). can be generated b y t h e
transi t l o n rules:
A1
-+ TO
o r L1 o r A2
L1
+ TO
o r L1 o r A 2 ( l a b e l
(label
-+)
+)
TO 4 TO o r L 1 o r A 2 Clabel
A 2 4 A l o r F
F
--=+A 1
or F
(label
(label
+
+
-+-
1
where any "sentence1' which
c o n t a i n s a t l e a s t t h r e e j u n c t i o n s f rorn the s e t
ELI,
Al,
A2,
FI
represents a r e a l s u r f a c e o f t h i s k l n d . Such s u r f a c e s i n c l u d e the
faces o f a l l convex polyhedra such as prisms,
cubes,
pyramids,
e t c . whlch have o n l y t r i h e d r a l v e r t i c e s .
We can a l s o l i s t t r a n s l t l o n r u l e s f o r a l l t y p e SA shadow
-regions and r e g i o n s bounded on1 y by convex edges,
as we1 1 as each
of t h e preceedlng types as I t appears when p a r t i a l l y obscured by
u n d e r s t a n d i n g Scenes W i t h Shadows. Page 7 8
o t h e r o b j e c t s . We may be a b l e t o use t h e s e o b s e r v a t i o n s t o
r e c o g n i z e some r e g i o n s d i r e c t l y ,
r a t h e r t h a n a r r i v i n g a t labelinx
f o r regions on t h e b a s i s o f the
l i n e s surrounding the region
w h i c h have a l r e a d y been l a b e l e d .
V I I I.F.
A f i n e r c l a s s i f i c a t i o n o f j u n c t i o n types
I n a l l t h e work so f a r ,
I have assumed t h a t we a r e u n a b l e t o
t e l l t h e d i f f e r e n c e between a n g l e s l e s s t h a n o r g r e a t e r t h a n 9 0
degrees. ' A g l a n c e a t t h e ar'row and t e e l a b e l i n g s o f f i g u r e 1
s h o u l d c o n v i n c e y o u t h a t I f we have more preci'se i n f o r m a t I o n about
angles,
we can f u r t h e . r p a r t i t i o n t h e p o s s i b l e j u n c t i o n l a b e l l n g s
and t h u s make t h e p r o b l e m o f a s s i g n i n g 1-abels e a s i e r .
Understanding Scenes l J i t h Shadows. Page 7 7
IX.
Surnmary and plans f o r r e s e a r c h
I have demonstrated t h e use o f a l a r g e number o f r a t h e r
i s o l a t e d f a c t s and methods,
and would now l i k e t o say a 1 i t t l e
about how these can a l l be t i e d t o g e t h e r i n a conceptual .
framework. As a f i r s t a p p r o x i m a t i o n t o a t o t a l p i c t u r e o f t h i s
system,
we can o r d e r t h e v a r i o u s scene v a r i a b l e s a c c o r d i n g t o how
w i d e l y any p a r t i c u l a r p i e c e o f i n f o r m a t i o n can d i r e c t l y I n f l u e n c e
o t h e r scene v a r i a b l e s .
1. L i n e l a b e l s a f f e c t ( d i r e c t l y ) o n l y t h e j u n c t i o n s a t b o t h
ends o f t h e l i n e segment,
2.
L i n e l a b e l s p l u s o r i e n t a t i o n of the l i n e s a f f e c t the
j u n c t i o n s a t b o t h ends p l u s t h e r e g i o n s on b o t h s i d e s of t h e 1 lne
segment.
3. J u n c t i o n l a b e l s a f f e c t a l l l i n e s whlcli are p a r t o f t h e
j u n c t i o n and may i n some cases be i n t e r r e l a t e d w i t h l i g h t and e y e
position,
4,
J u n c t i o n l a b e l s p l u s o r i e n t a t i o n a f f e c t a l l l i n e s whlch
are p a r t o f the junction,
and eye p o s i t i o n ,
f r e q u e n t l y are i n t e r r e l a t e d w l t h l i g h t
and a l s o a f f e c t a l l r e g i o n s which bound t h e
junction,
5.
region,
Region l a b e l s a f f e c t a l l j u n c t i o n s and l i n e s bounding t h e
a r e i n t e r r e l a t e d w i t h eye and 1 i g h t p o s i t i o n ,
a f f e c t a d j a c e n t regions.
and may
Understanding Scenes W i t h S h i ~ d o w s . P a . 7 ~E r j
6.
Region l a b e l s p l u s a l i n e l a b e l d e f i n i t e l y a f f e c t
the
a d j a c e n t r e g i o n s as w e l l as t h e o t h e r v a r i a b l e s frotq (5).
7.
Path l a b e l s a f f e c t a l l r e g i o n s i n s i d e t h e p a t h ( a s i n
f i n d i n g an i s o l a t e d subscene o r rnoving a-round t h e
scene/background boundary,).
8.
L i g h t source and eye p o s i t i o n s can p o t e n t l a l l y a f f e c t a l l
j u n c t i o n s and r e g i o n s ,
T h i s i s a v e r y rough h i e r a r c h y ,
b u t I t h i n k i t g i v e s some
i d e a o f t h e amount and types o f interdependence between t h e
v a r i o u s scene v a r i a b l e s ,
My p l a n s f o r r e s e a r c h I n c l u d e as a t o p p r i o r i t y t h e g a i n i n g
o f g r e a t e r i n s i g h t i n t o j u s t what t y p e o f conceptual framework we
need t o be a b l e t o l u c i d l y d e s c i be t h e n a t u r e of t h e problem and
t h e methods f o r s o l v i n g i t . I a l s o i n t e n d t o f o l l o w up a l l t h e
t o p i c s mentioned i n t h e p r e v i o u s section,
and f i n a l l y I p l a n t o
w r i t e a program embody1 ng t h e r e s u l t s o f my work.
Understand1 ng Scenes W i t h Shadows. Page 8 1
Bib1 iography
(1) Clotves M.B.
On s e e i n g t h i n g s
A l Journal, S p r i n g 1971
( 2 ) Dowson M.
What c o r n e r s l o o k l i k e
MIT/AI V l s i o n F l a s h 13,
June 1 9 7 1
( 3 ) Dowson M. & W a l t z D.L.
Shadows and c r a c k s
MIT/AI V i s i o n F l a s h 14, June 1 9 7 1
(
(4) F i n i n T.
F i n d i n g the skeleton of a b r i c k
MIT/AI V i s i o n F l a s h 19, August 1971
( 5 ) Guzman A,
Computer r e c o g n i t i o n o f t h r e e - d i m e n s l o n a l
o b j e c t s I n a v i s u a l scene
P r o j e c t MAC Techn i c a l r e p o r t MAC-TR-59
December 19 68
( 6 ) Huffman D.A.
I m p o s s i b l e o b j e c t s as nonesense sentences
Machine I n t e l l i g e n c e 6, 1970
Af
Understanding Scenes W i t h Shadows. Page
Appendix 1. Exact r e l a t i o n s between scene f e a t u r e s
and l i g h t - e y e placement
The f i r s t problem i s t o f i n d hays o f o b t a i n i n g t h e l i g h t - e y e
angles w i t h r e s p e c t t o t h e scene. U s i n g t h e c o o r d i n a t e system C
defined i n section IV.C.,
adir
assuming t h a t t h e p r o j e c t i o n s o f
t h e scene x and y axes d e f l n e t h e r e t i n a l x and y axes,
we then
know t h a t any v e c t o r i n t h e scene:
appears i n t h e r e t i n a l p r o j e c t i o n as
A
axp+ ( b
A
where xc,
A
and y,
ys,
A
z,
-
A
c)yp
h
a r e u n i t v e c t o r s . in. t h e scene c o o r d i n a t e s and x,
are u n i t vectors i n the r e t i n a l coordinates.
Appendix 1.A.
Eye e l e v a t i o n
I f t h e r e a r e r e c t a n g u l a r b l o c k s i n t h e scene which r e s t on
t h e support surface,
then we can e x t r a c t t h e eye e l e v a t i o n b y
l o o k i n g a t t h e appparent angles of t h e c o r n e r o f a block.
A 1 shows such a corner.
Figure
Understanding Scenes W i t h Shadows, Page A1.Z
Figure A 1
The t h r e e v i s i b l e edges d e f i n e an orthogonal c o o r d i n a t e
system C",
r o t a t e d b y an a n ~ l e
QEB
w i t h respect t o t h e eye
c o o r d i n a t e system C as d e f i n e d I n t h e p r e v i o u s s e c t i o n .
This i s
p i c t u r e d i n F i g u r e A2, where t h e corner i s drawn as i t would
appear f r o m a p o i n t on t h e scene y-axis.
where
7," and
the retina.
'
We have
a r e def ined w i t h respect t o
C",
not
Understanding Scenes W i t h Shadows. Page
I n t h e scene coordinates
tan
7," t a c t yi'
(11
(
Furthermore we know from elementary geometry t h a t these angles
a r e r e l a t e d t o those on t.he r e t i n a l p r ~ j e c , t l o nby
7,
= t a n 7,''
s i n &E
tan
and
(2)
t a n ~ ~ = t a n ~ ~ ' s i (n
3) ~ E
and plugging i n t o (1) we get
tan
and s i n c e
solve f o r s i n
7,/ s i n
$m
sin
/ tan
'7(,
(10
can o n l y be between 0 and 90 degrees, we can
(Pt:
Al.3
Understanding Scenes W i t h Shadows. Page Ai.4
sin
cpG
= (tan
tan
7' 1.Y
O f course i f t h e r e a r e no r e c t a n g u l a r c o r n e r s i n t h e scene,
w e cannot use t h i s method.
Appendix 1 . B .
L i g h t placement
Whenever t h e r e
Is a v e r t i c a l edge, v i s i b l e I n i t s e n t i r e t y ,
w h i c h c a p t s a shadow,
v i s i b l e t o i t s f l r s t junction,
we can f i n d
t h e l l g h t placement as a f u n c t i o n o f t h e eye e l e v a t i o n .
Such a
j u n c t i o n i s shown i n F i g u r e A3.
F i g u r e A3
The a n g l e
CK
i s r e l a t e d t o t h e angle
((ACB)
BE',
o r a n g u l a r d1fferenc.e between t h e p r o j e c t . l o n s o f t h e l l g h t source
and eye o n t o t h e s u p p o r t plane,
t h e l l g h t source e l e v a t l o n ,
and is independent of
As above,
Understandl ng Scenes W i t h Sharlows. Paee A1.5
eEL=
t a n DC s i n
tan
4s
To f i n d t h e l i g h t e l e v a t i o n I s a l i t t l e more complicated.
F i r s t consider the vector
$in
t h e scene c o o o r d i n a t e s .
p o i n t s toward t h e l i g h t source from t h e o r i g i n .
-+
A'B'
I f the l i ~ h t
#L and i t s angle w i t h r e s p e c t t o t h e eye
source e l e v a t i o n i s
is
Thls
gEL , then
= (cos +Lsin
egL):
+ (sin
I f t h e eye e l e v a t i o n i s g l v a n by
+L)
C
+E
+
(cos 4 L c o s
, then
%L) hz .
t h i s transforms
o n t o t h e r e t i n a as
+ ( s i n &cos
Assuming
oC
(arc t a n b i n
Solving f o r tan
and
-
+E
p.
&-
cos
+pas
sin
€L
A
Y.
a r e b o t h between 0 and 1 8 0 degrees,
f
cos
gives
&OS
$Lsin$E)/cos
# p i n
ea))
Understa.ndlng Scenes W i t h Shadows. Page Al.6
tan
(pL= cos egbtan
Appendix 1.C.
+G+
(sine
/cos & ) t a n
CL
(6+ a < - 9 0
I.
F i n d i n g s u r f a c e normals
Whenever a r e g i o n borders t h e supppart surface and shares an
edge l a b e l e d "concave1' w i t h i t , we can use i t s angle w i t h r e s p e c t
t o the horizontal t o find
, the
@
angle o f t h e p r o j e c t i o n
o f t h e s u r f a c e normal w i t h r e s p e c t t o the z a x l s of t h e scene.
From t h e r e t i n a we can o b t a i p
degrees,
where
,
,
, between
- 1 8 0 and 1 8 0
Is measured f r o m t h e x axis ( h o r i z o n t a l 1
t o t h e edge, w i t h respect t o t h e j u n c t i o n encountered f i r s t when
movlng c l o c k w i s e f r o m i n s i d e t h e background. r e g i o n around t h e
scene/background boundary.
T h i s i s lll u s t r a t e d I n F i g u r e A 4 .
0
Figure A4
Understandl,ng Scenes Mi t h Shadows. Page
3
and
tan
es
A1,7
a r e r e l a t e d b y t h e equet I o n
eS= t a n rb; / s i n
I t i s n o t always posslbla t o f i n d
+E,
&,
t h e angle between
t h e support surface and t h e normal t o another surface,
Whenever
t h e surface i s v e r t i c a l and rect.angu.lar,
.then t t.s edges. w f 1 1
appear as v e r t i c a l i n t h e p r o j e c t i o n on the r e t i n a .
these cases
rectangular,
+s
= 0.
I f we assume t h a t t h e surface i s
then we can o b t a i n t h e a n g l e
+5
and
F i g u r e A5
(Assume t h a t R 1 i s r e c t a n g u l a r )
I n t h e scene
Obvfously i n
as a f u n c t l o n
U n d e r s t a n d l ng Scenes W t t h Shadows. Page
A1,8
On t h e r e t i n a
*
A B =
+
-(sin
( C O S +5
+J
cos
c/)~
+
so
s i n QS
and s o l v i n g f o r t a n
)/(cos
+s
eS I $
- ( s i n Cb5sin
s i n +S cos
tan
+S
We g e t
cos
&'
&
eS s i n
cPE
h
y,
=
+
s i n +=cos
QS
sin
1,
Page A1,9
U n d e r s t a n d i n g Scenes W i t h Shadows.
Appendix 1.D.
An example
Suppose t h a t we encounter t h e scene p o r t i o n shown i n F i g u r e
A6,
and suppose f u r t h e r t h a t we have beeen a b l e t o l a b e l i t a s
shown.
I w i l l show how t o f i n d t h e v a r i o u s scene v a l u e s ,
Eye e l e v a t i o n
sin
(PE =
(tan
(B)
tan%%(^))'/s =
= ((.577)(.577)?~=
( ( t a n 30')(tan
.577
Light-eye angle
tan
eELs( t a n O< ( B ) )
sin
&
= ( t a n ( - 7 ~ ~ ) ) ( . 5 7 7 )= (-3.73)(.577)
= -2.16
6
!&
30))
U n d e r s t a n d i n g Scenes W l t h Shadows.
Page
Al,ld.
Light elevation
tan
= Cos eGLtan
= (.420)(.708)
= .298
+
(bG+( s i n
/ ,816
+ (-.909
(-l.ll)(tan
ecL/cos(bg
+L.=
( p + oc -
t a n (30°+ 75'-
-
= .298
(-15°))
tan
90°)
90°)
= .596
(1.11)(-.268)
30.8~
Assuming t h a t a n g l e FEA i s a r i g h t a n g l e i n t h e scene,
we
can t h e n f i n d t h e s u r f a c e orientations as w e l l ,
Orientation o f R 1
Weassume t h a t
BD i s v e r t i c a l .
$S(~
= 0l, ) i.e.
R 1 i s vertical,
Whether o r n o t t h i s i s so, we can f i n d
tan
and assuming s i n
8(~1)=
(PE(A)
t a n %,(A)
= sin
tan
4,
/ s i n &(A)
( R )we g e t
€I+RI)
=
(tan ( - 3 0 ~ ) ) / ( s i n (35.3Q))
= (-.577)
/ (.577)
= -1.
since
Understanding Scen.es W i t h Shadows. Pafie ALL2
e S ( ~=l )-45'
= 315~.
O r i e n t a t l o n o f R2
+=(~2)
Asain assume t h a t s i n c e BD and GF a r e v e r t i c a l ,
t a n t 3 5 ( ~ 2 ) = tan
a
t a n 30°/
s i n 35.3O
( T h i s I s r e a l l y a double-check.
7,(B)
*
/ sin &(B)
( . 5 7 7 ) ./ (.577)
eS ( R 2 )
=
= 1.
tf we assumed f o r f i n d l n g
t h a t a n g l e ABG was a r i g h t angle i n t h e scene,
-45O )
= 0.
then
es(ql)
=
$so.)
Orientation of R3
.(
Here t h e surface i s not v = r t i c a l , so we. .kill f i n d
first.
= )t a n T l ( ~ ) / . s i n
t a n eS(~3
+E(~).
Q s( R 3 )
Under s t a n d l ng Scenes W i t h Shadows. Page A1.13
+E ( E l =
and a g a i n assuming t h a t
+E
t a n e S ( ~ 3 )= t a n ( - 1 5 0 ~ ) / ( s i n 35.3')
(8)
= 1.
R ( R=~225O.
)
( N o t i c e t h a t h e r e we have t o use some a d d i t i o n a l i n f o r m a t t o n t o
t e l l t h a t t h i s a n g l e i s not 45O.I
tan
= ( ( - s i ' n % ( ~ ) / cos & ( E ) )
= (-(-.707)
=
&(~3
/ (.816)(3.73)
( ,232 +
-
tan-' ~ ( E I cos
.5
- (-.707)(.701)
= 1 / .732
dL(~3)
=
53.8'
eS(~lt a n & E ) )
= 1.37
1