Extended SQL for Multimedia Queries
select <whats> from <whatr>
where <preds>
orderby <whato)
groupby <whatg>
<whats> can be alphanum, images, functions
<functions> can be numeric-valued, vectorvalued (such as histograms),
image-valued (such as extractors),
aggregates,….
<preds> can be hard constraints or similarity
functions
<whato) can be alphanum or similarities
<whatg> can be alphanum or similarities
1
Queries from Current System
select A.id, A.img, simfMRI(A.img, B.img)
from fMRI A, fMRI B
where B.id = 1
orderby simfMRI(A.img,B.img)
select A.id, I.img, S.prob
from fMRI A, simfMRI S
where S.img = A.img
orderby S.prob
fMRI
id img
simfMRI
img prob
similarities of
all images to a
selected one with
id = 1
*Assumes simfMRI is a global comparison of 2 fMRI images.
2
Queries from Current System
User query:
select A.id, A.img
from fMRI A, fMRI B
where B.id = 1 and simfMRI(A.img, B.img)
MystiQ query:
select distinct A.id, A.img
from fMRI A, simfMRI S
where S.fid = A.fid
fMRI
fid id img
simfMRI
img prob
similarities of
all images to a
selected one with
id = 1
*Assumes simfMRI is a global comparison of 2 fMRI images.
3
Potential Queries from Very Simple
and Beyond
Return the similarity between the fMRI images of
patient 12 and patient 37.
select simfMRI(A.img,B.img)
from fMRI A, fMRI B
where A.id = 12 and B.id = 37
fMRI
id img
simfMRI
prob
single similarity value
4
Q3
Find pairs of patients in the database having the
same age and similar fMRI images.
select A.id, B.id, P.age, simfMRI(A.img, B.img)
from patient P, patient Q, fMRI A, fMRI B
where P.ID = A.id and Q.ID = B.id and
A.age = B.age
orderby simfMRI(A.img, B.img)
patient
id age
fMRI
id img
simfMRI
img img prob
pairs of database images in
order of similarity
5
Q3
Find pairs of patients in the database having the
same age and similar fMRI images.
User query:
select A.id, B.id, P.age,
from patient P, patient Q, fMRI A, fMRI B
where P.ID = A.id and Q.ID = B.id and
A.age = B.age and simfMRI(A.img, B.img)
patient
id age
fMRI
fid id img
simfMRI
fid1 fid2 prob
pairs of database
images
MystiQ query:
Note: two
patients A.id,
B.id may have
multiple similar
images. The
probability is
non-trivial.
select distinct A.id, B.id, P.age,
from patient P, patient Q, fMRI A, fMRI B, simfMRI
where P.ID = A.id and Q.ID = B.id and A.age = B.age
and simfMRI.fid1 = A.fid and simfMRI.fid2=B.fid 6
Q4
Given a query image called thisimg, return the id’s
and similarities for database images that are similar
enough.
select A.id, P.age, A.img, simfMRI(thisimg, A.img)
where P.ID = A.id and simfMRI(thisimg,A.img) > thresh
orderby simfMRI(thisimg, A.img)
thisimg thresh
patient
id age
fMRI
id img
simfMRI
img prob
7
Q4
Given a query image called thisimg, return the id’s
and similarities for database images that are similar
enough.
select A.id, P.age, A.img
User query:
from patient P, fMRI A
where P.ID = A.id and
simfMRI(thisimg,A.img) > thresh
and simfMRI(thisimg, A.img)
thisimg
patient
fMRI
simfMRI
id age
img
fid id img
fid prob
MystiQ query:
select A.id, P.age, A.img, simfMRI(thisimg, A.img)
from patient P, fMRI A, simFMRI S
where P.ID = A.id and simfMRI(thisimg,A.img) > thresh
8
and A.fid = S.fid
Q4
Given a query image called thisimg, return the id’s
and similarities for database images that are similar
enough.
Better user query:
select A.id, P.age, A.img
from patient P, fMRI A
where P.ID = A.id and
and HIGH(simfMRI(thisimg,A.img)) [weight=7]
and simfMRI(thisimg, A.img) [weight=0]
thisimg
img
patient
id age
fMRI
fid id img
simfMRI
fid prob
9
Weights
I’m proposing extending the probabilistic predicates
and/or probabilistic tables with weights
E.g. FROM simfMRI S [weight=3.5]
WHERE simfMRI(img1,img2) [weight=-0.5]
Default weight = 0.0
Positive weight means the predicate is more important
Negative weight means it is less important.
10
High/Low Predicates
I’m also proposing introducing two new
probabilistic predicates:
High(value) = more “true” the higher “value” is
Low(value) = more “true” the lower “value” is
These predicates make sense only within a query:
Different values are compared and the higher (lower)
receive higher probability.
11
Q5
Given an image and a location, return id’s, images, and
activation levels from database images that have high
enough activation levels at that location in order by
similarity at that particular location.
select A.id, A.img, activation(A.img,loc),
simsignal(signal(A.img,loc), signal(thisimg,loc))
from fMRI A
where activation(A.img,loc) > thresh
orderby simsignal(signal(A.img,loc), signal(thisimg,loc)
thisimg loc thresh fMRI
fid id img
simsignal
activation signal
(,)
signal prob (, )
activation is a function that returns the mean activation level at a given location.
signal is a function that returns the signal at a given location.
12
locations denote particular voxels.
Q5
Given an image and a location, return id’s, images, and
activation levels from database images that have high
enough activation levels at that location in order by
similarity at that particular location.
User query:
select A.id, A.img, activation(A.img,loc)
from fMRI A, thisimg T
where activation(A.img,loc) > thresh
and simsignal(signal(A.img,loc), signal(T,loc))
thisimg loc thresh fMRI
fid id img
simsignal
activation signal
(,)
signal prob (, )
activation is a function that returns the mean activation level at a given location.
signal is a function that returns the signal at a given location.
13
locations denote particular voxels.
Q5
Given an image and a location, return id’s, images, and
activation levels from database images that have high
enough activation levels at that location in order by
similarity at that particular location.
Better user query:
select A.id, A.img, activation(A.img,loc)
from fMRI A, thisimg T
where high(activation(A.img,loc)) [weight=10.0]
and simsignal(signal(A.img,loc), signal(T,loc))
thisimg loc thresh fMRI
fid id img
simsignal
activation signal
(,)
signal prob (, )
activation is a function that returns the mean activation level at a given location.
signal is a function that returns the signal at a given location.
14
locations denote particular voxels.
Q6
Return ids and images for images (A) that have the
highest activation level of all database images (B) at a
given location. For each of these, return id’s and
images (C) of all database images whose activation
is above a threshold at that location, ordered by
their signal similarity to A at that location.
select A.id, A.img, C.id, C.img
from fMRI A, fMRI B, fMRI C
where activation(A.img,loc) = max(activation(B.img,loc))
and activation(C.img,loc) > thresh
orderby simsignal(signal(A.img,loc),signal(C.img,loc))
loc thresh
fMRI simsignal
id img signal prob
activation
(, )
signal
(,)
15
Q6
Return ids and images for images (A) that have the
highest activation level of all database images at a
given location. For each of these, return id’s and
images (C) of all database images whose activation
is above a threshold at that location, ordered by
their signal similarity to A at that location.
User query:
select A.id, A.img, C.id, C.img
from fMRI A, fMRI C
where high(activation(A.img,loc)) [weight = 100.0]
and high(activation(C.img,loc)) [weight = 0.0]
and simsignal(signal(A.img,loc),signal(C.img,loc)) [weight=0.0]
loc thresh fMRI simsignal
activation
(, )
signal prob
fid id img
signal
(,)
16
Q7
Extract and return a neighborhood of a given size about
a given location from a given query image.
select nbd(thisimg, loc, size)
from thisimg
thisimg loc size
Simple example of an extraction function.
17
Q8
Return id’s, images, and locations for images that have
similar neighborhoods of a given size at that location
to neighborhoods of the query image thisimg at location
thisloc of the same given size.
select A.id, A.img, I.loc,
simnbd(nbd(A.img, I.loc, size),
nbd(thisimg, thisloc, size)) as S
from fMRI A, intloc I
where A.img = I.img
orderby S
thisimg thisloc fMRI
id img
thisimg
A
intloc
simnbd
img loc img nbd prob
The intloc relation contains the locations of interesting voxels in an image.
This is not straightforward, as we are computing similarity of neighborhoods of
18
images, which are obtained through function calls, not through relations.
Q8
Return id’s, images, and locations for images that have
similar neighborhoods of a given size at that location
to neighborhoods of the query image thisimg at location
thisloc of the same given size.
User query:
thisimg
select A.id, A.img, I.loc
from fMRI A, intloc I
where A.fid = I.fid and
A
simnbd(nbd(A.img, I.loc, size),
nbd(thisimg, thisloc, size))
thisimg thisloc fMRI
fid id img
intloc
fid loc prob
simnbd
img nbd prob
The intloc relation contains the locations of interesting voxels in an image.
This is not straightforward, as we are computing similarity of neighborhoods of
19
images, which are obtained through function calls, not through relations.
Q9
Return id’s, images and pattern locations of database images
that have an interesting pattern at a location that is similar to
one of the locations of language sites in the same
patient’s SUR data.
select A.id,A.img, I.loc
from fMRI A, SUR S, intloc I
where A.img = I.img and
simloc(I.loc, S.loc)
fMRI
id img
SUR
id loc
intloc
img loc
simloc
loc loc prob
20
Q9
Return id’s, images and pattern locations of database images
that have an interesting pattern at a location that is similar to
one of the locations of language sites in the same
patient’s SUR data.
User query:
fMRI
id img
select A.id,A.img, I.loc
from fMRI A, SUR S, intloc I
where l.fid = A.fid
and simloc(I.loc, S.loc)
SUR
intloc
simloc
id loc fid loc prob loc loc prob
21
Q10
We don’t understand this query yet
Return id’s, images and pattern locations of database images
that have an interesting pattern at a location that is similar to
one of the locations of language sites in the same
patient’s SUR data and whose fMRI signal at the fMRI location
has a similar peak pattern to the SUR signal at its location.
select A.id,A.img, I.loc, S.loc
from fMRI A, SUR S, intloc I
where A.img = I.img and A.id = S.id and
simloc(I.loc, S.loc) and
simpeaks(signal(A.img,I.loc),S.signal)
fMRI
SUR
intloc
simloc
id img
id loc signal
img loc
loc loc prob
simpeaks
22
signal signal prob