Provenance as
Dependency Analysis
Provenance in Databases Workshop
May 21, 2007
James Cheney
University of Edinburgh
Amal Ahmed Umut Acar
Toyota Technological Institute, Chicago
What is provenance?
●
●
“Where did that result come from?”
–
Where-provenance, annotation propagation
–
Idea: Trace sequence of “copies”
“Why is that result in the output?”
–
Why-provenance, lineage
–
Idea: Trace data on which output “depends”
Previous work
●
●
Lots!
–
“Polygen” model [Wang, Madnick '90]
–
Lineage [Cui, Widom, Wiener '00]
–
Why- and where- [Buneman Khanna Tan '01]
–
Copy-paste [Buneman, Chapman, C,
Vansummeren '06, '07]
–
Semirings [Green, Karvounarakis, Tannen '07]
(Why) Aren't
we done?
Why we're not happy
There is a big gap between the strong
informal motivations/descriptions
“relevant”, “depends on”, “comes from”, "causes"
and the formal definitions of
previous work on provenance.
I find most of them somewhat ad hoc.
including the ones I've worked on.
No offense!
What is provenance for?
●
Understand/debug query
–
●
Show what data a result “relies on”
–
●
What we must trust to believe result
Show where errors in result “come from”
–
●
show what query is “really” doing
What to blame if there is an error
Propagate annotations on input to
“relevant” output
What else is good for similar
problems?
●
Program dependence graphs
–
●
Information flow analysis
–
●
for tracking data secrecy and integrity
Program slicing
–
●
for understanding/debugging programs
for identifying parts of program on which
wrong output depends
Key idea: dependency analysis
Goal
●
●
Explore connection to PL ideas
–
dependency
–
information flow
–
program slicing
Improve understanding of provenance
–
How to generalize to richer query languages?
–
What formal guarantees can/should it
provide?
Simple dependency analysis
w=5
x=1
y=0
z=1
x := y;
y := 12;
z := w + y;
w=5
x=0
y = 12
z = 17
Simple dependency analysis
w=5
If we change
part of input...
x=1
y=0
z=1
x := y;
y := 12;
z := w + y;
w=5
x=0
y = 12
z = 17
Simple dependency analysis
w=5
If we change
part of input...
x=1
y=0
x := y;
z=1
What part of
output might
change?
y := 12;
z := w + y;
w=5
x=0
y = 12
z = 17
Simple dependency analysis
w=5
x=1
y=0
z=1
Trace back to find
last “use”
(assignment)
x := y;
y := 12;
z := w + y;
w=5
x=0
y = 12
z = 17
Simple dependency analysis
w=5
x=1
y=0
x := y;
z=1
Use-def
Make use-def
chainschains
y := 12;
z := w + y;
w=5
x=0
y = 12
z = 17
Simple dependency analysis
w=5
x=1
y=0
x := y;
z=1
Now simplify
y := 12;
z := w + y;
w=5
x=0
y = 12
z = 17
Simple dependency analysis
w=5
x=1
y=0
x := y;
z=1
Now simplify
y := 12;
z := w + y;
w=5
x=0
y = 12
z = 17
Simple dependency analysis
w=5
x=1
y=0
z=1
f
g
c
h
w=5
x=0
y = 12
z = 17
This dependence information correctly
approximates program behavior if:
there exists f, g, h, c such that
w' = f(w) x' = g(y) y' = c z' = h(w,y)
Dependency analysis: if-then
w=5
x=1
y=0
x := y;
if (z > 0) x := w;
else y := z;
z=1
Dependency analysis: if-then
w=5
x=1
y=0
z=1
x := y;
if (z > 0) x := w;
else y := z;
w=5
x=5
y=0
z=1
Dependency analysis: if-then
w=5
x=1
y=0
z = -42
x := y;
if (z > 0) x := w;
else y := z;
w=5
x=0
y = -42
z = -42
Dependency analysis: if-then
Control dependences
needed to reflect
w=5
x=1
alternative
branch
y=0
z=1
x := y;
if (z > 0) x := w;
else y := z;
w=5
x=5
y=1
z=1
Dependency analysis: if-then
●
w=5
x=1
y=0
z=1
w=5
x=5
y=1
z=1
To get all dependences,
–
need to consider what did happen
–
but also what might have happened
Extending to DBs: main idea
●
●
Given input DB, query Q
Link each “part” p of input to all “parts”
of output that depend on p
–
Intuition: If we change DB at p, then linked
parts may change
–
The rest of the DB not depending on p must
not change if we change the DB at p.
1000 words
Q
DB
p'
Q(DB)
1000 words
q3
Q
DB
p'
Q(DB)
q1
q2
1000 words
q3
Q
DB
p'
Provenance
links are
“dependency
correct” if...
Q(DB)
q1
q2
1000 words
q3
Q
DB
q1
p'
Whenever DB
and DB' are
“the same
except at p”
Q(DB)
DB'
p'
q2
1000 words
Q
DB
Q(DB)
q1
p'
Whenever DB
and DB' are
“the same
except at p”
Then Q(DB)
and Q(DB')
are “the
same except
q3' at q ,...,q ”
1
n
q2
q3'
DB'
p'
Q
Q(DB')
q1'
q2'
Easier said than done.
●
Several challenges
–
How do we address “parts” of DB?
–
What is a “change”?
–
What do we mean by “same place” in two
query results on changed input?
●
Gloss over here, details in paper.
●
Show basic idea using pictures
–
Much easier and more fun
–
but hard to formalize...
Selection
R
1
2
3
1
3
4
2
3
4
A=1(R)
1
2
3
1
3
4
Selection
R
1
42 3
1
3
4
2
3
4
Change here
Visible here
A=1(R)
1
42 3
1
3
4
Selection
R
1
42 3
1
3
4
2
3
4
A=1(R)
1
42 3
1
3
4
Selection
R
42 2
3
1
3
4
2
3
4
Visible here
Change here
A=1(R)
1
3
4
Selection
R
42 2
3
1
3
4
2
3
4
A=1(R)
1
3
4
Selection
R
1
2
3
1
3
4
1
3
4
Change here
Visible here
A=1(R)
1
2
3
1
3
4
1
3
4
Selection
R
1
2
3
“Data”
dependences
1
3
4
2
3
4
“Control”
dependences
A=1(R)
1
2
3
1
3
4
Projection
R
1
2
3
1
3
4
2
3
Only data
dependences
4
23(R)
2
3
3
4
3
4
Join
R
1
S
2
1
3
2
3
2
3
Mix of
4 3 4
dependences
2
R JOIN S
1
2
3
1
2
4
1
3
4
2
3
4
Union
R
1
S
2
1
3
2
3
2
3
2
Only data
4 4
dependences
4
RUS
1
2
1
3
2
3
2
3
2
4
4
4
Difference
R
1
S
2
1
3
2
3
2
R-S
1
2
1
3
3
2 4 control
3 4
Only
dependences
Grouping/aggregation
R
1
Data
dependence on
aggregated
field
2
1
3
2
3
1
5
2
Control
dependence on
grouped field
SUM
3
What is actually in the paper
●
●
●
Precise definition & formalization of
dependency correctness for NRC queries
Proof that “minimal” dependency
provenance noncomputable
Define provenance semantics that is
dependency-correct
–
●
but might be "inaccurate"
Define static approximation
–
via type-based analysis
Future work
●
●
Negation, = handled inelegantly
–
As in most/all other techniques
–
“Equal except for” is too strong
–
Can we do better?
Implementation
–
Prototype implementation now
–
How to scale dynamic tracking to large DBs?
Conclusions
●
●
Dependence analysis
–
provides a solid foundation for provenance
–
can track data, row, table dependences
–
can deal with grouping, aggregation
Minimizing nontrivial (undecidable)
–
●
Introduced dynamic and static approximation
techniques
Lots of possible variations to explore!