Program Synthesis for
Computer-aided Education
Sumit Gulwani
sumitg@microsoft.com
Microsoft Research, Redmond
August 2013
Marktoberdorf Summer School Lectures: Part 3
Potential Users of Synthesis Technology
Algorithm
Designers
Software Developers
Most Useful
Target
Most
Transformational
Target
End-Users
Students and Teachers
• Vision for End-users: Enable people to have (automated)
personal assistants.
• Vision for Education: Enable interactive & personalized
Education.
1
Connections between End-User Programming
& Computer-aided Education
• End-User teaches Computer & Computer uses
that knowledge to automate repetitive tasks.
Teacher teachers Computer & Computer uses
that knowledge to teach Students.
• Interactivity between End-User & Computer
to resolve ambiguities.
Interactivity between Student & Computer
to resolve student’s confusions.
Implications:
• End-User Programming research can fuel
Computer-aided Education research.
• And vice-versa!
2
Computer-aided Education
Various tasks
• Problem Synthesis
• Solution Synthesis
• Feedback Synthesis
• Content Authoring
Various subject-domains
• Arithmetic, Algebra, Geometry
• Programming, Automata, Logic
• Board Games, Language Learning
• ...
Reference: Example Based Learning in Computer-aided STEM Education
MSR-TR-2013-50, Gulwani.
3
Content Classification
• Procedural
– Mathematical Procedures
• Addition, Long division, GCD/LCM, Gaussian Elimination
– Algorithmic Procedures
• Students asked to show understanding of classical algorithms
on specific inputs.
– BFS, insertion sort, shortest path
– translating regular expression into an automaton.
• Conceptual
– Proofs
• Algebraic theorems, Natural deduction, Non-regularity
– Constructions
• Geometric ruler/compass based constructions, Automata
constructions, Algorithms
4
Problem Synthesis
Problem Synthesis
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent cheating in MOOCs (Unsynchronized instruction)
• Problems of a given difficulty level and concept usage.
– Generate progressions
– Generate personalized workflows
Key Ideas
 Procedural Content: Test input generation techniques
6
Problem Synthesis: Addition Procedure
Concept
Single digit addition
Multiple digit w/o carry
Single carry
Two single carries
Double carry
Triple carry
Extra digit in i/p & new digit in o/p
Reference: A Trace-based Framework for Analyzing and Synthesizing Educational
Progressions; CHI 2013; Andersen, Gulwani, Popovic.
7
Problem Synthesis: Addition Procedure
Concept
Trace Characteristic
Single digit addition
L
Multiple digit w/o carry
LL+
Single carry
L* (LC) L*
Two single carries
L* (LC) L+ (LC) L*
Double carry
L* (LCLC) L*
Triple carry
L* (LCLCLCLC) L*
Extra digit in i/p & new digit in o/p L* CLDCE
Reference: A Trace-based Framework for Analyzing and Synthesizing Educational
Progressions; CHI 2013; Andersen, Gulwani, Popovic.
8
Problem Synthesis: Addition Procedure
Concept
Trace Characteristic
Sample Input
Single digit addition
L
3+2
Multiple digit w/o carry
LL+
1234 +8765
Single carry
L* (LC) L*
1234 + 8757
Two single carries
L* (LC) L+ (LC) L*
1234 + 8857
Double carry
L* (LCLC) L*
1234 + 8667
Triple carry
L* (LCLCLCLC) L*
1234 + 8767
Extra digit in i/p & new digit in o/p L* CLDCE
9234 + 900
Reference: A Trace-based Framework for Analyzing and Synthesizing Educational
Progressions; CHI 2013; Andersen, Gulwani, Popovic.
9
Comparing Progressions: Integer Comparison
Green Progression: Jump Math
Blue Progression: Skill Sharpeners
Green Progression moves (quickly) into involved
problems that involve comparing more digits.
But it ignore an entire class of levels (H and L)!
10
Problem Synthesis
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent cheating in MOOCs (Unsynchronized instruction)
• Problems of a given difficulty level and concept usage.
– Generate progressions
– Generate personalized workflows
Key Ideas
• Procedural Content: Test input generation techniques
• Conceptual Content
 Template based generalization
11
Trigonometry Problem
Example Problem: sec 𝑥 + cos 𝑥
Query: 𝑇1 𝑥 ± 𝑇2 (𝑥)
𝑇1 ≠ 𝑇5
sec 𝑥 − cos 𝑥 = tan2 𝑥 + sin2 𝑥
𝑇3 𝑥 ± 𝑇4 𝑥
= 𝑇52 𝑥 ± 𝑇62 (𝑥)
New problems generated:
csc 𝑥 + cos 𝑥 csc 𝑥 − cos 𝑥 = cot 2 𝑥 + sin2 𝑥
(csc 𝑥 − sin 𝑥)(csc 𝑥 + sin 𝑥) = cot 2 𝑥 + cos 2 𝑥
(sec 𝑥 + sin 𝑥)(sec 𝑥 − sin 𝑥) = tan2 𝑥 + cos 2 𝑥
:
(tan 𝑥 + sin 𝑥)(tan 𝑥 − sin 𝑥) = tan2 𝑥 − sin2 𝑥
(csc 𝑥 + cos 𝑥)(csc 𝑥 − cos 𝑥) = csc 2 𝑥 − cos 2 𝑥
:
Reference: Automatically generating algebra problems;
AAAI 2012; Rohit Singh, Sumit Gulwani, Sriram Rajamani.
12
Trigonometry Problem
Example Problem: sec 𝑥 + cos 𝑥
Query: 𝑇1 𝑥 ± 𝑇2 (𝑥)
𝑇1 ≠ 𝑇5
sec 𝑥 − cos 𝑥 = tan2 𝑥 + sin2 𝑥
𝑇3 𝑥 ± 𝑇4 𝑥
= 𝑇52 𝑥 ± 𝑇62 (𝑥)
New problems generated:
csc 𝑥 + cos 𝑥 csc 𝑥 − cos 𝑥 = cot 2 𝑥 + sin2 𝑥
(csc 𝑥 − sin 𝑥)(csc 𝑥 + sin 𝑥) = cot 2 𝑥 + cos 2 𝑥
(sec 𝑥 + sin 𝑥)(sec 𝑥 − sin 𝑥) = tan2 𝑥 + cos 2 𝑥
:
(tan 𝑥 + sin 𝑥)(tan 𝑥 − sin 𝑥) = tan2 𝑥 − sin2 𝑥
(csc 𝑥 + cos 𝑥)(csc 𝑥 − cos 𝑥) = csc 2 𝑥 − cos 2 𝑥
:
Reference: Automatically generating algebra problems;
AAAI 2012; Rohit Singh, Sumit Gulwani, Sriram Rajamani.
13
Limits/Series Problem
𝑛
Example Problem:
𝑛
Query:
lim
𝑛→∞
𝑖=0
lim
𝑛→∞
𝑖=0
2𝑖 2 + 𝑖 + 1
5
=
𝑖
2
5
𝐶0 𝑖 2 + 𝐶1 𝑖 + 𝐶2
𝐶3 𝑖
𝐶4
=
𝐶5
C0 ≠ 0 ∧ gcd 𝐶0 , 𝐶1 , 𝐶2 = gcd 𝐶4 , 𝐶5 = 1
New problems generated:
𝑛
lim
𝑛→∞
𝑖=0
𝑛
lim
𝑛→∞
𝑖=0
𝑛
2
3𝑖 + 2𝑖 + 1
7
=
𝑖
3
7
lim
𝑛→∞
𝑛
2
𝑖
3
=
𝑖
2
3
𝑖=0
lim
𝑛→∞
𝑖=0
3𝑖 2 + 3𝑖 + 1
=4
𝑖
4
5𝑖 2 + 3𝑖 + 3
=6
𝑖
6
14
Integration Problem
Example Problem:
Query:
(csc 𝑥) (csc 𝑥 − cot 𝑥) 𝑑𝑥 = csc 𝑥 − cot 𝑥
𝑇0 𝑥 𝑇1 𝑥 ± 𝑇2 𝑥 𝑑𝑥 = 𝑇4 𝑥 ± 𝑇5 (𝑥)
𝑇1 ≠ 𝑇2 ∧ 𝑇4 ≠ 𝑇5
New problems generated:
(tan 𝑥) (cos 𝑥 + sec 𝑥) 𝑑𝑥 = sec 𝑥 − cos 𝑥
(sec 𝑥) (tan 𝑥 + sec 𝑥) 𝑑𝑥 = sec 𝑥 + cot 𝑥
(cot 𝑥) (sin 𝑥 + csc 𝑥) 𝑑𝑥 = sin 𝑥 − csc 𝑥
15
Determinant Problem
Ex. Problem
𝑥+𝑦
𝑧𝑥
𝑦𝑧
2
𝑧𝑥
𝑦+𝑧
𝑥𝑦
𝐹0 (𝑥, 𝑦, 𝑧) 𝐹1 (𝑥, 𝑦, 𝑧)
Query 𝐹3 (𝑥, 𝑦, 𝑧) 𝐹4 (𝑥, 𝑦, 𝑧)
𝐹6 (𝑥, 𝑦, 𝑧) 𝐹7 (𝑥, 𝑦, 𝑧)
2
𝑧𝑦
𝑥𝑦
𝑧+𝑥
= 2𝑥𝑦𝑧 𝑥 + 𝑦 + 𝑧
3
2
𝐹2 (𝑥, 𝑦, 𝑧)
𝐹5 (𝑥, 𝑦, 𝑧)
𝐹8 (𝑥, 𝑦, 𝑧)
= 𝐶10 𝐹9 (𝑥, 𝑦, 𝑧)
𝐹𝑖 ≔ 𝐹𝑗 𝑥 → 𝑦; 𝑦 → 𝑧; 𝑧 → 𝑥 𝑤ℎ𝑒𝑟𝑒 𝑖, 𝑗 ∈ { 4,0 , 8,4 , 5,1 , … }
New problems generated:
𝑦2
𝑧+𝑦
𝑧2
2
𝑦𝑧 + 𝑦 2
𝑦𝑧
𝑧𝑥
𝑥2
𝑧2
𝑥+𝑧
𝑥𝑦
𝑧𝑥 + 𝑧 2
𝑧𝑥
2
𝑦+𝑥
𝑦2
𝑥2
𝑥𝑦
𝑦𝑧
𝑥𝑦 + 𝑥 2
2
= 2 𝑥𝑦 + 𝑦𝑧 + 𝑧𝑥
3
= 4𝑥 2 𝑦 2 𝑧 2
16
Randomized Polynomial Identity Testing
• Problem: Given two polynomials P1 and P2, determine
whether they are equivalent.
• The naïve deterministic algorithm of expanding
polynomials to compare them term-wise is exponential.
• A simple randomized test is probabilistically sufficient:
– Choose random values r for polynomial variables x
– If P1(r) ≠ P2(r), then P1 is not equivalent to P2.
– Otherwise P1 is equivalent to P2 with high probability.
Proof:
• by induction over number of polynomial variables.
• Base case: If P1 ≡ P2, then P1-P2 has at most d zeros
where d = max(degree(P1), degree(P2)).
17
Randomized Identity Testing of Analytic Functions
Let 𝑓 𝑋 , 𝑔(𝑋) be non-identical real-valued analytic fns over 𝑅𝑛 .
Let 𝑌 ∈ 𝑅𝑛 be selected uniformly at random.
Then, with high probability over this selection, 𝑓 𝑌 ≠ 𝑔 𝑌 .
Proof:
• Analytic fns have isolated zeroes, i.e., for every zero point of
an analytic fn, there exists a neighborhood in which the fn is
non-zero.
• Hence, # of non-zero points of the non-zero analytic fn
𝑓 𝑋 − 𝑔(𝑋) dominates # of zero points.
18
Problem Synthesis
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent cheating in MOOCs (Unsynchronized instruction)
• Problems of a given difficulty level and concept usage.
– Generate progressions
– Generate personalized workflows
Key Ideas
• Procedural Content: Test input generation techniques
• Conceptual Content
 Template based generalization
19
Problem Synthesis: Sentence Completion
1. The principal characterized his pupils as _________
they were pampered and spoiled by their indulgent parents.
becaus
2. The commentator characterized the electorate as _________ becaus
was unpredictable and given to constantly shifting moods.
(a) cosseted
(b) disingenuous
(c) corrosive
(d) laconic
(e) mercurial
One of the problems is a real problem from SAT (standardized US exam),
while the other one was automatically generated!
From problem 1, we generate template T1: *1 characterized *2 as *3 because *4
We specialize T1 to template T2: *1 characterized *2 as mercurial because *4
Problem 2 is an instance of T2.
Problem Synthesis
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent cheating in MOOCs (Unsynchronized instruction)
• Problems of a given difficulty level and concept usage.
– Generate progressions
– Generate personalized workflows
Key Ideas
• Procedural Content: Test input generation techniques
• Conceptual Content
– Template based generalization
 Symbolic methods (solution generation in reverse)
21
Natural Deduction
Prove that: 𝑥1 ∨ 𝑥2 ∧ 𝑥3
and 𝑥1 → 𝑥4
and 𝑥4 → 𝑥5
implies 𝑥2 ∨ 𝑥5
Inference Rule
Premises
Conclusion
Modus Ponens (MP)
𝑝 → 𝑞, 𝑝
𝑞
Hypothetical Syllogism (HS)
𝑝 → 𝑞, 𝑞 → 𝑝
𝑝→𝑟
Disjunctive Syllogism (DS)
𝑝 ∨ 𝑞, ¬𝑝
𝑞
Simplification (Simp)
𝑝∧𝑞
𝑞
Replacement Rule
Proposition
Equiv. Proposition
Distribution
𝑝 ∨ (𝑞 ∧ 𝑟)
Double Negation
𝑝
¬¬𝑝
Implication
𝑝→𝑞
¬𝑝 ∨ 𝑞
Equivalence
𝑝≡𝑞
𝑝 ∨ 𝑞 ∧ (𝑝 ∨ 𝑟)
𝑝 → 𝑞 ∧ (𝑞 → 𝑝)
Reference: Automatically Generating Problems and Solutions for Natural Deduction
22
IJCAI 2013; Umair Ahmed, Sumit Gulwani, Amey Karkare
Similar Problem Synthesis: Natural Deduction
Similar Problems = those that have a minimal proof with
the same sequence of inference rules as used by a
minimal proof of given problem.
Premise 1
Premise 2
Premise 3
Conclusion
𝑥1 ∨ (𝑥2 ∧ 𝑥3 )
𝑥1 → 𝑥4
𝑥4 → 𝑥5
𝑥2 ∨ 𝑥5
Similar Problems
Premise 1
Premise 2
Premise 3
Conclusion
𝑥1 ≡ 𝑥2
𝑥3 → ¬𝑥2
(𝑥4 → 𝑥5 ) → 𝑥3
𝑥1 → (𝑥𝑦 ∧ ¬𝑥5 )
𝑥1 ∧ (𝑥2 → 𝑥3 )
𝑥1 ∨ 𝑥4 → ¬𝑥5 𝑥2 ∨ 𝑥5
(𝑥1 ∨ 𝑥2 ) → 𝑥3
𝑥3 → 𝑥1 ∧ 𝑥4
(𝑥1 → 𝑥2 ) → 𝑥3
𝑥3 → ¬𝑥4
𝑥1 → 𝑥2 ∧ 𝑥3
𝑥4 → ¬𝑥2
𝑥1 ∧ 𝑥4 → 𝑥5
𝑥1 ∨ 𝑥5 ∨ 𝑥4
𝑥3 ≡ 𝑥5 → 𝑥4
𝑥1 ∨ 𝑥4 ∧ ¬𝑥5
𝑥1 → 𝑥5
𝑥5 ∨ 𝑥2 → 𝑥1
𝑥1 → 𝑥3 ≡ ¬𝑥5
23
Parameterized Problem Synthesis: Natural Deduction
Parameters:
# of premises = 3, Size of propositions ≤ 4
# of variables = 3, # of inference steps = 2
Inference rules = { DS, HS }
Parameterized Problems
Premise 1
Premise 2
Premise 3 Conclusion
(𝑥1 → 𝑥3 ) → 𝑥2
𝑥2 → 𝑥3
¬𝑥3
𝑥1 ∧ ¬𝑥3
𝑥3 ≡ 𝑥1 → 𝑥2 ¬𝑥2
𝑥1 ∧ ¬𝑥3
𝑥3 → 𝑥1
𝑥1 ≡ 𝑥3 ∨ 𝑥1 ≡ 𝑥2
(𝑥1 ≡ 𝑥2 ) → 𝑥3
¬𝑥3
𝑥1 ≡ 𝑥3
𝑥1 ≡ ¬𝑥3
𝑥2 ∨ 𝑥1
𝑥3 → ¬𝑥2
𝑥1 ∧ ¬𝑥3
𝑥3 → 𝑥1
𝑥1 → 𝑥2 ∧ 𝑥3
𝑥3 → ¬𝑥2
¬𝑥3
24
Problem Synthesis
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent cheating in MOOCs (Unsynchronized instruction)
• Problems of a given difficulty level and concept usage.
– Generate progressions
– Generate personalized workflows
Key Ideas
• Procedural Content: Test input generation techniques
• Conceptual Content
– Template based generalization
 Symbolic methods (solution generation in reverse)
25
Problem Generation: Board Games
Game Rules
• allows variants of classic games.
Tic-Tac-Toe
4*4
Row/Column only
Expertise level of player/opponent
• allows leveling the playing field.
Reference: Automatic Generation of Starting Positions in Board Games;
Technical Report; Umair Ahmed, Krishnendu Chatterjee, Sumit Gulwani.
Solution Synthesis
Solution Synthesis
Motivation
• Synthesize sample solutions to new problems.
• Synthesize customized solutions or complete unfinished ones.
• Inform solution characteristics during problem generation.
Key Ideas
 Procedural Content: Programming by Demonstration
28
Solution Synthesis: GCF Computation
Time-stamped sequence
of (location, value) pairs
Programming by
Demonstration
Spreadsheet
Program
Can also be used to synthesize incorrect procedures from buggy traces!
Reference: Programming by Demonstration framework applied to procedural math
29
problems; Technical Report; Andersen, Gulwani, Popovic.
Solution Synthesis
Motivation
• Synthesize sample solutions to new problems.
• Synthesize customized solutions or complete unfinished ones.
• Inform solution characteristics during problem generation.
Key Ideas
• Procedural Content: Programming by Demonstration
 Conceptual Content
– Perform reasoning over models (vs. abstract symbolic reasoning).
– Reduce solution space to solutions with small length.
30
Ruler/Compass based Geometry Constructions
Given a triangle XYZ,
construct circle C such that
C passes through X, Y, and Z.
C
L1
L2
N
Z
X
Reference: Synthesizing Geometry Constructions
PLDI 2011; Gulwani, Korthikanti, Tiwari
Y
31
Other Examples of Geometry Constructions
• Draw a regular hexagon given a side.
• Given 3 parallel lines, draw an equilateral triangle
whose vertices lie on the parallel lines.
• Given 4 points, draw a square whose sides contain
those points.
32
Significance
• Good platform for teaching logical reasoning.
– Visual Nature:
• Makes it more accessible.
• Exercises both logical/visual abilities of left/right brain.
– Fun Aspect:
• Ruler/compass restrictions make it fun, as in sports.
• Application in dynamic geometry or animations.
– “Constructive” geometry macros (unlike numerical
methods) enable fast re-computation of derived objects
from free (moving) objects.
33
Programming Language for Geometry Constructions
Types: Point, Line, Circle
Methods:
• Ruler(Point, Point) -> Line
• Compass(Point, Point) -> Circle
• Intersect(Circle, Circle) -> Pair of Points
• Intersect(Line, Circle) -> Pair of Points
• Intersect(Line, Line) -> Point
Geometry Program: A straight-line composition of
the above methods.
34
Example: Geometry Program
Given a triangle XYZ, construct circle C such that C
passes through X, Y, and Z.
1. C1 = Compass(X,Y);
2. C2 = Compass(Y,X);
3. <P1,P2> = Intersect(C1,C2);
4. L1 = Ruler(P1,P2);
5. D1 = Compass(Z,X);
6. D2 = Compass(X,Z);
7. <R1,R2> = Intersect(D1,D2);
8. L2 = Ruler(R1,R2);
9. N = Intersect(L1,L2);
10. C = Compass(N,X);
C
L1
N
C1
D1
R1
Z
R2
P1
X
D2
L2
C2
Y
P2
35
Specification Language for Geometry Programs
Conjunction of predicates over arithmetic expressions
Predicates
p := e1 = e2
| e1  e2
| e1 · e2
Arithmetic Expressions
e :=
|
|
|
Distance(Point, Point)
Slope(Point, Point)
e 1 § e2
c
36
Example: Precondition/Postcondition
Given a triangle XYZ, construct circle C such that C
passes through X, Y, and Z.
Precondition:
Slope(X,Y)  Slope(X,Z) Æ Slope(X,Y)  Slope(Z,X)
Postcondition:
LiesOn(X,C) Æ LiesOn(Y,C) Æ LiesOn(Z,C)
Where LiesOn(X,C) ´
Distance(X,Center(C)) = Radius(C)
37
Verification/Synthesis Problem for Geometry Programs
• Let P be a geometry program that computes
outputs O from inputs I.
• Verification Problem: Check the validity of the
following Hoare triple.
Assume Pre(I);
P
Assert Post(I,O);
• Synthesis Problem: Given Pre(I), Post(I,O), find P
such that the above Hoare triple is valid.
38
Approaches to Verification Problem
Pre(I), P, Post(I,O)
a) Symbolic decision procedures are complex.
39
Approaches to Verification Problem
Pre(I), P, Post(I,O)
a) Symbolic decision procedures are complex.
b)
1.
2.
3.
New efficient approach: Random Testing!
Choose I’ randomly from the set { I | Pre(I) }.
Compute O’ := P(I’).
If O’ satisfies Post(I’,O’) output “Verified”.
Correctness Proof of (b):
• Objects constructed by P can be described using
polynomial ops (+,-,*), square-root & division operator.
• The randomized polynomial identity testing algorithm
lifts to square-root and division operators as well !
40
Idea 1 (from Theory): Symbolic Reasoning -> Concrete
Synthesis Algorithm:
// First obtain a random input-output example.
1. Choose I’ randomly from the set { I | Pre(I) }.
2. Compute O’ s.t. Post(I’,O’) using numerical methods.
// Now obtain a construction that can generate O’ from I’
(using exhaustive search).
3. S := I’;
4. While (S does not contain O’)
5.
S := S [ { M(O1,O2) | Oi 2 S, M 2 Methods }
6. Output construction steps for O’.
41
Error Probability of the algorithm is extremely low.
Given a triangle XYZ, construct circle C such that C
passes through X, Y, and Z.
C
L1
…
L1 = Ruler(P1,P2);
N
…
L2 = Ruler(R1,R2);
N = Intersect(L1,L2);
Z
C = Compass(N,X);
X
L2
Y
• For an equilateral 4XYZ, incenter
coincides with circumcenter N.
• But what are the chances of choosing a
random 4XYZ to be an equilateral one?
42
Idea 2 (from PL): High-level Abstractions
Synthesis algorithm times out because programs are large.
• Identify a library of commonly used patterns (pattern =
“sequence of geometry methods”)
– E.g., perpendicular/angular bisector, midpoint, tangent, etc.
S := S [ { M(O1,O2) | Oi 2 S, M 2 Methods }
S := S [ { M(O1,O2) | Oi 2 S, M 2 LibMethods }
• Two key advantages:
– Search space: large depth -> small depth
– Easier to explain solutions to students.
43
Use of high-level abstractions reduces program size
Given a triangle XYZ, construct circle C such that C
passes through X, Y, and Z.
1. C1 = Compass(X,Y);
2. C2 = Compass(Y,X);
3. <P1,P2> = Intersect(C1,C2);
4. L1 = Ruler(P1,P2);
5. D1 = Compass(Z,X);
6. D2 = Compass(X,Z);
7. <R1,R2> = Intersect(D1,D2);
8. L2 = Ruler(R1,R2);
9. N = Intersect(L1,L2);
10. C = Compass(N,X);
1.
2.
3.
4.
L1 = PBisector(X,Y);
L2 = PBisector(X,Z);
N = Intersect(L1,L2);
C = Compass(N,X);
44
Idea 3 (from AI): Goal Directed Search
Synthesis algorithm is inefficient because the search space
is too wide and hence still huge.
• Prune forward search by using A* style heuristics.
S := S [ { M(O1,O2) | Oi 2 S, M 2 LibMethods }
S := S [ {M(O1,O2) | Oi2S, M2LibMethods, IsGood(M(O1,O2)) }
• Example: If a method constructs a line L that passes
through a desired output point, then L is “good” (i.e.,
worth constructing).
45
Effectiveness of Goal-directed search
Given a triangle XYZ,
construct circle C such that
C passes through X, Y, and Z.
C
L1
L2
N
Z
X
Y
• L1 and L2 are immediately constructed since
they pass through output point N.
• On the other hand, other lines like angular
bisectors are not eagerly constructed.
46
Experimental Results
25 benchmark problems.
• Performance: 18 problems less than 1 second.
4 problems between 1-3 seconds.
3 problems 13-82 seconds.
• Idea 2 (high-level abstractions) reduces programs
of size 3-45 to 3-13.
• Idea 3 (goal-directedness) improves performance
by factor of 10-1000 times on most problems.
47
Search space Exploration: With/without goal-directness
48
Solution Synthesis
Motivation
• Synthesize sample solutions to new problems.
• Synthesize customized solutions or complete unfinished ones.
• Inform solution characteristics during problem generation.
Key Ideas
• Procedural Content: Programming by Demonstration
 Conceptual Content
– Perform reasoning over models (vs. abstract symbolic reasoning).
– Reduce solution space to solutions with small length.
49
Natural Deduction
Prove that: 𝑥1 ∨ 𝑥2 ∧ 𝑥3
and 𝑥1 → 𝑥4
and 𝑥4 → 𝑥5
implies 𝑥2 ∨ 𝑥5
Inference Rule
Premises
Conclusion
Modus Ponens (MP)
𝑝 → 𝑞, 𝑝
𝑞
Hypothetical Syllogism (HS)
𝑝 → 𝑞, 𝑞 → 𝑝
𝑝→𝑟
Disjunctive Syllogism (DS)
𝑝 ∨ 𝑞, ¬𝑝
𝑞
Simplification (Simp)
𝑝∧𝑞
𝑞
Replacement Rule
Proposition
Equiv. Proposition
Distribution
𝑝 ∨ (𝑞 ∧ 𝑟)
Double Negation
𝑝
¬¬𝑝
Implication
𝑝→𝑞
¬𝑝 ∨ 𝑞
Equivalence
𝑝≡𝑞
𝑝 ∨ 𝑞 ∧ (𝑝 ∨ 𝑟)
𝑝 → 𝑞 ∧ (𝑞 → 𝑝)
Reference: Automatically Generating Problems and Solutions for Natural Deduction
50
IJCAI 2013; Umair Ahmed, Sumit Gulwani, Amey Karkare
Key Ideas
• Represent propositions using (bitvector based) truth table
representation.
– avoids expensive symbolic reasoning and reduces inference
rule applications to simple bitvector operations.
• Two-step proof search
– Abstract proof (involving only inference rules) over truthtables.
– Concretize proof (by adding replacement rules) in between
any two neighboring inference rule application.
51
Solution Synthesis: Natural Deduction
Prove that: 𝑥1 ∨ 𝑥2 ∧ 𝑥3
and 𝑥1 → 𝑥4
and 𝑥4 → 𝑥5
Step Proposition
Reason
P1
𝑥1 ∨ 𝑥2 ∧ 𝑥3
Premise
implies: 𝑥2 ∨ 𝑥5
P2
𝑥1 → 𝑥4
Premise
P3
𝑥4 → 𝑥5
Premise
1
(𝑥1 ∨ 𝑥2 ) ∧ (𝑥1 ∨ 𝑥3 ) P1, Distr.
2
𝑥1 ∨ 𝑥2
1, Simp.
3
𝑥1 → 𝑥5
P2, P3, HS
4
𝑥2 ∨ 𝑥1
2, Comm.
Step Truth
Table
Reason
P1
1048575
Premise
P2
4294914867
Premise
5
¬¬𝑥2 ∨ 𝑥1
4. Double Neg.
P3
3722304989
Premise
6
¬𝑥2 → 𝑥1
5, Implication
1
16777215
P1,Simp.
7
6,3, HS.
2
4294923605
P2,P3,HS
¬𝑥2 → 𝑥5
8
¬¬𝑥2 ∨ 𝑥5
7, Implication
Concl 1442797055 1,2,HS
Abstract Proof
Concl 𝑥2 ∨ 𝑥5
Concrete Proof
8, Double Neg.
52
Key Challenge
How to apply inference rules over truth-tables?
• Apply inference rules over propositions and project
them to truth-tables.
• Compute & store this information offline (in a datastructure that we call Universal Proof Graph).
– Space and time requirements?
• We leverage the fact that real-world problems employ
small-sized propositions.
53
# of propositions/truth-tables over n variables & size s
Number of propositions over n variables and having size at most s.
Size of proposition = number of binary operators.
Number of truth-tables over n variables and having size at most s.
Size of truth-table = minimum size of any corresponding proposition.
54
Universal Proof Graph (UPG)
Size of the UPG for various values of n and size at most 4
&
Time taken to compute the UPG.
55
Feedback Synthesis
56
Feedback Synthesis
Motivation
• Makes teachers more effective.
– Saves them time.
– Provides immediate insights on where students are struggling.
• Can enable rich interactive experience for students.
– Generation of hints.
– Pointer to simpler problems depending on kind of mistake.
Key Ideas:
 Procedural Content: Use PBE techniques to learn buggy
procedures in a student’s mind.
57
Feedback Synthesis for Procedural Problems
• Ashlock identified buggy computational patterns for
various procedures. Following are some bugs for addition.
– Add each column and write the sum below each column, even
if it is greater than nine.
– Add each column, from left to right. If the sum > 9, write
the tens digit beneath the column and the ones digit above
the column to the right.
• Van Lehn identified >100 bugs in subtraction alone.
Key Idea: Use Programming by Demonstration technology
(also used for solution synthesis for procedural problems)
for synthesizing such buggy procedures.
58
Feedback Synthesis
Motivation
• Makes teachers more effective.
– Saves them time.
– Provides immediate insights on where students are struggling.
• Can enable rich interactive experience for students.
– Generation of hints.
– Pointer to simpler problems depending on kind of mistake.
Key Ideas:
• Procedural Content: Use PBE techniques to learn buggy
procedures in a student’s mind.
• Conceptual Content: Various feedback metrics
 Counterexamples: Inputs on which the solution is not correct
59
Counterexample based Feedback: PexForFun
60
Buggy Program for Array Reverse
using System;
6:28::50 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
61
Buggy Program for Array Reverse
using System;
6:32::01 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
62
Buggy Program for Array Reverse
using System;
6:32::32 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Sign of Frustation?
63
Buggy Program for Array Reverse
using System;
6:33::19 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i <= a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
64
Buggy Program for Array Reverse
using System;
6:33::55 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
Same as initial attempt except Console.Writeline!
65
Buggy Program for Array Reverse
using System;
6:34::06 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
No change! Sign of Frustation?
66
Buggy Program for Array Reverse
using System;
6:34::56 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i <= a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
The student has tried this before!
67
Buggy Program for Array Reverse
using System;
6:36::24 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Same as initial attempt!
68
Buggy Program for Array Reverse
using System;
6:37::39 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
The student has tried this before!
69
Buggy Program for Array Reverse
using System;
6:38::11 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i > 0; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Almost correct! (a[i-1] instead of a[i] in loop body)
70
Buggy Program for Array Reverse
using System;
6:38::44 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i >= 0; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Student going in wrong direction!
71
Buggy Program for Array Reverse
using System;
6:39::33 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Back to bigger error!
72
Buggy Program for Array Reverse
using System;
6:39::45 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Frustation!
73
Buggy Program for Array Reverse
using System;
6:40::27 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! More Frustation!!
74
Buggy Program for Array Reverse
using System;
6:40::57 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Too Frustated now!!! Gives up.
75
Counterexamples are not sufficient!
"Not only did it take 1-2 weeks to grade problem, but the
comments were entirely unhelpful in actually helping us fix our
errors. …. Apparently they don't read the code -- they just ran
their tests and docked points mercilessly. What if I just had a
simple typo, but my algorithm was fine? ....“
- Student Feedback from MIT 6.00 course, 2013.
76
Feedback Synthesis
Motivation
• Makes teachers more effective.
– Saves them time.
– Provides immediate insights on where students are struggling.
• Can enable rich interactive experience for students.
– Generation of hints.
– Pointer to simpler problems depending on kind of mistake.
Key Ideas:
• Procedural Content: Use PBE techniques to learn buggy
procedures in a student’s mind.
• Conceptual Content: Various feedback metrics
– Counterexamples: Inputs on which the solution is not correct.
 Edit distance to a nearest correct solution.
77
Correction Model for Programming Assignments
Array Index:
v[a] -> v[{a+1, a-1}]
Increment:
v++ -> { ++v, v--, --v }
Conditional:
a op b -> a ops b
where ops = { <, >, <=, >=, ==, != }
Initialization:
v=n -> v={n+1, n-1, 0}
Return Value:
return v -> return ?v
Key Idea: Finding minimal set of such changes that convert a
buggy solution into a correct one is phrased as a sketch problem.
Reference: Automated Feedback Generation for Introductory Programming
Assignments; PLDI 2013; Singh, Gulwani, Solar-Lezama
78
Array Reverse
i = 1
front <= back
i <= a.Length
--back
Experimental Results
13,365 incorrect attempts for 13 Python problems.
(obtained from Introductory Programming course at
MIT and its MOOC version on the EdX platform)
• Average time for feedback = 10 seconds
• Feedback generated for 64% of those attempts.
• Reasons for failure to generate feedback
– Completely incorrect solutions
– Big conceptual errors
– Timeout (4 min)
Tool accessible at: http://sketch1.csail.mit.edu/python-autofeedback/ 80
Feedback Synthesis
Motivation
• Makes teachers more effective.
– Saves them time.
– Provides immediate insights on where students are struggling.
• Can enable rich interactive experience for students.
– Generation of hints.
– Pointer to simpler problems depending on kind of mistake.
Key Ideas:
• Procedural Content: Use PBE techniques to learn buggy
procedures in a student’s mind.
• Conceptual Content: Various feedback metrics
– Counterexamples: Inputs on which the solution is not correct.
– Edit distance to a nearest correct solution.
 Edit distance to the problem description (of a nearest
corresponding problem description).
81
Finite State Automata
Draw a DFA that accepts: { s | ‘ab’ appears in s exactly 2 times }
Grade: 9/10
Feedback: One more state
should be made final
Attempt 1
Based on Edit-distance to a correct solution
Grade: 6/10
Feedback: The DFA is
incorrect on the string ‘ababb’
Attempt 2
Based on counterexamples
Grade: 5/10
Feedback: The DFA accepts
{s | ‘ab’ appears in s at least 2 times}
Attempt 3
Based on Edit-distance to the problem description
Reference: Automated Grading of DFA Constructions,
IJCAI 2013; Alur, d’Antoni, Gulwani, Kini, Viswanathan
82
Feedback Synthesis Algorithm
Problems described in MOSEL (syntactically enriched MSO).
– Can be automatically generated from natural language.
• Counterexample
– Grade: Fn of density of counterexamples (weighted by size).
• Edit distance to nearest solution
– Grade: Fn of number & kinds of edits required to transform
the student’s automaton to a correct one.
• Edit distance to the problem description P
– Synthesize a problem description P’ matching student’s automaton.
• Brute-force search to enumerate MOSEL formulas (in size order).
• Similar to synthesis of graph algorithms (OOPSLA 2010)!
– Grade: Fn of number & kinds of edits to transform P to P’. 83
Experimental Results
800+ attempts to 6 automata problems (obtained from
automata course at UIUC) graded by tool and 2 instructors.
• 95% problems graded in <6 seconds each
• Out of 131 attempts (108 distinct) for one of those problems:
– 6 attempts: instructors were incorrect (gave full marks to an
incorrect attempt)
– 20 attempts: instructors were inconsistent (gave different
marks to syntactically equivalent attempts)
– 34 attempts: >= 3 point discrepancy between instructor and
tool; in 20 of those, instructor agreed that tool was more fair.
• More agreement between instructor/tool than instructors.
• Instructors concluded that tool should be preferred over
humans for consistency & scalability.
Tool accessible at: http://www.automatatutor.com/
84
Content Authoring
85
Content Authoring
Motivation: Make it easy to create and edit
 Equations
• Drawings
Key Ideas
• Allow multi-modal NUIs such as ink, speech, touch.
• Error correction.
• Prediction.
86
Challenges in creating Equations
Text editors like Latex
• Unreadable text in prefix notation
WYSIWIG editors like Microsoft Word
• Change of cursor positions multiple times.
• Back and forth switching between mouse & keyboard
Our solution: An intelligent predictive editor.
• Mathematical text has low entropy and hence
amenable to prediction!
Joint work with Alex Polozov and Sriram Rajamani.
87
Structure Prediction
We can predict likely structures from a flattened representation.
This can enable math-by-speech and auto-parenthesis-error-fix.
1+cos 𝐴
1−cos 𝐴
1 + cos A / 1 − cos A
((1 + cos A ) / (1 − cos A ))
Term Prediction (Syntactic Intellisense)
tan 3𝑥 tan 2𝑥 tan 𝑥 = tan 3𝑥 − tan 2𝑥 − tan 𝑥
𝑦𝑧 − 𝑥 2
𝑧𝑥 − 𝑦 2
𝑥𝑦 − 𝑧 2
𝐴1 sin3 𝛼
𝐴2 sin 𝛼
𝑧𝑥 − 𝑦 2
𝑥𝑦 − 𝑧 2
𝑦𝑧 − 𝑥 2
𝐵1 sin3 𝛽
𝐵2 sin 𝛽
𝑥𝑦 − 𝑧 2
𝑦𝑧 − 𝑥 2
𝑧𝑥 − 𝑦 2
𝐶1 sin3 𝛾
𝐶2 sin 𝛾
Our tool can infer colored text.
89
Term Prediction (Semantic Intellisense)
Prove (csc 𝑥 − sin 𝑥)(sec 𝑥 − cos 𝑥)(tan 𝑥 + cot 𝑥) = 1
L.H.S. =
1
− sin 𝑥
sin 𝑥
=
1 − sin2 𝑥
sin 𝑥
=
cos 2 𝑥
sin 𝑥
1
− cos 𝑥
cos 𝑥
1 − cos2 𝑥
cos 𝑥
sin2 𝑥
cos 𝑥
sin 𝑥 cos 𝑥
+
cos 𝑥 sin 𝑥
sin2 𝑥 + cos 2 𝑥
cos 𝑥 sin 𝑥
1
cos 𝑥 sin 𝑥
=1
90
Content Authoring
Motivation: Make it easy to create and edit
• Equations
 Drawings
Key Ideas
• Allow multi-modal NUIs such as ink, speech, touch.
• Error correction.
• Prediction.
91
Challenges in creating drawings
Sometimes require extreme precision.
• Our solution: allow pen-based sketching
and use constraint solving to beautify.
Can be tedious to draw sometimes.
Copy-Paste doesn’t address positioning of copied
objects & transformations on copied objects!
• Our Solution: use synthesis to predict
repetitive features.
Reference: QuickDraw: Improving Drawing Experience for
Geometric Diagrams; CHI 2012: Cheema, Gulwani, LaViola.
92
(Ink) Sketching meets Synthesis
Beautification
Beautification
Prediction
Prediction
Architecture
(Partial) Sketch/Ink Strokes
Sketch Recognition Engine [HCI]
Circle/Line Objects
Constraint Inference Engine [Machine Learning]
Constraints between Objects
Model Synthesis/Beautification Engine [Constraint Solving]
(Partial) Drawing
Pattern Synthesis Engine [Program Synthesis]
Suggestions for Drawing
Completion
94
Computer Aided Education: Conclusion
• Technical Perspective
– Aspects: Problem/Solution/Feedback Synthesis, Content Authoring
– Domains: Math, Programming, Logic, Language Learning, ...
• Inter-disciplinary research area
–
–
–
–
Synthesis/Search techniques
HCI
Natural language processing (for word problems)
Machine learning (for analytics)
• Value Proposition:
– Short term: Improve Education (personalized + interactive)
– Long Term: Ultra-intelligent computer, Model of human mind,
Inter-stellar travel 
95
Outline
• Part 1: From verification to synthesis
–
–
–
–
–
Bitvector algorithms (PLDI 2011, ICSE 2012)
General loopy programs (POPL 2010)
SIMD algorithms (PPoPP 2013)
Program inverses (PLDI 2011)
Graph algorithms (OOPSLA 2010)
• Part 2: End-user Programming (Examples & Natural Language)
–
–
–
–
Syntactic string transformations: Flash Fill (POPL 2011)
Semantic string transformations (VLDB 2012)
Table layout transformations (PLDI 2011)
Smartphone scripts (MobiSys 2013)
• Part 3: Computer-aided Education
–
–
–
–
Problem Synthesis (AAAI 2012, CHI 2013)
Solution Synthesis (PLDI 2011, IJCAI 2013)
Feedback Synthesis (PLDI 2013, IJCAI 2013)
Content Authoring (CHI 2012)
96
References
• Program Synthesis webpage
– http://research.microsoft.com/en-us/um/people/sumitg/pubs/synthesis.html
• Dimensions in Program Synthesis
– PPDP 2010 Keynote paper
• Programming by Example
– CACM 2012 Research highlights paper
• Programming by Natural Language
– Mobisys 2013 paper
• Education
– Example based learning in Computer-aided STEM Education;
MSR Technical Report;
http://research.microsoft.com/en-us/um/people/sumitg/pubs/education13.pdf
97
Some More Resources on Synthesis
• Bret Victor’s talks
– https://vimeo.com/71278954
– http://www.youtube.com/watch?v=PUv66718DII
• Program Sketching (Solar-Lezama and Bodik)
– http://people.csail.mit.edu/asolar/sketch2012/
• Implicit Programming (Kuncak)
– http://lara.epfl.ch/w/impro
• Eran Yahav’s Synthesis projects
– http://www.cs.technion.ac.il/~yahave/
• Synthesis Competitions
– ICFP Programming Competition: http://icfpc2013.cloudapp.net/
– CAV 2014: http://www.syntcomp.org/synthesis-competitions-at-cav-2014/
98
Dimensions & Applications
Synthesis is a hard problem that can be tamed with proper
choices for the various dimensions.
• domain-specific language: design proper abstractions incl. operators,
control-flow templates.
• intent specification: allow interactivity & under-specifications
• search techniques
There are applications where you can show impact/relevance
with creative algorithms that need not scale to large instances.
• Especially in End-User Programming and Education.
99
Search Techniques
Leverage existing paradigms/technologies
• Build over verification methodology, SMT solvers
Two alternate approaches
• Construct programs as composition of smaller programs
that solve sub-problems (Local Analysis)
– PBE, SIMD, Graph
• Program = Components + Relations (Global Analysis)
– PBNL, Bitvector, Inverses
Leverage Web, Big data & offline computation in
appropriate instances.
• for code/data search, natural language understanding,
intelligent tutoring
100
Some Interesting/Challenging Problems
• New Programming Environments structured around synthesis
– direct manipulation of data
– allow under-specifications including examples and NL
– How would a debugging environment be for such a system?
• End-User Programming of Robots
• Design an Ultra-intelligent tutoring system for a subject
domain of your choice
101