Program Synthesis for
Computer-aided Education
Sumit Gulwani
Microsoft Research, Redmond
(Joint work with several collaborators)
ExCAPE Summer School
June 2013
Connections between End-User Programming
& Computer-aided Education
• End-User teaches Computer & Computer uses
that knowledge to automate repetitive tasks.
Teacher teachers Computer & Computer uses
that knowledge to teach Students.
• Interactivity between End-User & Computer
to resolve ambiguities.
Interactivity between Student & Computer
to resolve student’s confusions.
Implications:
• End-User Programming research can fuel
Computer-aided Education research.
• And vice-versa!
1
Computer-aided Education
Various tasks
• Problem Synthesis
• Solution Synthesis
• Feedback Synthesis
• Content Authoring
Various subject-domains
• Arithmetic, Algebra, Geometry
• Programming, Automata, Logic
• Board Games, Language Learning
• ...
MSR-TR-2013-50: Formal Methods for Computer-aided STEM Education
2
Content Classification
• Procedural
– Mathematical Procedures
• Addition, Long division, GCD/LCM, Gaussian Elimination
– Algorithmic Procedures
• Students asked to show understanding of classical algorithms
on specific inputs.
– BFS, insertion sort, shortest path
– translating regular expression into an automaton.
• Conceptual
– Proofs
• Algebraic theorems, Natural deduction, Non-regularity
– Constructions
• Geometric ruler/compass based constructions, Automata
constructions, Algorithms
3
Problem Synthesis
Problem Synthesis
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent cheating in MOOCs (Unsynchronized instruction)
• Problems of a given difficulty level and concept usage.
– Generate progressions
– Generate personalized workflows
Key Ideas
 Procedural Content: Test input generation techniques
5
Problem Synthesis: Addition Procedure
Concept
Single digit addition
Multiple digit w/o carry
Single carry
Two single carries
Double carry
Triple carry
Extra digit in i/p & new digit in o/p
CHI 2013: Erik Andersen, Sumit Gulwani, Zoran Popovic.
6
Problem Synthesis: Addition Procedure
Concept
Trace Characteristic
Single digit addition
L
Multiple digit w/o carry
LL+
Single carry
L* (LC) L*
Two single carries
L* (LC) L+ (LC) L*
Double carry
L* (LCLC) L*
Triple carry
L* (LCLCLCLC) L*
Extra digit in i/p & new digit in o/p
L* CLDCE
CHI 2013: Erik Andersen, Sumit Gulwani, Zoran Popovic.
7
Problem Synthesis: Addition Procedure
Concept
Trace Characteristic Sample Input
Single digit addition
L
3+2
Multiple digit w/o carry
LL+
1234 +8765
Single carry
L* (LC) L*
1234 + 8757
Two single carries
L* (LC) L+ (LC) L*
1234 + 8857
Double carry
L* (LCLC) L*
1234 + 8667
Triple carry
L* (LCLCLCLC) L*
1234 + 8767
Extra digit in i/p & new digit in o/p
L* CLDCE
9234 + 900
CHI 2013: Erik Andersen, Sumit Gulwani, Zoran Popovic.
8
Comparing Progressions: Integer Comparison
Green Progression: Jump Math
Blue Progression: Skill Sharpeners
Green Progression moves (quickly) into involved
problems that involve comparing more digits.
But it ignore an entire class of levels (H and L)!
9
Problem Synthesis
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent cheating in MOOCs (Unsynchronized instruction)
• Problems of a given difficulty level and concept usage.
– Generate progressions
– Generate personalized workflows
Key Ideas
• Procedural Content: Test input generation techniques
• Conceptual Content
 Template based guess and verify
10
Trigonometry Problem
Example Problem: sec 𝑥 + cos 𝑥
Query: 𝑇1 𝑥 ± 𝑇2 (𝑥)
𝑇1 ≠ 𝑇5
sec 𝑥 − cos 𝑥 = tan2 𝑥 + sin2 𝑥
𝑇3 𝑥 ± 𝑇4 𝑥
= 𝑇52 𝑥 ± 𝑇62 (𝑥)
New problems generated:
csc 𝑥 + cos 𝑥 csc 𝑥 − cos 𝑥 = cot 2 𝑥 + sin2 𝑥
(csc 𝑥 − sin 𝑥)(csc 𝑥 + sin 𝑥) = cot 2 𝑥 + cos 2 𝑥
(sec 𝑥 + sin 𝑥)(sec 𝑥 − sin 𝑥) = tan2 𝑥 + cos 2 𝑥
:
(tan 𝑥 + sin 𝑥)(tan 𝑥 − sin 𝑥) = tan2 𝑥 − sin2 𝑥
(csc 𝑥 + cos 𝑥)(csc 𝑥 − cos 𝑥) = csc 2 𝑥 − cos 2 𝑥
:
AAAI 2012: Rohit Singh, Sumit Gulwani, Sriram Rajamani.
11
Trigonometry Problem
Example Problem: sec 𝑥 + cos 𝑥
Query: 𝑇1 𝑥 ± 𝑇2 (𝑥)
𝑇1 ≠ 𝑇5
sec 𝑥 − cos 𝑥 = tan2 𝑥 + sin2 𝑥
𝑇3 𝑥 ± 𝑇4 𝑥
= 𝑇52 𝑥 ± 𝑇62 (𝑥)
New problems generated:
csc 𝑥 + cos 𝑥 csc 𝑥 − cos 𝑥 = cot 2 𝑥 + sin2 𝑥
(csc 𝑥 − sin 𝑥)(csc 𝑥 + sin 𝑥) = cot 2 𝑥 + cos 2 𝑥
(sec 𝑥 + sin 𝑥)(sec 𝑥 − sin 𝑥) = tan2 𝑥 + cos 2 𝑥
:
(tan 𝑥 + sin 𝑥)(tan 𝑥 − sin 𝑥) = tan2 𝑥 − sin2 𝑥
(csc 𝑥 + cos 𝑥)(csc 𝑥 − cos 𝑥) = csc 2 𝑥 − cos 2 𝑥
:
AAAI 2012: Rohit Singh, Sumit Gulwani, Sriram Rajamani.
12
Limits/Series Problem
𝑛
Example Problem:
𝑛
Query:
lim
𝑛→∞
𝑖=0
lim
𝑛→∞
𝑖=0
2𝑖 2 + 𝑖 + 1
5
=
𝑖
2
5
𝐶0 𝑖 2 + 𝐶1 𝑖 + 𝐶2
𝐶3 𝑖
𝐶4
=
𝐶5
C0 ≠ 0 ∧ gcd 𝐶0 , 𝐶1 , 𝐶2 = gcd 𝐶4 , 𝐶5 = 1
New problems generated:
𝑛
lim
𝑛→∞
𝑖=0
𝑛
lim
𝑛→∞
𝑖=0
𝑛
2
3𝑖 + 2𝑖 + 1
7
=
𝑖
3
7
lim
𝑛→∞
𝑛
2
𝑖
3
=
𝑖
2
3
𝑖=0
lim
𝑛→∞
𝑖=0
3𝑖 2 + 3𝑖 + 1
=4
𝑖
4
5𝑖 2 + 3𝑖 + 3
=6
𝑖
6
13
Integration Problem
Example Problem:
Query:
(csc 𝑥) (csc 𝑥 − cot 𝑥) 𝑑𝑥 = csc 𝑥 − cot 𝑥
𝑇0 𝑥 𝑇1 𝑥 ± 𝑇2 𝑥 𝑑𝑥 = 𝑇4 𝑥 ± 𝑇5 (𝑥)
𝑇1 ≠ 𝑇2 ∧ 𝑇4 ≠ 𝑇5
New problems generated:
(tan 𝑥) (cos 𝑥 + sec 𝑥) 𝑑𝑥 = sec 𝑥 − cos 𝑥
(sec 𝑥) (tan 𝑥 + sec 𝑥) 𝑑𝑥 = sec 𝑥 + cot 𝑥
(cot 𝑥) (sin 𝑥 + csc 𝑥) 𝑑𝑥 = sin 𝑥 − csc 𝑥
14
Determinant Problem
Ex. Problem
𝑥+𝑦
𝑧𝑥
𝑦𝑧
2
𝑧𝑥
𝑦+𝑧
𝑥𝑦
𝐹0 (𝑥, 𝑦, 𝑧) 𝐹1 (𝑥, 𝑦, 𝑧)
Query 𝐹3 (𝑥, 𝑦, 𝑧) 𝐹4 (𝑥, 𝑦, 𝑧)
𝐹6 (𝑥, 𝑦, 𝑧) 𝐹7 (𝑥, 𝑦, 𝑧)
2
𝑧𝑦
𝑥𝑦
𝑧+𝑥
= 2𝑥𝑦𝑧 𝑥 + 𝑦 + 𝑧
3
2
𝐹2 (𝑥, 𝑦, 𝑧)
𝐹5 (𝑥, 𝑦, 𝑧)
𝐹8 (𝑥, 𝑦, 𝑧)
= 𝐶10 𝐹9 (𝑥, 𝑦, 𝑧)
𝐹𝑖 ≔ 𝐹𝑗 𝑥 → 𝑦; 𝑦 → 𝑧; 𝑧 → 𝑥 𝑤ℎ𝑒𝑟𝑒 𝑖, 𝑗 ∈ { 4,0 , 8,4 , 5,1 , … }
New problems generated:
𝑦2
𝑧+𝑦
𝑧2
2
𝑦𝑧 + 𝑦 2
𝑦𝑧
𝑧𝑥
𝑥2
𝑧2
𝑥+𝑧
𝑥𝑦
𝑧𝑥 + 𝑧 2
𝑧𝑥
2
𝑦+𝑥
𝑦2
𝑥2
𝑥𝑦
𝑦𝑧
𝑥𝑦 + 𝑥 2
2
= 2 𝑥𝑦 + 𝑦𝑧 + 𝑧𝑥
3
= 4𝑥 2 𝑦 2 𝑧 2
15
Problem Synthesis
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent cheating in MOOCs (Unsynchronized instruction)
• Problems of a given difficulty level and concept usage.
– Generate progressions
– Generate personalized workflows
Key Ideas
• Procedural Content: Test input generation techniques
• Conceptual Content
 Template based guess and verify
16
Problem Synthesis: Sentence Completion
1. The principal characterized his pupils as _________ because
they were pampered and spoiled by their indulgent parents.
2. The commentator characterized the electorate as _________
because it was unpredictable and given to constantly shifting
moods.
(a) cosseted
(b) disingenuous
(c) corrosive
(d) laconic
(e) mercurial
One of the above problems is a real SAT problem,
while the other one was automatically generated!
Problem Synthesis
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent cheating in MOOCs (Unsynchronized instruction)
• Problems of a given difficulty level and concept usage.
– Generate progressions
– Generate personalized workflows
Key Ideas
• Procedural Content: Test input generation techniques
• Conceptual Content
– Template based guess and verify
 Symbolic methods (solution generation in reverse)
18
Natural Deduction
Prove that: 𝑥1 ∨ 𝑥2 ∧ 𝑥3
and 𝑥1 → 𝑥4
and 𝑥4 → 𝑥5
implies 𝑥2 ∨ 𝑥5
Inference Rule
Premises
Conclusion
Modus Ponens (MP)
𝑝 → 𝑞, 𝑝
𝑞
Hypothetical Syllogism (HS)
𝑝 → 𝑞, 𝑞 → 𝑝
𝑝→𝑟
Disjunctive Syllogism (DS)
𝑝 ∨ 𝑞, ¬𝑝
𝑞
Simplification (Simp)
𝑝∧𝑞
𝑞
Replacement Rule
Proposition
Equiv. Proposition
Distribution
𝑝 ∨ (𝑞 ∧ 𝑟)
Double Negation
𝑝
¬¬𝑝
Implication
𝑝→𝑞
¬𝑝 ∨ 𝑞
Equivalence
𝑝≡𝑞
𝑝 ∨ 𝑞 ∧ (𝑝 ∨ 𝑟)
𝑝 → 𝑞 ∧ (𝑞 → 𝑝)
IJCAI 2013: Umair Ahmed, Sumit Gulwani, Amey Karkare
19
Similar Problem Synthesis: Natural Deduction
Similar Problems = those that have a minimal proof with
the same sequence of inference rules as used by a
minimal proof of given problem.
Premise 1
Premise 2
Premise 3
Conclusion
𝑥1 ∨ (𝑥2 ∧ 𝑥3 )
𝑥1 → 𝑥4
𝑥4 → 𝑥5
𝑥2 ∨ 𝑥5
Similar Problems
Premise 1
Premise 2
Premise 3
Conclusion
𝑥1 ≡ 𝑥2
𝑥3 → ¬𝑥2
(𝑥4 → 𝑥5 ) → 𝑥3
𝑥1 → (𝑥𝑦 ∧ ¬𝑥5 )
𝑥1 ∧ (𝑥2 → 𝑥3 )
𝑥1 ∨ 𝑥4 → ¬𝑥5 𝑥2 ∨ 𝑥5
(𝑥1 ∨ 𝑥2 ) → 𝑥3
𝑥3 → 𝑥1 ∧ 𝑥4
(𝑥1 → 𝑥2 ) → 𝑥3
𝑥3 → ¬𝑥4
𝑥1 → 𝑥2 ∧ 𝑥3
𝑥4 → ¬𝑥2
𝑥1 ∧ 𝑥4 → 𝑥5
𝑥1 ∨ 𝑥5 ∨ 𝑥4
𝑥3 ≡ 𝑥5 → 𝑥4
𝑥1 ∨ 𝑥4 ∧ ¬𝑥5
𝑥1 → 𝑥5
𝑥5 ∨ 𝑥2 → 𝑥1
𝑥1 → 𝑥3 ≡ ¬𝑥5
20
Parameterized Problem Synthesis: Natural Deduction
Parameters:
# of premises = 3, Size of propositions ≤ 4
# of variables = 3, # of inference steps = 2
Inference rules = { DS, HS }
Parameterized Problems
Premise 1
Premise 2
Premise 3 Conclusion
(𝑥1 → 𝑥3 ) → 𝑥2
𝑥2 → 𝑥3
¬𝑥3
𝑥1 ∧ ¬𝑥3
𝑥3 ≡ 𝑥1 → 𝑥2 ¬𝑥2
𝑥1 ∧ ¬𝑥3
𝑥3 → 𝑥1
𝑥1 ≡ 𝑥3 ∨ 𝑥1 ≡ 𝑥2
(𝑥1 ≡ 𝑥2 ) → 𝑥3
¬𝑥3
𝑥1 ≡ 𝑥3
𝑥1 ≡ ¬𝑥3
𝑥2 ∨ 𝑥1
𝑥3 → ¬𝑥2
𝑥1 ∧ ¬𝑥3
𝑥3 → 𝑥1
𝑥1 → 𝑥2 ∧ 𝑥3
𝑥3 → ¬𝑥2
¬𝑥3
21
Problem Synthesis
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent cheating in MOOCs (Unsynchronized instruction)
• Problems of a given difficulty level and concept usage.
– Generate progressions
– Generate personalized workflows
Key Ideas
• Procedural Content: Test input generation techniques
• Conceptual Content
– Template based guess and verify
 Symbolic methods (solution generation in reverse)
22
Problem Generation: Board Games
Game Rules
• allows variants of classic games.
Tic-Tac-Toe
4*4
Row/Column only
Expertise level of player/opponent
• allows leveling the playing field.
Automatic Generation of Starting Positions in Board Games;
Umair Ahmed, Krishnendu Chatterjee, Sumit Gulwani.
Solution Synthesis
Solution Synthesis
Motivation
• Synthesize sample solutions to new problems.
• Synthesize customized solutions or complete unfinished ones.
• Inform solution characteristics during problem generation.
Key Ideas
 Procedural Content: Programming by Demonstration
25
Solution Synthesis: GCF Computation
Time-stamped sequence
of (location, value) pairs
Programming by
Demonstration
Spreadsheet
Program
Can also be used to synthesize incorrect procedures from buggy traces!
Joint work with Erik Andersen and Zoran Popovic.
26
Solution Synthesis
Motivation
• Synthesize sample solutions to new problems.
• Synthesize customized solutions or complete unfinished ones.
• Inform solution characteristics during problem generation.
Key Ideas
• Procedural Content: Programming by Demonstration
 Conceptual Content
– Perform reasoning over models (vs. abstract symbolic reasoning).
– Reduce solution space to solutions with small length.
27
Ruler/Compass based Geometry Constructions
Given a triangle XYZ,
construct circle C such that
C passes through X, Y, and Z.
C
L1
L2
N
Z
X
Y
PLDI 2011: Sumit Gulwani, Vijay Korthikanti, Ashish Tiwari.
28
Problem Description
Given a triangle XYZ, construct circle C such that C
passes through X, Y, and Z.
Precondition:
Slope(X,Y)  Slope(X,Z) Æ Slope(X,Y)  Slope(Z,X)
Postcondition:
LiesOn(X,C) Æ LiesOn(Y,C) Æ LiesOn(Z,C)
Where LiesOn(X,C) ´
Distance(X,Center(C)) = Radius(C)
29
Solution is a program!
Geometry Program: A straight-line composition of
geometry methods.
Geometry Types: Point, Line, Circle
Geometry Methods:
• Ruler(Point, Point) -> Line
• Compass(Point, Point) -> Circle
• Intersect(Circle, Circle) -> Pair of Points
• Intersect(Line, Circle) -> Pair of Points
• Intersect(Line, Line) -> Point
30
Example of Geometry Program
Given a triangle XYZ, construct circle C such that C
passes through X, Y, and Z.
1. C1 = Compass(X,Y);
2. C2 = Compass(Y,X);
3. <P1,P2> = Intersect(C1,C2);
4. L1 = Ruler(P1,P2);
5. D1 = Compass(Z,X);
6. D2 = Compass(X,Z);
7. <R1,R2> = Intersect(D1,D2);
8. L2 = Ruler(R1,R2);
9. N = Intersect(L1,L2);
10. C = Compass(N,X);
C
L1
N
C1
D1
R1
Z
R2
P1
X
D2
L2
C2
Y
P2
31
Verification/Synthesis Problem for Geometry Programs
• Let P be a geometry program that computes
outputs O from inputs I.
• Verification Problem: Check the validity of the
following Hoare triple.
Assume Pre(I);
P
Assert Post(I,O);
• Synthesis Problem: Given Pre(I), Post(I,O), find P
such that the above Hoare triple is valid.
32
Verification Algorithm
Pre(I), P, Post(I,O)
a) Existing decision procedures are complex.
b)
1.
2.
3.
New efficient approach: Random Testing!
Choose I’ randomly from the set { I | Pre(I) }.
Compute O’ := P(I’).
If O’ satisfies Post(I’,O’) output “Verified”.
Correctness Proof of (b):
• Objects constructed by P can be described using
polynomial ops (+,-,*), square-root & division operator.
• The randomized polynomial identity testing algorithm
lifts to square-root and division operators as well !
33
Synthesis Algorithm
// First obtain a random input-output example.
1. Choose I’ randomly from the set { I | Pre(I) }.
2. Compute O’ s.t. Post(I’,O’) using numerical methods.
// Now obtain a construction that can generate O’ from I’
(using goal-directed search).
3. S := I’;
4. While (S does not contain O’)
5.
S := S [ { M(O1,O2) | Oi 2 S, M 2 Methods }
6. Output construction steps for O’.
34
Error Probability of the algorithm is extremely low.
Given a triangle XYZ, construct circle C such that C
passes through X, Y, and Z.
C
L1
…
L1 = Ruler(P1,P2);
N
…
L2 = Ruler(R1,R2);
N = Intersect(L1,L2);
Z
C = Compass(N,X);
X
L2
Y
• For an equilateral 4XYZ, incenter
coincides with circumcenter N.
• But what are the chances of choosing a
random 4XYZ to be an equilateral one?
35
Solution Synthesis
Motivation
• Synthesize sample solutions to new problems.
• Synthesize customized solutions or complete unfinished ones.
• Inform solution characteristics during problem generation.
Key Ideas
• Procedural Content: Programming by Demonstration
 Conceptual Content
– Perform reasoning over models (vs. abstract symbolic reasoning).
– Reduce solution space to solutions with small length.
36
Natural Deduction
Prove that: 𝑥1 ∨ 𝑥2 ∧ 𝑥3
and 𝑥1 → 𝑥4
and 𝑥4 → 𝑥5
implies 𝑥2 ∨ 𝑥5
Inference Rule
Premises
Conclusion
Modus Ponens (MP)
𝑝 → 𝑞, 𝑝
𝑞
Hypothetical Syllogism (HS)
𝑝 → 𝑞, 𝑞 → 𝑝
𝑝→𝑟
Disjunctive Syllogism (DS)
𝑝 ∨ 𝑞, ¬𝑝
𝑞
Simplification (Simp)
𝑝∧𝑞
𝑞
Replacement Rule
Proposition
Equiv. Proposition
Distribution
𝑝 ∨ (𝑞 ∧ 𝑟)
Double Negation
𝑝
¬¬𝑝
Implication
𝑝→𝑞
¬𝑝 ∨ 𝑞
Equivalence
𝑝≡𝑞
𝑝 ∨ 𝑞 ∧ (𝑝 ∨ 𝑟)
𝑝 → 𝑞 ∧ (𝑞 → 𝑝)
IJCAI 2013: Umair Ahmed, Sumit Gulwani, Amey Karkare
37
Solution Synthesis: Natural Deduction
Prove that: 𝑥1 ∨ 𝑥2 ∧ 𝑥3
and 𝑥1 → 𝑥4
and 𝑥4 → 𝑥5
Step Proposition
Reason
P1
𝑥1 ∨ 𝑥2 ∧ 𝑥3
Premise
implies: 𝑥2 ∨ 𝑥5
P2
𝑥1 → 𝑥4
Premise
P3
𝑥4 → 𝑥5
Premise
1
(𝑥1 ∨ 𝑥2 ) ∧ (𝑥1 ∨ 𝑥3 ) P1, Distr.
2
𝑥1 ∨ 𝑥2
1, Simp.
3
𝑥1 → 𝑥5
P2, P3, HS
4
𝑥2 ∨ 𝑥1
2, Comm.
Step Truth
Table
Reason
P1
1048575
Premise
P2
4294914867
Premise
5
¬¬𝑥2 ∨ 𝑥1
4. Double Neg.
P3
3722304989
Premise
6
¬𝑥2 → 𝑥1
5, Implication
1
16777215
P1,Simp.
7
6,3, HS.
2
4294923605
P2,P3,HS
¬𝑥2 → 𝑥5
8
¬¬𝑥2 ∨ 𝑥5
7, Implication
Concl 1442797055 1,2,HS
Abstract Proof
Concl 𝑥2 ∨ 𝑥5
Concrete Proof
8, Double Neg.
38
Feedback Synthesis
Feedback Synthesis
Motivation
• Makes teachers more effective.
– Saves them time.
– Provides immediate insights on where students are struggling.
• Can enable rich interactive experience for students.
– Generation of hints.
– Pointer to simpler problems depending on kind of mistake.
Key Ideas: Various feedback metrics
 Inputs on which the solution is not correct (counterexamples).
40
Counterexample based Feedback: PexForFun
41
Buggy Program for Array Reverse
using System;
6:28::50 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
42
Buggy Program for Array Reverse
using System;
6:32::01 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
43
Buggy Program for Array Reverse
using System;
6:32::32 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Sign of Frustation?
44
Buggy Program for Array Reverse
using System;
6:33::19 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i <= a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
45
Buggy Program for Array Reverse
using System;
6:33::55 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
Same as initial attempt except Console.Writeline!
46
Buggy Program for Array Reverse
using System;
6:34::06 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
No change! Sign of Frustation?
47
Buggy Program for Array Reverse
using System;
6:34::56 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i <= a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
The student has tried this before!
48
Buggy Program for Array Reverse
using System;
6:36::24 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Same as initial attempt!
49
Buggy Program for Array Reverse
using System;
6:37::39 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
The student has tried this before!
50
Buggy Program for Array Reverse
using System;
6:38::11 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i > 0; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Almost correct! (a[i-1] instead of a[i] in loop body)
51
Buggy Program for Array Reverse
using System;
6:38::44 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i >= 0; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Student going in wrong direction!
52
Buggy Program for Array Reverse
using System;
6:39::33 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Back to bigger error!
53
Buggy Program for Array Reverse
using System;
6:39::45 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Frustation!
54
Buggy Program for Array Reverse
using System;
6:40::27 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! More Frustation!!
55
Buggy Program for Array Reverse
using System;
6:40::57 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Too Frustated now!!! Gives up.
56
Feedback Synthesis
Motivation
• Makes teachers more effective.
– Saves them time.
– Provides immediate insights on where students are struggling.
• Can enable rich interactive experience for students.
– Generation of hints.
– Pointer to simpler problems depending on kind of mistake.
Key Ideas: Various feedback metrics
• Inputs on which the solution is not correct (counterexamples).
 Edit distance to a nearest correct solution.
57
Correction Model for Programming Assignments
Array Index:
v[a] -> v[{a+1, a-1}]
Increment:
v++ -> { ++v, v--, --v }
Conditional:
a op b -> a ops b
where ops = { <, >, <=, >=, ==, != }
Initialization:
v=n -> v={n+1, n-1, 0}
Return Value:
return v -> return ?v
Key Idea: Finding minimal set of such changes that convert a
buggy solution into a correct one is phrased as a sketch problem.
PLDI 2013: Rishabh Singh, Sumit Gulwani, Armando Solar-Lezama
58
Feedback Synthesis: Array Reverse
i = 1
front <= back
i <= a.Length
--back
http://sketch1.csail.mit.edu/python-autofeedback/
Feedback Synthesis
Motivation
• Makes teachers more effective.
– Saves them time.
– Provides immediate insights on where students are struggling.
• Can enable rich interactive experience for students.
– Generation of hints.
– Pointer to simpler problems depending on kind of mistake.
Key Ideas: Various feedback metrics
• Inputs on which the solution is not correct (counterexamples).
• Edit distance to a nearest correct solution.
 Edit distance to the problem description (of a nearest
corresponding problem description).
60
Feedback Synthesis: Finite State Automata
Draw a DFA that accepts: { s | ‘ab’ appears in s exactly 2 times }
Attempt 1
Grade: 9/10
Feedback: One more state
should be made final
Based on Edit-distance to a
correct solution
Grade: 6/10
Feedback: The DFA is
incorrect on the string ‘ababb’
Attempt 2
Based on Counterexample
Grade: 5/10
Feedback: The DFA accepts
{s | ‘ab’ appears in s at least 2 times}
Attempt 3
Based on Edit-distance to
the problem description
IJCAI 2013: Alur, Loris d’Antoni, Gulwani, Dileep Kini, Viswanathan 61
Content Authoring
Content Authoring
Motivation: Make it easy to create and edit
 Equations
• Drawings
Key Ideas
• Allow multi-modal NUIs such as ink, speech, touch.
• Error correction.
• Prediction.
63
Challenges in creating Equations
Text editors like Latex
• Unreadable text in prefix notation
WYSIWIG editors like Microsoft Word
• Change of cursor positions multiple times.
• Back and forth switching between mouse & keyboard
Our solution: An intelligent predictive editor.
• Mathematical text has low entropy and hence
amenable to prediction!
Joint work with Alex Polozov and Sriram Rajamani.
64
Structure Prediction
We can predict likely structures from a flattened representation.
This can enable math-by-speech and auto-parenthesis-error-fix.
1+cos 𝐴
1−cos 𝐴
1 + cos A / 1 − cos A
((1 + cos A ) / (1 − cos A ))
Term Prediction (Syntactic Intellisense)
tan 3𝑥 tan 2𝑥 tan 𝑥 = tan 3𝑥 − tan 2𝑥 − tan 𝑥
𝑦𝑧 − 𝑥 2
𝑧𝑥 − 𝑦 2
𝑥𝑦 − 𝑧 2
𝐴1 sin3 𝛼
𝐴2 sin 𝛼
𝑧𝑥 − 𝑦 2
𝑥𝑦 − 𝑧 2
𝑦𝑧 − 𝑥 2
𝐵1 sin3 𝛽
𝐵2 sin 𝛽
𝑥𝑦 − 𝑧 2
𝑦𝑧 − 𝑥 2
𝑧𝑥 − 𝑦 2
𝐶1 sin3 𝛾
𝐶2 sin 𝛾
Our tool can infer colored text.
66
Term Prediction (Semantic Intellisense)
Prove (csc 𝑥 − sin 𝑥)(sec 𝑥 − cos 𝑥)(tan 𝑥 + cot 𝑥) = 1
L.H.S. =
1
− sin 𝑥
sin 𝑥
=
1 − sin2 𝑥
sin 𝑥
=
cos 2 𝑥
sin 𝑥
1
− cos 𝑥
cos 𝑥
1 − cos2 𝑥
cos 𝑥
sin2 𝑥
cos 𝑥
sin 𝑥 cos 𝑥
+
cos 𝑥 sin 𝑥
sin2 𝑥 + cos 2 𝑥
cos 𝑥 sin 𝑥
1
cos 𝑥 sin 𝑥
=1
67
Content Authoring
Motivation: Make it easy to create and edit
• Equations
 Drawings
Key Ideas
• Allow multi-modal NUIs such as ink, speech, touch.
• Error correction.
• Prediction.
68
Challenges in creating drawings
Sometimes require extreme precision.
• Our solution: allow pen-based sketching
and use constraint solving to beautify.
Can be tedious to draw sometimes.
Copy-Paste doesn’t address positioning of copied
objects & transformations on copied objects!
• Our Solution: use synthesis to predict
repetitive features.
CHI 2012: Salman Cheema, Sumit Gulwani, Joseph LaViola.
69
(Ink) Sketching meets Synthesis
Beautification
Beautification
Prediction
Prediction
Conclusion
• Technical Perspective
– Aspects: Problem/Solution/Feedback Synthesis, Content Authoring
– Domains: Math, Programming, Logic, Language Learning, ...
• Inter-disciplinary research area
–
–
–
–
Synthesis/Search techniques
HCI
Natural language processing (for word problems)
Machine learning (for analytics)
• Value Proposition:
– Short term: Improve Education (personalized + interactive)
– Long Term: Ultra-intelligent computer, Model of human mind,
Inter-stellar travel 
71