Synthesis for
Intelligent Tutoring Systems
Sumit Gulwani
Microsoft Research, Redmond
(Joint work with several collaborators)
ExCAPE Webinar Series
January 2013
Background: Program Synthesis
• Application domain
– Software Developers
– End-Users
– Education
• Specification of intent
– Logic
– Examples
– Natural Language
• Search techniques
– SAT/SMT solvers (Formal Methods)
– Brute-force goal-directed search (AI)
– Version space algebras (Machine Learning)
SYNASC ‘12: Synthesis from Examples: Interaction Models & Algorithms
1
Connections between End-User Programming (EUP)
& Intelligent Tutoring Systems (ITS)
• End-User teaches Computer & Computer applies that
knowledge to automate repetitive tasks.
Teacher teachers Computer & Computer applies that
knowledge to teach Students.
• Interactivity between End-User & Computer to resolve
ambiguities
Interactivity between Student & Computer to resolve
student’s confusions
Implications:
Research in EUP can fuel research in ITS
and vice-versa!
2
Intelligent Tutoring Systems
• Various Aspects
–
–
–
–
Problem Generation
Solution Generation
Grading/Feedback
Content Entry
• Various Domains
–
–
–
–
Geometry, Algebra, Physics, Chemistry
Programming, Automata, Logic
Board Games
Language Learning
3
Problem Generation
Problem Generation
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent plagiarism in MOOCs
• Massive number of students; Unsynchronized instruction
• Problems of a given difficulty level and that require testing
of certain concepts.
– Compare progressions (from different textbooks)
– Generate personalized workflows
Key Ideas
 Guess and verify
• Often requires extremely fast verification capability.
5
Trigonometry Problem
Example Problem: sec 𝑥 + cos 𝑥
Query: 𝑇1 𝑥 ± 𝑇2 (𝑥)
𝑇1 ≠ 𝑇5
sec 𝑥 − cos 𝑥 = tan2 𝑥 + sin2 𝑥
𝑇3 𝑥 ± 𝑇4 𝑥
= 𝑇52 𝑥 ± 𝑇62 (𝑥)
New problems generated:
csc 𝑥 + cos 𝑥 csc 𝑥 − cos 𝑥 = cot 2 𝑥 + sin2 𝑥
(csc 𝑥 − sin 𝑥)(csc 𝑥 + sin 𝑥) = cot 2 𝑥 + cos 2 𝑥
(sec 𝑥 + sin 𝑥)(sec 𝑥 − sin 𝑥) = tan2 𝑥 + cos 2 𝑥
:
(tan 𝑥 + sin 𝑥)(tan 𝑥 − sin 𝑥) = tan2 𝑥 − sin2 𝑥
(csc 𝑥 + cos 𝑥)(csc 𝑥 − cos 𝑥) = csc 2 𝑥 − cos 2 𝑥
:
AAAI 2012: Rohit Singh, Sumit Gulwani, Sriram Rajamani.
6
Trigonometry Problem
Example Problem: sec 𝑥 + cos 𝑥
Query: 𝑇1 𝑥 ± 𝑇2 (𝑥)
𝑇1 ≠ 𝑇5
sec 𝑥 − cos 𝑥 = tan2 𝑥 + sin2 𝑥
𝑇3 𝑥 ± 𝑇4 𝑥
= 𝑇52 𝑥 ± 𝑇62 (𝑥)
New problems generated:
csc 𝑥 + cos 𝑥 csc 𝑥 − cos 𝑥 = cot 2 𝑥 + sin2 𝑥
(csc 𝑥 − sin 𝑥)(csc 𝑥 + sin 𝑥) = cot 2 𝑥 + cos 2 𝑥
(sec 𝑥 + sin 𝑥)(sec 𝑥 − sin 𝑥) = tan2 𝑥 + cos 2 𝑥
:
(tan 𝑥 + sin 𝑥)(tan 𝑥 − sin 𝑥) = tan2 𝑥 − sin2 𝑥
(csc 𝑥 + cos 𝑥)(csc 𝑥 − cos 𝑥) = csc 2 𝑥 − cos 2 𝑥
:
AAAI 2012: Rohit Singh, Sumit Gulwani, Sriram Rajamani.
7
Limits/Series Problem
𝑛
Example Problem:
𝑛
Query:
lim
𝑛→∞
𝑖=0
lim
𝑛→∞
𝑖=0
2𝑖 2 + 𝑖 + 1
5
=
𝑖
2
5
𝐶0 𝑖 2 + 𝐶1 𝑖 + 𝐶2
𝐶3 𝑖
𝐶4
=
𝐶5
C0 ≠ 0 ∧ gcd 𝐶0 , 𝐶1 , 𝐶2 = gcd 𝐶4 , 𝐶5 = 1
New problems generated:
𝑛
lim
𝑛→∞
𝑖=0
𝑛
lim
𝑛→∞
𝑖=0
𝑛
2
3𝑖 + 2𝑖 + 1
7
=
𝑖
3
7
lim
𝑛→∞
𝑛
2
𝑖
3
=
𝑖
2
3
𝑖=0
lim
𝑛→∞
𝑖=0
3𝑖 2 + 3𝑖 + 1
=4
𝑖
4
5𝑖 2 + 3𝑖 + 3
=6
𝑖
6
8
Integration Problem
Example Problem:
Query:
(csc 𝑥) (csc 𝑥 − cot 𝑥) 𝑑𝑥 = csc 𝑥 − cot 𝑥
𝑇0 𝑥 𝑇1 𝑥 ± 𝑇2 𝑥 𝑑𝑥 = 𝑇4 𝑥 ± 𝑇5 (𝑥)
𝑇1 ≠ 𝑇2 ∧ 𝑇4 ≠ 𝑇5
New problems generated:
(tan 𝑥) (cos 𝑥 + sec 𝑥) 𝑑𝑥 = sec 𝑥 − cos 𝑥
(sec 𝑥) (tan 𝑥 + sec 𝑥) 𝑑𝑥 = sec 𝑥 + cot 𝑥
(cot 𝑥) (sin 𝑥 + csc 𝑥) 𝑑𝑥 = sin 𝑥 − csc 𝑥
9
Determinant Problem
Ex. Problem
𝑥+𝑦
𝑧𝑥
𝑦𝑧
2
𝑧𝑥
𝑦+𝑧
𝑥𝑦
𝐹0 (𝑥, 𝑦, 𝑧) 𝐹1 (𝑥, 𝑦, 𝑧)
Query 𝐹3 (𝑥, 𝑦, 𝑧) 𝐹4 (𝑥, 𝑦, 𝑧)
𝐹6 (𝑥, 𝑦, 𝑧) 𝐹7 (𝑥, 𝑦, 𝑧)
2
𝑧𝑦
𝑥𝑦
𝑧+𝑥
= 2𝑥𝑦𝑧 𝑥 + 𝑦 + 𝑧
3
2
𝐹2 (𝑥, 𝑦, 𝑧)
𝐹5 (𝑥, 𝑦, 𝑧)
𝐹8 (𝑥, 𝑦, 𝑧)
= 𝐶10 𝐹9 (𝑥, 𝑦, 𝑧)
𝐹𝑖 ≔ 𝐹𝑗 𝑥 → 𝑦; 𝑦 → 𝑧; 𝑧 → 𝑥 𝑤ℎ𝑒𝑟𝑒 𝑖, 𝑗 ∈ { 4,0 , 8,4 , 5,1 , … }
New problems generated:
𝑦2
𝑧+𝑦
𝑧2
2
𝑦𝑧 + 𝑦 2
𝑦𝑧
𝑧𝑥
𝑥2
𝑧2
𝑥+𝑧
𝑥𝑦
𝑧𝑥 + 𝑧 2
𝑧𝑥
2
𝑦+𝑥
𝑦2
𝑥2
𝑥𝑦
𝑦𝑧
𝑥𝑦 + 𝑥 2
2
= 2 𝑥𝑦 + 𝑦𝑧 + 𝑧𝑥
3
= 4𝑥 2 𝑦 2 𝑧 2
10
Problem Generation
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent plagiarism in MOOCs
• Massive number of students; Unsynchronized instruction
• Problems of a given difficulty level and that require testing
of certain concepts.
– Compare progressions (from different textbooks)
– Generate personalized workflows
Key Ideas
 Guess and verify
• Often requires extremely fast verification capability.
11
Logic (Natural Deduction) Problem
Joint work with Umair Ahmed and Amey Karkare (IIT Kanpur).
Problem Generation
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent plagiarism in MOOCs
• Massive number of students; Unsynchronized instruction
• Problems of a given difficulty level and that require testing
of certain concepts.
– Compare progressions (from different textbooks)
– Generate personalized workflows
Key Ideas
• Guess and verify
 Symbolic methods (solution generation in reverse)
13
Board Game Problem
Joint work with Umair Ahmed and Krishnendu Chatterjee
Problem Generation
Motivation
• Problems similar to a given problem.
– Avoid copyright issues
– Prevent plagiarism in MOOCs
• Massive number of students; Unsynchronized instruction
• Problems of a given difficulty level and that require testing
of certain concepts.
– Compare progressions (from different textbooks)
– Generate personalized workflows
Key Ideas
• Guess and verify
• Symbolic methods (solution generation in reverse)
 Test input generation techniques (for procedural problems)
15
Classifying Problems: Addition
CHI 2012: Erik Andersen, Sumit Gulwani, Zoran Popovic.
16
Comparing Progressions: Integer Comparison
Green Progression: Jump Math
Blue Progression: Skill Sharpeners
Green Progression moves (quickly) into involved
problems that involve comparing more digits.
But it ignore an entire class of levels (H and L)!
17
Solution Generation
Solution Generation
Motivation
• Check solution characteristics during problem generation.
• Generate sample solutions to new problems.
• Generate customized solutions.
• Complete unfinished solutions.
Key Ideas
 Program synthesis from logic (and examples)
19
Ruler/Compass based Geometry Constructions
Given a triangle XYZ,
construct circle C such that
C passes through X, Y, and Z.
C
L1
L2
N
Z
X
PLDI 2011: Gulwani, Korthikanti, Tiwari.
Y
20
Formal specification of the problem
Given a triangle XYZ, construct circle C such that C
passes through X, Y, and Z.
Precondition:
Slope(X,Y)  Slope(X,Z) Æ Slope(X,Y)  Slope(Z,X)
Postcondition:
LiesOn(X,C) Æ LiesOn(Y,C) Æ LiesOn(Z,C)
Where LiesOn(X,C) ´
Distance(X,Center(C)) = Radius(C)
21
Solution is a program!
Geometry Program: A straight-line composition of
geometry methods.
Geometry Types: Point, Line, Circle
Geometry Methods:
• Ruler(Point, Point) -> Line
• Compass(Point, Point) -> Circle
• Intersect(Circle, Circle) -> Pair of Points
• Intersect(Line, Circle) -> Pair of Points
• Intersect(Line, Line) -> Point
22
Example of Geometry Program
Given a triangle XYZ, construct circle C such that C
passes through X, Y, and Z.
1. C1 = Compass(X,Y);
2. C2 = Compass(Y,X);
3. <P1,P2> = Intersect(C1,C2);
4. L1 = Ruler(P1,P2);
5. D1 = Compass(Z,X);
6. D2 = Compass(X,Z);
7. <R1,R2> = Intersect(D1,D2);
8. L2 = Ruler(R1,R2);
9. N = Intersect(L1,L2);
10. C = Compass(N,X);
C
L1
N
C1
D1
R1
Z
R2
P1
X
D2
L2
C2
Y
P2
23
Verification/Synthesis Problem for Geometry Programs
• Let P be a geometry program that computes
outputs O from inputs I.
• Verification Problem: Check the validity of the
following Hoare triple.
Assume Pre(I);
P
Assert Post(I,O);
• Synthesis Problem: Given Pre(I), Post(I,O), find P
such that the above Hoare triple is valid.
24
Verification Algorithm
Pre(I), P, Post(I,O)
a) Existing decision procedures are complex.
b)
1.
2.
3.
New efficient approach: Random Testing!
Choose I’ randomly from the set { I | Pre(I) }.
Compute O’ := P(I’).
If O’ satisfies Post(I’,O’) output “Verified”.
Correctness Proof of (b):
• Objects constructed by P can be described using
polynomial ops (+,-,*), square-root & division operator.
• The randomized polynomial identity testing algorithm
lifts to square-root and division operators as well !
25
Synthesis Algorithm
// First obtain a random input-output example.
1. Choose I’ randomly from the set { I | Pre(I) }.
2. Compute O’ s.t. Post(I’,O’) using numerical methods.
// Now obtain a construction that can generate O’ from I’
(using goal-directed search).
3. S := I’;
4. While (S does not contain O’)
5.
S := S [ { M(O1,O2) | Oi 2 S, M 2 Methods }
6. Output construction steps for O’.
26
Error Probability of the algorithm is extremely low.
Given a triangle XYZ, construct circle C such that C
passes through X, Y, and Z.
C
L1
…
L1 = Ruler(P1,P2);
N
…
L2 = Ruler(R1,R2);
N = Intersect(L1,L2);
Z
C = Compass(N,X);
X
L2
Y
• For an equilateral 4XYZ, incenter
coincides with circumcenter N.
• But what are the chances of choosing a
random 4XYZ to be an equilateral one?
27
Solution Generation
Motivation
• Check solution characteristics during problem generation.
• Generate sample solutions to new problems.
• Generate customized solutions.
• Complete unfinished solutions.
Key Ideas
 Program synthesis from logic (and examples)
28
Classic Problem in Automata Theory Course
Let L be the language containing all strings over {a,b}
that have the same number of occurrences of “ab” as
occurrences of “ba”. Construct an automata that accepts
L, or prove that L is non-regular.
Solution Generation Engine
“Regular”, <Automata for L>
Joint work with Cerny, Henzinger, Zufferey, Radhakrishna
29
Classic Problem in Automata Theory Course
Let L be the language containing all strings over {a,b}
that have the same number of occurrences of “a” as
occurrences of “b”. Construct an automata that accepts
L, or prove that L is non-regular.
Solution Generation Engine
“Non-regular”, <Proof of non-regularity>
30
Methodology
 Problem Description Language
31
User-friendly Logic for describing problems
• same number of occurrences of “ab” as that of “ba”.
– 𝑐𝑜𝑢𝑛𝑡(𝑤, “𝑎𝑏”) = 𝑐𝑜𝑢𝑛𝑡(𝑤, “𝑏𝑎”)
• all occurrences of “ab” start on an even position.
– ∀𝑝: ((1 < 𝑝 < 𝐿𝑒𝑛𝑔𝑡ℎ 𝑤 ∧ 𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠 𝑤, “𝑎𝑏”, 𝑝) ⇒
𝑝 𝑚𝑜𝑑 2 = 0 )
• consists of n a’s followed by n b’s.
– 𝑤 = 𝑐𝑜𝑛𝑐𝑎𝑡𝑒𝑛𝑎𝑡𝑒(𝑟𝑒𝑝𝑒𝑎𝑡(“𝑎”, 𝑛), 𝑟𝑒𝑝𝑒𝑎𝑡(“𝑏”, 𝑛))
32
Methodology
• Problem Description Language
• Solution Description Language
– Automata
 Non-regularity proofs
33
Myhill-Nerode Theorem: Non-regularity Condition
A language 𝐿 is non-regular iff there exist functions
𝐹: 𝑁 → Σ ∗ and 𝐺: 𝑁 → Σ ∗ such that:
𝐹 𝑖 𝐺 𝑖, 𝑗 ∈ 𝐿 ∧ 𝐹 𝑗 𝐺 𝑖, 𝑗 ∉ 𝐿
∀𝑖, 𝑗: 1 ≤ 𝑗 < 𝑖 ⇒
𝑜𝑟
𝐹 𝑖 𝐺 𝑖, 𝑗 ∉ 𝐿 ∧ 𝐹 𝑗 𝐺 𝑖, 𝑗 ∈ 𝐿
We refer to F/G as congruence/witness functions.
Examples:
• Same number of occurrences of “a” as that of “b”.
Congruence Function: 𝑎𝑖
Witness Function: 𝑏 𝑖
•
𝑤𝑤 𝑅 𝑤 ∈ 0,1 ∗ }.
Congruence Function: 𝑎𝑖 𝑏 Witness Function: 𝑏𝑎𝑖
34
Language for Congruence/Witness Functions
The following language is expressive enough to
represent Congruence function F(i) and Witness
Function G(i,j) for several classroom problems.
𝑆 ≔ 𝑡1 𝑡2 … 𝑡𝑛
𝑡 ≔ 𝑐 𝑐𝑖 𝑐𝑗
where c is any alphabet.
35
Methodology
• Problem Description Language
• Solution Description Language
– Automata
– Non-regularity proofs
• Synthesis Algorithms
– Problem description to Automata
• Based on Angluin’s L* automata synthesis algorithm
– Problem description to Non-regularity proofs
• Based on guess-test-verify method.
• Guessing is done using inductive synthesis.
36
Solution Generation
Motivation
• Check solution characteristics during problem generation.
• Generate sample solutions to new problems.
• Generate customized solutions.
• Complete unfinished solutions.
Key Ideas
• Program synthesis from logic (and examples)
 Program synthesis from demonstrations (for procedural
problems)
37
Solution Generation from Demonstrations
Joint work with Erik Andersen and Zoran Popovic
38
Automated Grading
Automated Grading
Motivation
• Answer-scripts graded immediately.
• Can enable rich tutoring experience such as hint generation
or simpler practice problems.
Key Ideas: Various feedback metrics
 Inputs on which the solution is not correct.
40
Background: PexForFun
41
Buggy Program for Array Reverse
using System;
6:28::50 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
42
Buggy Program for Array Reverse
using System;
6:32::01 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
43
Buggy Program for Array Reverse
using System;
6:32::32 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Sign of Frustation?
44
Buggy Program for Array Reverse
using System;
6:33::19 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i <= a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
45
Buggy Program for Array Reverse
using System;
6:33::55 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
Same as initial attempt except Console.Writeline!
46
Buggy Program for Array Reverse
using System;
6:34::06 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
No change! Sign of Frustation?
47
Buggy Program for Array Reverse
using System;
6:34::56 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i <= a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
The student has tried this before!
48
Buggy Program for Array Reverse
using System;
6:36::24 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Same as initial attempt!
49
Buggy Program for Array Reverse
using System;
6:37::39 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
The student has tried this before!
50
Buggy Program for Array Reverse
using System;
6:38::11 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i > 0; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Almost correct! (a[i-1] instead of a[i] in loop body)
51
Buggy Program for Array Reverse
using System;
6:38::44 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i >= 0; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Student going in wrong direction!
52
Buggy Program for Array Reverse
using System;
6:39::33 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Back to bigger error!
53
Buggy Program for Array Reverse
using System;
6:39::45 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Frustation!
54
Buggy Program for Array Reverse
using System;
6:40::27 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! More Frustation!!
55
Buggy Program for Array Reverse
using System;
6:40::57 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Too Frustated now!!! Gives up.
56
Automated Grading
Motivation
• Answer-scripts graded immediately.
• Can enable rich tutoring experience such as hint generation
or simpler practice problems.
Key Ideas: Various feedback metrics
• Inputs on which the solution is not correct.
• Bug localization by synthesizing nearest correct solution.
 Program Sketching
57
Error Model for Grading Programming Assignments
Array Index:
v[a] -> v[{a+1, a-1}]
Increment:
v++ -> { ++v, v--, --v }
Conditional:
a op b -> a ops b
where ops = { <, >, <=, >=, ==, != }
Initialization:
v=n -> v={n+1, n-1, 0}
Return Value:
return v -> return ?v
Key Idea: Find minimal set of such changes that convert the
solution into a correct one. Can be phrased as a sketch problem.
Joint work with Rishabh Singh and Armando Solar-Lezama
58
Automated Grading (Array Reverse)
i = 1
front <= back
i <= a.Length
--back
http://sketch1.csail.mit.edu/python-autofeedback/
http://sketch1.csail.mit.edu/autograder-v1/tool/index1.php
Automated Grading
Motivation
• Answer-scripts graded immediately.
• Can enable rich tutoring experience such as hint generation
or simpler practice problems.
Key Ideas: Various feedback metrics
• Inputs on which the solution is not correct.
• Bug localization by synthesizing nearest correct solution.
– Program Sketching
– Mutation
• Problem for which the given solution is correct.
60
Automata Grading: Various Feedback Metrics
Draw an automata that accepts strings with even number of a’s.
Input based
feedback
Mutation based
bug localization
Problem based
feedback
Joint work with Rajeev Alur, Loris D’Antoni, Dileep Kini, Mahesh Viswanathan
61
Content Entry
Content Entry
Motivation: Make it easy to create and edit
 Equations
• Drawings
Key Ideas:
• Allow multi-modal input such as ink, speech, touch.
• Error correction to infer likely concept.
• Prediction as in Intellisense.
63
Challenges in creating Equations
Text editors like Latex
• Unreadable text in prefix notation
WYSIWIG editors like Microsoft Word
• Change of cursor positions multiple times.
• Back and forth switching between mouse & keyboard
Our solution: An intelligent predictive editor.
• Mathematical text has low entropy and hence
amenable to prediction!
Joint work with Alex Polozov and Sriram Rajamani.
64
Structure Prediction
We can predict likely structures from a flattened representation.
This can enable math-by-speech and auto-parenthesis-error-fix.
1+cos 𝐴
1−cos 𝐴
1 + cos A / 1 − cos A
((1 + cos A ) / (1 − cos A ))
Term Prediction (Syntactic Intellisense)
tan 3𝑥 tan 2𝑥 tan 𝑥 = tan 3𝑥 − tan 2𝑥 − tan 𝑥
𝑦𝑧 − 𝑥 2
𝑧𝑥 − 𝑦 2
𝑥𝑦 − 𝑧 2
𝐴1 sin3 𝛼
𝐴2 sin 𝛼
𝑧𝑥 − 𝑦 2
𝑥𝑦 − 𝑧 2
𝑦𝑧 − 𝑥 2
𝐵1 sin3 𝛽
𝐵2 sin 𝛽
𝑥𝑦 − 𝑧 2
𝑦𝑧 − 𝑥 2
𝑧𝑥 − 𝑦 2
𝐶1 sin3 𝛾
𝐶2 sin 𝛾
Our tool can infer colored text.
66
Term Prediction (Semantic Intellisense)
Prove (csc 𝑥 − sin 𝑥)(sec 𝑥 − cos 𝑥)(tan 𝑥 + cot 𝑥) = 1
L.H.S. =
1
− sin 𝑥
sin 𝑥
=
1 − sin2 𝑥
sin 𝑥
=
cos 2 𝑥
sin 𝑥
1
− cos 𝑥
cos 𝑥
1 − cos2 𝑥
cos 𝑥
sin2 𝑥
cos 𝑥
sin 𝑥 cos 𝑥
+
cos 𝑥 sin 𝑥
sin2 𝑥 + cos 2 𝑥
cos 𝑥 sin 𝑥
1
cos 𝑥 sin 𝑥
=1
67
Content Entry
Motivation: Make it easy to create and edit
• Equations
 Drawings
Key Ideas:
• Allow multi-modal input such as ink, speech, touch.
• Error correction to infer likely concept.
• Prediction as in Intellisense.
68
Challenges in creating drawings
Sometimes require extreme precision.
• Our solution: allow pen-based sketching
and use constraint solving to beautify.
Can be tedious to draw sometimes.
Copy-Paste doesn’t address positioning of copied
objects & transformations on copied objects!
• Our Solution: use synthesis to predict
repetitive features.
CHI 2012: Salman Cheema, Sumit Gulwani, Joseph LaViola.
69
(Ink) Sketching meets Synthesis
Beautification
Beautification
Prediction
Prediction
Conclusion
• Technical Perspective
– Aspects: Solution+Problem Generation, Grading, Content Entry
– Domains: Math/Science, Programming, Language Learning
• Inter-disciplinary research area
–
–
–
–
Formal methods, Synthesis/Search techniques
HCI
Natural language processing (for word problems)
Machine learning (for analytics)
• Value Proposition:
– Short term: Improve Education (interactivity; exploration;
social)
– Long Term: Ultra-intelligent computer, Model of human mind,
71
Inter-stellar travel 