Dimensions in Synthesis
Part 2: Applications
(Intelligent Tutoring Systems)
Sumit Gulwani
sumitg@microsoft.com
Microsoft Research, Redmond
May 2012
Recap
Domain
Insight
Bit-vector
Algorithms
Geometry
Constructions
Testing &
Symbolic verification
Testing is
Verification runs in reasonable time probabilistically
sound
Synthesis
Strategy
Counter-example
guided inductive
synthesis
Brute-force search:
Generate and Test
1
Potential Users of Synthesis Technology
Algorithm
Designers
Software Developers
Most Useful
Target
Most
Transformational
Target
End-Users
Students and Teachers
• Vision for End-users: Enable people to have (automated)
personal assistants.
• Vision for Education: Enable every student to have
access to free & high-quality education.
2
Intelligent Tutoring Systems
• Motivation
– Online learning sites: Khan academy, Edx, Udacity, Coursera
• Increasing class sizes with even less personal attention
– New technologies: Tablets/Smartphones, NUI, Cloud
• Various Aspects
–
–
–
–
Solution Generation
Problem Generation
Automated Grading/Feedback
Content Entry
• Various Domains
– K-12: Mathematics, Physics, Chemistry
– Undergraduate: Introductory Programming, Automata Theory
– Language Learning
3
Intelligent Tutoring Systems
• Aspects
–
–
–
–
Solution Generation
Problem Generation
Automated Grading
Content Entry
• Domains
–
–
–
–
–
–
Geometry
Algebra
Introductory Programming
Automata Theory
Physics
Chemistry
⋮
Joint work with: Cerny, Henzinger, Radhakrishna, Zufferey
4
Classic Problem in Automata Theory Course
Let L be the language containing all strings over {a,b}
that have the same number of occurrences of “ab” as
occurrences of “ba”. Construct an automata that accepts
L, or prove that L is non-regular.
Solution Generation Engine
“Regular”, <Automata for L>
5
Classic Problem in Automata Theory Course
Let L be the language containing all strings over {a,b}
that have the same number of occurrences of “a” as
occurrences of “b”. Construct an automata that accepts
L, or prove that L is non-regular.
Solution Generation Engine
“Non-regular”, <Proof of non-regularity>
6
Outline
 Formal Description of Input
• Regular Languages
– Algorithm for automata synthesis.
• Non-regular Languages
– Formal description of non-regularity proof.
– Algorithm for proof synthesis.
7
Formal Description of Input
Problem Description Languages
Examples
• User-friendly logic
• Context-free grammar
Interfaces
• Membership Test
– Required for inductive synthesis of automata or
non-regularity proof.
• Symbolic Membership Test
– Required for verification of non-regularity proof.
8
User-friendly Logic
• same number of occurrences of “ab” as that of “ba”.
– 𝑐𝑜𝑢𝑛𝑡(𝑤, “𝑎𝑏”) = 𝑐𝑜𝑢𝑛𝑡(𝑤, “𝑏𝑎”)
• all occurrences of “ab” start on an even position.
– ∀𝑝: ((1 < 𝑝 < 𝐿𝑒𝑛𝑔𝑡ℎ 𝑤 ∧ 𝑐𝑜𝑛𝑡𝑎𝑖𝑛𝑠 𝑤, “𝑎𝑏”, 𝑝) ⇒
𝑝 𝑚𝑜𝑑 2 = 0 )
• consists of n a’s followed by n b’s.
– 𝑤 = 𝑐𝑜𝑛𝑐𝑎𝑡𝑒𝑛𝑎𝑡𝑒(𝑟𝑒𝑝𝑒𝑎𝑡(“𝑎”, 𝑛), 𝑟𝑒𝑝𝑒𝑎𝑡(“𝑏”, 𝑛))
9
Outline
• Formal Description of Input
• Regular Languages
 Algorithm for automata synthesis.
• Non-regular Languages
– Formal description of non-regularity proof.
– Algorithm for proof synthesis.
10
Automata Generation for Regular Languages
We use Angluin’s L* algorithm.
• Membership Oracle
– Provided by the PDL interface.
• Equivalence Oracle
– Can be simulated using Membership Oracle on bit-strings
of size at most 2n, where n is the # of automata states.
– Theorem: Let 𝐴 be a DFA with 𝑘1 states, let 𝐵 be a DFA
with 𝑘2 states. If 𝐿 𝐴 ≠ 𝐿 𝐵 , then there exists a word
𝑤 in L A − L B ∪ 𝐿 𝐵 − 𝐿 𝐴 , such that the length
of 𝑤 is at most 𝑘1 + 𝑘2 − 1.
11
Distribution of Automata sizes
35
30
25
# of 20
problems 15
10
5
0
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16
Automata Size
The automata size of educational problems is small!
12
Distribution of counterexample lengths
(relative to automata size)
40
35
30
25
# of
20
problems
15
10
5
0
-11 -10 -9 -8 -7 -6 -5 -4 -3 -2
-1
0
1
2
Counterexample length – automata size
The counterexample lengths are quite smaller than
worst-case possibility of twice the automata size.
13
Outline
• Formal Description of Input
• Regular Languages
– Algorithm for automata synthesis.
• Non-regular Languages
 Formal description of non-regularity proof.
– Algorithm for proof synthesis.
14
Myhill-Nerode Theorem: Non-regularity Condition
A language 𝐿 is non-regular iff there exist functions
𝐹: 𝑁 → Σ ∗ and 𝐺: 𝑁 → Σ ∗ such that:
∀𝑖, 𝑗: 1 ≤ 𝑗 < 𝑖 ⇒
𝐹 𝑖 𝐺 𝑖, 𝑗 ∈ 𝐿 ∧ 𝐹 𝑗 𝐺 𝑖, 𝑗 ∉ 𝐿
𝑜𝑟
𝐹 𝑖 𝐺 𝑖, 𝑗 ∉ 𝐿 ∧ 𝐹 𝑗 𝐺 𝑖, 𝑗 ∈ 𝐿
The above characterization can be shown equivalent to:
∀𝑖, 𝑗: 1 ≤ 𝑗 < 𝑖 ⇒ 𝐹 𝑖 𝐺 𝑖, 𝑗 ∈ 𝐿 ∧ 𝐹 𝑗 𝐺 𝑖, 𝑗 ∉ 𝐿
𝑜𝑟
∀𝑖, 𝑗: 1 ≤ 𝑗 < 𝑖 ⇒ 𝐹 𝑖 𝐺 𝑖, 𝑗 ∉ 𝐿 ∧ 𝐹 𝑗 𝐺 𝑖, 𝑗 ∈ 𝐿
This allows for automation!
We refer to F/G as congruence/witness functions.
15
Examples of Congruence/Witness Functions
• Same number of occurrences of “a” as that of “b”.
Congruence Function: 𝑎𝑖
Witness Function: 𝑏 𝑖
• Same number of occurrences of “a” as that of “b”.
Congruence Function: 𝑎𝑖 𝑎
Witness Function: 𝑏 𝑖
• 𝑤𝑤 𝑅 𝑤 ∈ 0,1 ∗ }.
Congruence Function: 𝑎𝑖 𝑏
Witness Function: 𝑏𝑎𝑖
16
Language for Congruence/Witness Functions
The following language is expressive enough to
represent Congruence function F(i) and Witness
Function G(i,j) for several classroom problems.
𝑆 ≔ 𝑡1 𝑡2 … 𝑡𝑛
𝑡 ≔ 𝑐 𝑐𝑖 𝑐𝑗
where c is any alphabet.
17
Outline
• Formal Description of Input
• Regular Languages
– Algorithm for automata synthesis.
• Non-regular Languages
– Formal description of non-regularity proof.
 Algorithm for proof synthesis.
18
Algorithm for constructing Congruence/Witness Fns.
1. Approximation
–
Run L* algorithm using counterexamples of size ≤ 𝑘.
2. Generation (using Inductive Synthesis)
– For each eq. class, generate all congruence functions
that evaluate to a string in that class. Intersect pairs of
such sets to generate candidates.
– For each candidate congruence fn., use similar
methodology to generate candidate witness fn.
3. Testing based Validation
–
Test correctness of candidate fns. on 1 ≤ 𝑗 < 𝑖 ≤ 10
4. Verification
–
Use symbolic inference rules to verify correctness.
5. Refinement
–
If any of steps 2,3,4 fail, repeat with larger value of k.
19
Experimental Results
Id
PDL
1
UFPDL
5
0.1
𝑎𝑖
𝑏𝑖
0.1
0.1
2
UFPDL
5
0.2
𝑎𝑖
𝑏𝑖
0.1
0.1
3
UFPDL
5
0.9
𝑎𝑖
𝑏𝑖
0.1
0.1
3
CFG
5
0.4
𝑎𝑖
𝑏𝑖
0.1
0.1
4
UFPDL
5
1.6
𝑏𝑖
𝑎𝑖
0.1
0.2
5
CFG
10
0.2
𝑎𝑖 𝑏
𝑏𝑎𝑖
0.1
0.1
6
CFG
5
0.1
𝑏𝑖
𝑎𝑖
0.1
0.1
9
CFG
10
0.1
𝑎𝑖 𝑏
𝑎𝑖
0.1
0.1
11
UFPDL
8
12.1
𝑎𝑏 𝑖
𝑐𝑖
0.4
0.1
12
UFPDL
5
10.1
𝑎𝑏
𝑎𝑖
0.4
0.2
13
UFPDL
5
6.4
𝑎𝑖
𝑏𝑖 𝑐 𝑗
0.2
0.1
15
UFPDL
7
7.8
𝑎𝑖 𝑏
𝑎𝑖
1.1
0.1
15
CFG
7
0.1
𝑎𝑖
𝑏𝑎𝑖
0.1
0.1
16
CFG
5
0.2
𝑎𝑖
𝑏𝑖
0.1
0.1
17
UFPDL
5
1.6
𝑎𝑖
𝑏𝑖
0.4
0.1
18
CFG
5
0.3
𝑎𝑖 𝑏
𝑎𝑖
0.1
0.1
19
UFPDL
10
15.8
𝑎𝑏 𝑖
𝑐𝑖
4.5
1.7
21
CFG
5
0.5
(𝑖
)𝑖
0.1
0.1
𝑵≡
Generation
Congruence Fn.
𝑖
Witness Fn. Test Verify
20
Intelligent Tutoring Systems
• Aspects
–
–
–
–
Solution Generation
Problem Generation
Automated Grading
Content Entry
• Domains
–
–
–
–
–
–
Geometry
Algebra
Introductory Programming
Automata Theory
Physics
Chemistry
⋮
AAAI 2012: Singh, Gulwani, Rajamani.
21
Trigonometry Problem
Example Problem: sec 𝑥 + cos 𝑥
Query: 𝑇1 𝑥 ± 𝑇2 (𝑥)
𝑇1 ≠ 𝑇5
sec 𝑥 − cos 𝑥 = tan2 𝑥 + sin2 𝑥
𝑇3 𝑥 ± 𝑇4 𝑥
= 𝑇52 𝑥 ± 𝑇62 (𝑥)
New problems generated:
csc 𝑥 + cos 𝑥 csc 𝑥 − cos 𝑥 = cot 2 𝑥 + sin2 𝑥
(csc 𝑥 − sin 𝑥)(csc 𝑥 + sin 𝑥) = cot 2 𝑥 + cos 2 𝑥
(sec 𝑥 + sin 𝑥)(sec 𝑥 − sin 𝑥) = tan2 𝑥 + cos 2 𝑥
:
(tan 𝑥 + sin 𝑥)(tan 𝑥 − sin 𝑥) = tan2 𝑥 − sin2 𝑥
(csc 𝑥 + cos 𝑥)(csc 𝑥 − cos 𝑥) = csc 2 𝑥 − cos 2 𝑥
:
22
Algebra Problem Generation
Example Problem
Query Generation
Query
Refined Query
Query Execution
New Problems
Results OK?
No
Query Refinement
Yes
Similar Problems
23
Limits/Series Problem
𝑛
Example Problem:
𝑛
Query:
lim
𝑛→∞
𝑖=0
lim
𝑛→∞
𝑖=0
2𝑖 2 + 𝑖 + 1
5
=
𝑖
2
5
𝐶0 𝑖 2 + 𝐶1 𝑖 + 𝐶2
𝐶3 𝑖
𝐶4
=
𝐶5
C0 ≠ 0 ∧ gcd 𝐶0 , 𝐶1 , 𝐶2 = gcd 𝐶4 , 𝐶5 = 1
New problems generated:
𝑛
lim
𝑛→∞
𝑖=0
𝑛
lim
𝑛→∞
𝑖=0
𝑛
2
3𝑖 + 2𝑖 + 1
7
=
𝑖
3
7
lim
𝑛→∞
𝑛
2
𝑖
3
=
𝑖
2
3
𝑖=0
lim
𝑛→∞
𝑖=0
3𝑖 2 + 3𝑖 + 1
=4
𝑖
4
5𝑖 2 + 3𝑖 + 3
=6
𝑖
6
24
Integration Problem
Example Problem:
Query:
(csc 𝑥) (csc 𝑥 − cot 𝑥) 𝑑𝑥 = csc 𝑥 − cot 𝑥
𝑇0 𝑥 𝑇1 𝑥 ± 𝑇2 𝑥 𝑑𝑥 = 𝑇4 𝑥 ± 𝑇5 (𝑥)
𝑇1 ≠ 𝑇2 ∧ 𝑇4 ≠ 𝑇5
New problems generated:
(tan 𝑥) (cos 𝑥 + sec 𝑥) 𝑑𝑥 = sec 𝑥 − cos 𝑥
(sec 𝑥) (tan 𝑥 + sec 𝑥) 𝑑𝑥 = sec 𝑥 + cot 𝑥
(cot 𝑥) (sin 𝑥 + csc 𝑥) 𝑑𝑥 = sin 𝑥 − csc 𝑥
25
Determinant Problem
Ex. Problem
𝑥+𝑦
𝑧𝑥
𝑦𝑧
2
𝑧𝑥
𝑦+𝑧
𝑥𝑦
𝐹0 (𝑥, 𝑦, 𝑧) 𝐹1 (𝑥, 𝑦, 𝑧)
Query 𝐹3 (𝑥, 𝑦, 𝑧) 𝐹4 (𝑥, 𝑦, 𝑧)
𝐹6 (𝑥, 𝑦, 𝑧) 𝐹7 (𝑥, 𝑦, 𝑧)
2
𝑧𝑦
𝑥𝑦
𝑧+𝑥
= 2𝑥𝑦𝑧 𝑥 + 𝑦 + 𝑧
3
2
𝐹2 (𝑥, 𝑦, 𝑧)
𝐹5 (𝑥, 𝑦, 𝑧)
𝐹8 (𝑥, 𝑦, 𝑧)
= 𝐶10 𝐹9 (𝑥, 𝑦, 𝑧)
𝐹𝑖 ≔ 𝐹𝑗 𝑥 → 𝑦; 𝑦 → 𝑧; 𝑧 → 𝑥 𝑤ℎ𝑒𝑟𝑒 𝑖, 𝑗 ∈ { 4,0 , 8,4 , 5,1 , … }
New problems generated:
𝑦2
𝑧+𝑦
𝑧2
2
𝑦𝑧 + 𝑦 2
𝑦𝑧
𝑧𝑥
𝑥2
𝑧2
𝑥+𝑧
𝑥𝑦
𝑧𝑥 + 𝑧 2
𝑧𝑥
2
𝑦+𝑥
𝑦2
𝑥2
𝑥𝑦
𝑦𝑧
𝑥𝑦 + 𝑥 2
2
= 2 𝑥𝑦 + 𝑦𝑧 + 𝑧𝑥
3
= 4𝑥 2 𝑦 2 𝑧 2
26
Intelligent Tutoring Systems
• Aspects
–
–
–
–
Solution Generation
Problem Generation
Automated Grading
Content Entry
• Domains
–
–
–
–
–
–
Geometry
Algebra
Introductory Programming
Automata Theory
Physics
Chemistry
⋮
Arxiv TR 2012: Rishabh Singh, Gulwani, Armando Solar-Lezama .
27
Background: PexForFun
28
Buggy Program for Array Reverse
using System;
6:28::50 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
29
Buggy Program for Array Reverse
using System;
6:32::01 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
30
Buggy Program for Array Reverse
using System;
6:32::32 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Sign of Frustation?
31
Buggy Program for Array Reverse
using System;
6:33::19 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i <= a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
32
Buggy Program for Array Reverse
using System;
6:33::55 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
Same as initial attempt except Console.Writeline!
33
Buggy Program for Array Reverse
using System;
6:34::06 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
No change! Sign of Frustation?
34
Buggy Program for Array Reverse
using System;
6:34::56 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i <= a.Length; i--)
{
Console.Writeline(i);
b[count] = a[i];
count++;
}
return b;
}}
The student has tried this before!
35
Buggy Program for Array Reverse
using System;
6:36::24 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Same as initial attempt!
36
Buggy Program for Array Reverse
using System;
6:37::39 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length-1; i < a.Length-1; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
The student has tried this before!
37
Buggy Program for Array Reverse
using System;
6:38::11 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i > 0; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Almost correct! (a[i-1] instead of a[i] in loop body)
38
Buggy Program for Array Reverse
using System;
6:38::44 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i >= 0; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Student going in wrong direction!
39
Buggy Program for Array Reverse
using System;
6:39::33 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
Back to bigger error!
40
Buggy Program for Array Reverse
using System;
6:39::45 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Frustation!
41
Buggy Program for Array Reverse
using System;
6:40::27 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! More Frustation!!
42
Buggy Program for Array Reverse
using System;
6:40::57 AM
public class Program {
public static int[] Puzzle(int[] a) {
int[] b = new int[a.Length];
int count = 0;
for(int i=a.Length; i < a.Length; i--)
{
b[count] = a[i];
count++;
}
return b;
}}
No change! Too Frustated now!!! Gives up.
43
Proposal: Semantic Grading
Provides additional value over counterexample
feedback.
• More friendly feedback.
– Helpful for students who give up after several tries
(with only counterexample feedback).
• Grading
– Counterexample feedback does not distinguish
between a slightly incorrect solution and one that is
very far off from being correct.
44
Demo
45
Relation with Automated Bug Fixing
Simplifying Assumptions
• Correct solution is known.
• Errors are predictable.
• Programs are small.
Challenging Aspects
• No logical specification.
– Instead program equivalence.
• Higher density of errors than production code.
• Generate multiple fixes
– To remain faithful to the student’s thought process.
46
Technique
• Teacher provides:
– A reference implementation
– Model of errors that students make
• Sketch encoding:
– Fuzz the program using error model.
– Use a counter to keep track of number of changes.
• Sketch solving:
– To generate minimal fixes, assert (counter = i) for
increasing values of i.
– Use off-the-shelf SAT solver to explore the state
space of possible corrections.
47
Error Model for Array Reverse Problem
Array Index Fuzzing:
v[a] -> v[{a+1, a-1, v.Length-a-1}]
Initialization Fuzzing:
v=n -> v={n+1, n-1, 0}
Increment Fuzzing:
v++ -> { ++v, v--, --v }
Return Value Fuzzing:
return v -> return ?v
Conditional Fuzzing:
a op b -> a’ ops { a+1, a-1, 0 }
where ops = { <, >, <=, >=, ==, != }
48
Fraction of Programs fixed
Effectiveness of Error Models
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Array Reverse
Palindrome
Max
Factorial
isIncreasing
Sort
F1
F2
F3
F4
F5
Error Models
49
Efficiency of Error Models
Running Time (in s)
35
30
Array Reverse
25
Palindrome
20
Max
15
isIncreasing
10
Factorial
Sort
5
0
F1
F2
F3
F4
F5
Error Models
50
Number of Programs Fixed
Generality of Error Models
80
70
60
50
40
30
20
Specific Error
Model
Array Reverse
Error Model
10
0
51
Experimental Results
Benchmark
Total Fixed
Changes Time(s)
Array Reverse
305
254
1.73
2.69
String Palindrome
86
64
1.28
3.52
Array Maximum
99
68
1.25
7.47
Is Increasing Order
51
38
1.72
3.56
Array Sort
74
35
1.17
32.46
Factorial
70
30
1.25
4.99
Friday Rush
17
4
1
12.42
52
Potential Workflow
Teacher grades an ungraded answer-script.
System generalizes corrections into error models.
System performs automated grading by
considering all possible combinations and
instantiations of all error models.
Any ungraded
answer scripts?
Yes
No
53
Intelligent Tutoring Systems
• Aspects
–
–
–
–
Solution Generation
Problem Generation
Automated Grading
Content Entry
• Domains
–
–
–
–
–
–
Geometry
Algebra
Introductory Programming
Automata Theory
Physics
Chemistry
⋮
Joint work with: Alex Polozov and Sriram Rajamani
54
Mathematical Intellisense
State-of-the-art Mathematical Editors
• Text editors like Latex
– Unreadable text in prefix notation
• WYSIWIG editors like Microsoft Word
– Change of cursor positions multiple times.
– Back and forth switching between mouse & keyboard
Our proposal: An intelligent predictive editor.
• Mathematical text has low entropy and hence
amenable to prediction!
55
Reducing (Term) Prediction to Learning-By-Examples
Terms connected by the same AC operator can be
thought of as terms belonging to a sequence.
There are 2 opportunities for predicting such terms.
Sequence Creation: T1, T2, T3, …
• Learn a function F such that F(Ti) = Ti+1
Sequence Transformation: T1, T2, T3, T4 -> S1, S2, …
• Learn a function F such that F(Ti) = Si
56
Mathematical (Syntactic) Intellisense
tan 3𝑥 tan 2𝑥 tan 𝑥 = tan 3𝑥 − tan 2𝑥 − tan 𝑥
𝑦𝑧 − 𝑥 2
𝑧𝑥 − 𝑦 2
𝑥𝑦 − 𝑧 2
𝐴1 sin3 𝛼
𝐴2 sin 𝛼
𝑧𝑥 − 𝑦 2
𝑥𝑦 − 𝑧 2
𝑦𝑧 − 𝑥 2
𝐵1 sin3 𝛽
𝐵2 sin 𝛽
𝑥𝑦 − 𝑧 2
𝑦𝑧 − 𝑥 2
𝑧𝑥 − 𝑦 2
𝐶1 sin3 𝛾
𝐶2 sin 𝛾
57
Mathematical (Semantic) Intellisense
Prove (csc 𝑥 − sin 𝑥)(sec 𝑥 − cos 𝑥)(tan 𝑥 + cot 𝑥) = 1
L.H.S. =
1
− sin 𝑥
sin 𝑥
=
1 − sin2 𝑥
sin 𝑥
=
cos 2 𝑥
sin 𝑥
1
− cos 𝑥
cos 𝑥
1 − cos2 𝑥
cos 𝑥
sin2 𝑥
cos 𝑥
sin 𝑥 cos 𝑥
+
cos 𝑥 sin 𝑥
sin2 𝑥 + cos 2 𝑥
cos 𝑥 sin 𝑥
1
cos 𝑥 sin 𝑥
=1
58
Long-term Goals
• Ultra-intelligent computer
• Model of human mind
• Inter-stellar travel 
59