User Interaction Models for
Disambiguation in Programming by Example
UIST, Charlotte
Tuesday, November 10th, 2015
Mikaël Mayer
Joint work with Gustavo Soares, Maxim Grechkin, Vu Le, Mark Marron,
Oleksandr Polozov, Rishabh Singh, Ben Zorn, Sumit Gulwani
Billion of users with computers
No programming expertise
Challenge:
Repetitive tasks
Data wrangling
"[Programming by example] can give ordinary users
the ability to write programs while still operating
in the familiar user interface.“
Henry Lieberman, 2001
For more than 20 years….
An exception? Flash Fill (Microsoft Excel 2013)
Encouraging Initial Response
Flash Fill problems
“One of the most controversial new features in Excel 2013”
“Be very careful.”
“There might be exceptions that you don't notice unless you
examine the results very carefully.“
John Walkenbach
ConvertFrom-String
(underlying: FlashExtract)
ConvertFrom-String
Microsoft MVPs reaction:
“This is super cool!”
“One of the best improvement that came with WMF5.0“
“We’re running into an issue (…) We could look at the debug
output ”
ESSL((EndsWith(Dynamic Token(</td><td>)(</td><td>), ALL CAPS(\p{Lu}(\p{Lu})+),
Dynamic Token(</td></tr>)(</td></tr>))): 0,
1, ...: Dynamic Token(<tr><td>)(<tr><td>)...Alphabet([\p{Lu}\p{Ll}\-.]+),
Dynamic Token(</td><td>)(</td><td>), ALL CAPS(\p{Lu}(\p{Lu
})+), 1 + Camel Case(\p{Lu}(\p{Ll})+)...Dynamic Token(</td><td>)(</td><td>),
ALL CAPS(\p{Lu}(\p{Lu})+), Dynamic Token(</td></tr>)(</
td></tr>), 1)
“If you can understand this, you’re a better person than I am”
“However, while these [PbE] systems have resulted in
many publications in AI venues, none of the
technologies have yet achieved widespread adoption.
Usability remains a critical barrier to their success”
Tessa Lau, 2009.
Super Bowl III, IV, V are missing
Unexpected HTML comment
resulting in different line ending
Unexpected HTML comment
resulting in different line ending
Unexpected HTML comment
resulting in different line ending
T. Lau, 2009:
 Detect failure and fail gracefully
 Make it easy to correct the system
Paraphrasing challenges
T. Lau, 2009:
 Models users can understand
 Get rid of arcane instructions (“The char 1”)
Char(n)  “The {:0.charNum}”
charNum: [{regex: "^1$",
result: "beginning“},
{regex: "^(\d+)$", result: “{:1.ordinal} char“}],
Char(1) “The beginning”
Char(2)  “The second char”
Char(-3)  “The third char from the end”
Paraphrasing challenges
 Higher-level description rather lower level editing commands.
PosPair(Pos(Line(1), 1), Pos(Line(1),-1))
“extract the string between (the (beginning) of (second line)) and
(the (end) of (second line)))”
PosPair(Pos(?L, 1), Pos(?L, -1)) when {:0:L}={:1:L}
 “extract the {:L}”
“extract the (second line)”
Towards conversational clarification (disambiguation)
Input
Prog
Output
Towards conversational clarification (disambiguation)
Union
Join
Program set
Billions of programs

Towards conversational clarification (disambiguation)
Union + Rank
Join
2 best program variants
Program set
1234
123
1
12
#1
12
#2
1
1
1
1
2
1
1
1
2
1
1
#1 #2 #1 #1
Conversational clarification
Name:Mikael
Name:Gustavo
Name:McFly
#1
Name:Mikael
Name:Gustavo
Name:McFly
McFly or McFly
or McFly
Name:Mikael
Name:Gustavo
Name:McFly
Compute &
run programs
Compare outputs
against #1
or custom?
Ask clickable question
User study set up
29 participants
3 tasks with ambiguities
3 conditions
With Conversational Clarification (disambiguation questions)
With Program Navigation
(view and change the program)
Without new feature.
User study set up
0
2
With Program Navigation
With Conversational Clarification
4
Without new feature
6
8
Errors
1.
The correctness increases
when using one of the two new features.
2.
Conversational Clarification is perceived more useful
than Program Navigation
Conversational Clarification: 5.4 / 7
Program Navigation:
4.2 / 7
“The program [is] quite
understandable but it was not
clear how to modify the program”
3.
9 out of 13 users did not try to
modify the program
Conversational Clarification increases confidence
In the PBE system
W = 181.5, p = 0.07
“In the last example, in which I didn't have
[Conversational Clarification] as an option,
I felt like I miss it so much”
“It always helps me to find the right
matching”
Thank you
Questions
PROgram Synthesis by Example
http://microsoft.github.io/prose
http://bit.ly/microsoftprose
Answers
Story
Flash Fill (2013)
Flash Fill(2011)
FlashMeta
Flash Extract (2013)
PowerShell ConvertFrom-String (2014)
FlashRelate (2014)
FlashExtractWeb (2015)