http://pex4fun.com
Pex4Fun:
Teaching and Learning
Computer Science
via Social Gaming
Nikolai Tillmann, Jonathan de Halleux,
Judith Bishop, Michal Moskal
Microsoft Research
Tutorial
Tao Xie
NCSU
http://bit.ly/ioa4qf
http://pex4fun.com
Agenda
Pex4Fun:
Teaching and Learning
• Background:
Computer Science
Code Analysis by
via Social Gaming
Dynamic Symbolic Execution
•
•
•
•
•
Writing Code in a Browser
Coding Duels
Social Experience
Teaching Platform
Conclusion
Pex – The Pex4Fun Engine – History
DYNAMIC SYMBOLIC EXECUTION
IN THE CLOUD
Before the cloud…
http://research.microsoft.com/pex/
http://pex4fun.com/CoverMe
Dynamic Symbolic Execution in Pex
Choose next path
 Generates test data systematically
Solve
void CoverMe(int[] a)
{
if (a == null) return;
if (a.Length > 0)
if (a[0] == 1234567890)
throw new Exception("bug");
}
F
F
a.Length>0
a==null
T
T
Execute&Monitor
Constraints to solve
Input
Observed constraints
a!=null
null
{}
a==null
a!=null &&
!(a.Length>0)
a==null &&
a.Length>0 &&
a[0]!=1234567890
a==null &&
a.Length>0 &&
a[0]==1234567890
a!=null &&
a.Length>0
{0}
a!=null &&
a.Length>0 &&
a[0]==123456890
{123…}
Done: There is no path left.
a[0]==123…
F
T
http://pex4fun.com/HowDoesPexWork
Dynamic Symbolic Execution Exercises
• ArrayIndexLength
Pex knows about all implicit, exceptionthrowing control-flow branches
• ArrayHeap
Pex models the heap
• Assert, Assert123
Assertions connect code coverage and
correctness
http://pex4fun.com/DynamicSymbolicExecutionExercises
Limitations
• 16K characters of code
(single file / editor window) *
• Single-threaded only
• No environment interactions *
• No non-determinism *
• Try to avoid floating-point computations
* Limitation only applies to pex4fun.com, but not Pex tool
Dynamic Symbolic Execution
Summary
• “Ask Pex” sends code to cloud
• Code is compiled and analyzed in cloud
• Dynamic Symbolic Execution automatically
finds relevant interesting test inputs that
achieve high code coverage
• Results are shown in browser
Code Auto-Completion
WRITING CODE
IN A BROWSER
Auto-Completion
Writing Code in Browser
Exercise
1. Go to http://pex4fun.com
2. Click “New”
3. Write some code with public static “Puzzle” method
using System;
class Vector { int x, y;
public Vector(int x, int y) { this.x = x; this.y = y; }
public int Length() { return (int)Math.Sqrt(x*x+y*y); }
public Vector Normalize() { return new Vector(x/Length(), y/Length()); }
public bool Equals(Vector other) { return this.x==other.x && this.y == other.y; }
}
public class Program {
public static void Puzzle(Vector v) {
var n1 = v.Normalize();
var n2 = n1.Normalize();
if (!n1.Equals(n2)) throw new Exception(“bug in normalize”);
}
}
4. “Ask Pex!”
http://bit.ly/lOgPZJ
http://pex4fun.com/ParameterizedUnitTesting
Writing Code in Browser
Summary
• Code is compiled in cloud
• Code is executed in cloud
• Auto-completion via cloud/Javascript
in browser
Fun and Engaging Serious Game – Win by Writing Code
CODING DUELS
Coding Duels
Coding Duels
• Pex computes “semantic diff” in cloud
 code written in browser vs.
 secret reference implementation
• You win when there are no differences
Coding Duels
class VisibleProgram {
public static int Puzzle1(int x) {
return x;
} }
class SecretProgram {
public static int Puzzle2(int x) {
return x * x;
} }
public class MetaProgram {
public static void Puzzle(int x) {
if (VisibleProgram.Puzzle1(x) != SecretProgram.Puzzle2(x))
throw new System.Exception("mismatch!");
} }
Coding Duels
Teaching Debugging/Problem Solving Skills
• From Hypothesis to Theorem
– Step 1: Inference of hypothesized (delta) requirement
• When delta: Inference of failure-inducing conditions
• If difficult: forced generation of new data points
– By adding data-point-fitting conditionals
– Step 2: Writing of requirement-satisfying solution
• When delta:
– Derivation of hypothesized faulty line(s) of code
– Derivation of patches on the faculty line(s) of code
– Step 3: “Ask Pex” to validate the hypothesis
Initial almost-complete (delta) working implementation helps teach debugging skills
Coding Duels
Teaching Abstraction Skills
• Step 1: Inference of hypothesized (delta)
requirement from observed data points
– Pattern recognition
– Generalization from data points to formula
– Inversion of constraint solving
Teaching Programming, Algorithm, and
Program Understanding Skills
• Step 2: Writing of requirement-satisfying solution
Teacher can give guidance:
• Providing hints in comments
• Providing initial almost-complete
working implementation
• Many choices,
e.g. recursion vs. loop implementation
Coding Duels
Exercise
1.
2.
3.
Go to http://pex4fun.com; optionally: sign in with Windows Live ID
Click “New”
Write secret implementation with public static “Puzzle” method
using System;
public class Program {
public static int Puzzle(int x, int y) { return x + y – 42; }
}
4.
5.
6.
7.
8.
9.
10.
“Ask Pex!”
Enter Coding Duel Name
Click “Turn This Puzzle Into A Coding Duel”
Open Permalink
Edit Description
Click “Permalink”
Use it, or create ulta-short version, e.g. with http://bit.ly
http://pex4fun.com/CreatingAndPublishingCodingDuels
Coding Duels
Fun and Engaging
• Iterative gameplay
• Adaptive
• Personalized
• No cheating
• Clear winning criterion
Coding Duels
Summary
• Coding Duel: write function that matches
specification given by another implementation
• Semantic equivalence checked in cloud
• Student’s tasks:
– Inference of hypothesized (delta) requirement
– Writing of requirement-satisfying solution
http://pex4fun.com/CreatingAndPublishingCodingDuels
http://pex4fun.com/TipsAndTricksForGreatDuels
Connecting Students – Competitive yet Self-Paced Environment
SOCIAL EXPERIENCE
Social Experience
Social Experience
Summary
• Community
• High score lists, leaderboard
• Live feed
http://pex4fun.com/Community.aspx
http://pex4fun.com/Livefeed.aspx
Pages – Courses – Automated Assessments
PEX4FUN, A TEACHING PLATFORM
Teaching
Teaching
Summary
• How to become a teacher:
– Sign in, choose nickname
– Send us email with nickname
• Teachers can
– Create courses
– Reuse existing or author existing pages
– Get automated assessments of students
– See students code
http://pex4fun.com/TeachingWithPexForFun
Outlook I
• Analyzing code submissions
– Detecting duplicate submissions
– Ranking submissions (how correct, how beautiful)
– If faulty, give user guidance on “the next step”.
Outlook II
• Better code development experience
– Debugging
– Dynamic input/output table
– More languages
• Beyond the web
– Phone
More Reading
Nikolai Tillmann, Jonathan de Halleux, and Tao Xie
Pex for Fun: Engineering an Automated Testing
Tool for Serious Games in Computer Science
Microsoft Research, Technical report
MSR-TR-2011-41, March 2011
http://research.microsoft.com/apps/pubs/?id=147143
Conclusion
• Taking programming into the browser, cloud
– .NET: C# (with Intellisense), Visual Basic, F#
– Execution, Code analysis with Pex
• Fun learning experience with Coding Duels
• Social experience: Live feed, sharing duels
• Teaching with automatic grading
• Create your own courses!
http://pex4fun.com