PinADX: Customizable Debugging with
Dynamic Instrumentation
Gregory Lueck, Harish Patil, Cristiano Pereira
Intel Corporation
CGO 2012, San Jose, USA
Software & Services Group
1
Hypothetical Problem 1
(gdb) run
Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x0000000000000000 in ?? ()
Crash with bad PC and no stack trace
Corrupted return address someplace ...
Want to stop BEFORE bad “ret” instruction
Software & Services Group
2
Hypothetical Problem 2
thread stack thread stack thread stack
?
?
?
thread stack
...
?
Massively threaded application
How much stack space needed?
At what point does each thread use its max stack?
Software & Services Group
3
Traditional Debugger Breakpoints?
Original
Application
Foo:
Bar:
…
ret
…
ret
Application
In Debugger
Overwrite
each “ret”
with trap
Foo:
…
trap
ret
Bar:
1.
2.
3.
Debugger catches trap
Check if “ret” is to good PC
If yes, resume
…
ret
trap
How can debugger find all “ret” instructions?
Horribly slow to trap on each return
Software & Services Group
4
Dynamic Binary Instrumenation (DBI)
Application
Foo:
Bar:
…
ret
…
ret
Application
if (return to bad PC)
Breakpoint()
Instrumentation
Foo:
…
sub 0x60, %sp
if (stack too big)
Breakpoint()
Bar:
…
sub 0x10, %sp
Much faster – avoids trap overhead
DBI can find all “ret” instructions reliably
General approach – solves stack problem (and others)
BUT difficult to integrate with debugger
Software & Services Group
5
Pin Overview
Instrument
Fetch
Code
Cache
Optimize
Application
Tool
Store &
JIT
compiler execute
Traces
Tool controls
instrumentation
(e.g. “if return
to bad PC”)
Instrumented
instructions
stored in code
cache for
efficiency
JIT compiler fetches application
instructions, calls tool to
instrument
Software & Services Group
6
JIT Compiler Overview
Original Code
Code Cache
1
1’
3’
2
2’
5’
3
4
Tool inserts
instrumentation
(e.g. check if
return
to bad PC)
Dynamic
recompilation
makes debugging
hard
6’
5
6
Pin
Software & Services Group
7
PinADX Architecture
Application
Process running under
Pin
Tool extends debugger via
instrumentation.
Debugger
Tool
PinADX
core
Pin
PinADX presents “pure” view of application.
Hides effect of instrumentation and
recompilation.
GDB
or
Microsoft Visual
Studio 11
Supports Linux & Windows
Software & Services Group
8
Rest of the Talk
•
•
•
•
Introduction / Motivation
Example: Using “Stack Debugger” extension
Example: Authoring “Stack Debugger” extension
Implementing PinADX
Software & Services Group
9
Example – Stack Debugger
$ pin –appdebug –t stack-debugger.so -- ./my-application
Application stopped until continued from debugger.
Start GDB, then issue this command at the (gdb) prompt:
target remote :1234
Run application
under Pin
$ gdb ./my-application
(gdb) target remote :1234
Debugger
(gdb) break
(gdb)PrintHello
break PrintHello
connected to
Pin
Breakpoint
1 at 0x4004dd: file
file hw.c,
line 13
Breakpoint
1 at 0x4004dd:
hw.c,
line 13
(gdb) cont
(gdb) cont
Breakpoint 1, PrintHello () at hw.c:13
(gdb)
backtrace
Breakpoint
1, PrintHello () at hw.c:13
(gdb) backtrace
#0 PrintHello
() at ()hw.c:13
#0 PrintHello
at hw.c:13
main () at hw.c:7
(gdb)
x/2i#1
#1 main
()$pc
at
hw.c:7
(gdb) x/2i $pc
=> 0x4004dd
<PrintHello+4>:
mov
$0x4005e8,%edi
=> 0x4004dd
<PrintHello+4>: mov
$0x4005e8,%edi
0x4004e2
<PrintHello+9>: callq
0x4003b8
0x4004e2
<PrintHello+9>:
callq
0x4003b8
Software & Services Group
10
Example – Stack Debugger
Stop when application uses too much stack
(gdb) monitor
stackbreak
(gdb) monitor
stackbreak4000
4000
Breakthread
when thread
uses4000
4000 stack
bytesbytes
Break when
uses
stack
(gdb) cont
(gdb)
Stopped: Thread uses 4004 bytes of stack
(gdb) cont
backtrace
(gdb)
backtrace
Stopped:
Thread
4004 bytes of stack()
#0 0x3f07214445uses
in _dl_runtime_resolve
#0 0x3f07214445 in _dl_runtime_resolve ()
#1 0x00004004e7
in PrintHello () at
#1 0x00004004e7
in PrintHello
()hw.c:13
at hw.c:13
(gdb) monitor
stackbreak
10000
#2 0x00004004d2 in main () at hw.c:7
#2 0x00004004d2
in main10000
() at hw.c:7
(gdb) monitor stackbreak
Break when thread uses 10000 stack bytes
Break when thread uses 10000 stack bytes
(gdb) break
(gdb)exit
break exit
Breakpoint 2 at 0x7fffe60f9650
Breakpoint
2 at 0x7fffe60f9650
(gdb) cont
(gdb) monitor
cont
Breakpoint 2, 0x7fffe60f9650 in exit ()
(gdb)
stats
(gdb) monitor stats
Breakpointstack
2, 0x7fffe60f9650
in exit ()
Maximum
usage:
8560
bytes.
Maximum stack usage: 8560 bytes.
Software & Services Group
11
Rest of the Talk
•
•
•
•
Introduction / Motivation
Example: Using “Stack Debugger” extension
Example: Authoring “Stack Debugger” extension
Implementing PinADX
Software & Services Group
12
Stack Debugger – Instrumentation
Thread Start:
Record initial stack
StackBase = %esp;
MaxStack = 0;
[…]
sub $0x60, %esp
After each stack-changing
instruction
size = StackBase - %esp;
if (size > MaxStack) MaxStack = size;
if (size > StackLimit) TriggerBreakpoint();
cmp %esi, %edx
jle <L1>
Software & Services Group
13
Stack Debugger – Implementation
Analysis
Instrumentation
Instrument only instructions
Call after each instruction
VOID Instruction(INS ins, VOID *)
that change $SP
{
if (INS_RegWContain(ins, REG_STACK_PTR))
{
IPOINT where = (INS_HasFallThrough(ins)) ?
IPOINT_AFTER : IPOINT_TAKEN_BRANCH;
INS_InsertCall(ins, where, (AFUNPTR)OnStackChange,
IARG_REG_VALUE, REG_STACK_PTR,
IARG_THREAD_ID, IARG_CONTEXT, IARG_END);
}
}
VOID OnStackChange(ADDRINT sp, THREADID tid, CONTEXT *ctxt)
{
size_t size = StackBase - sp;
if (size > StackMax) StackMax = size;
if (size > StackLimit) {
ostringstream os;
os << "Stopped: Thread uses " << size << " stack bytes.";
PIN_ApplicationBreakpoint(ctxt, tid, FALSE, os.str());
}
}
Software & Services Group
14
Stack Debugger – Implementation
int main() {
[…]
PIN_AddDebugInterpreter(HandleDebugCommand, 0);
}
BOOL HandleDebugCommand(const string &cmd, string *result) {
if (cmd == "stats")
{
ostringstream os;
os << "Maximum stack usage: " << StackMax << " bytes.\n";
*result = os.str();
return TRUE;
}
else if (cmd.find("stackbreak ") == 0)
{
StackLimit = /* parse limit */;
ostringstream os;
os << "Break when thread uses " << limit << " stack bytes.";
*result = os.str();
return TRUE;
}
return FALSE; // Unknown command
}
Software & Services Group
15
Visual Studio IDE Extension
Software & Services Group
16
Other Debugger Extensions
• Intel Inspector XE Product
– Memory Checker
– Thread Checker
•
•
•
•
•
Intel SDE: Instruction emulation
Debug from log file (PinPlay, CGO 2010)
Dynamic slicing (Rajiv Gupta, UC Riverside)
Cmp$im: Cache simulator
Write your own!
Software & Services Group
17
Rest of the Talk
•
•
•
•
Introduction / Motivation
Example: Using “Stack Debugger” extension
Example: Authoring “Stack Debugger” extension
Implementing PinADX
Software & Services Group
18
PinADX Architecture
Application
Process running under
Pin
Tool extends debugger via
instrumentation.
Debugger
Tool
PinADX
core
Pin
GDB
or
Microsoft Visual
Studio 11
PinADX presents “pure” view of application.
Hides effect of instrumentation and
recompilation.
Software & Services Group
19
Communication Details
Pin
Commands
• Read / write registers, memory
• Set breakpoints
• Continue, single-step, stop
PinADX
core
Debugger
Notifications
• Breakpoint triggered
• Caught signal
• Application exited
• Very low level
• Symbol processing in debugger
• Expression evaluation in debugger
• Extension of GDB’s remote debugging protocol
Software & Services Group
20
Communication Details
Pin
Commands
• Read / write registers, memory
• Set breakpoints
• Continue, single-step, stop
PinADX
core
Debugger
Notifications
• Breakpoint triggered
• Caught signal
• Application exited
Breakpoint alternatives
• Insert real INT3 trap instruction
• Virtualize inside Pin VM
See paper for details
Software & Services Group
21
Breakpoint
Original Code
Code Cache
1
1’
2
2’
5
3
3’
Execution stops in Pin
Waits for GDB to continue
6
BP
4
PinADX
core
set
continue
breakpoint
breakpoint
at
notification
4
Debugger
Software & Services Group
22
Single Step
Original Code
Code Cache
1
1’
2
5
3
6
4
Execution stops in Pin
Waits for GDB to continue
PinADX
core
do single-step
step
complete
notification
Debugger
Software & Services Group
23
Thanks
•
•
•
•
Mark Charney – SDE software emulator
Andria Pazarloglou – Created VS11 GUI plugin
Gregg Miskelly – Microsoft VS11 debugger architect
Robert Cohn – Father of Pin
Software & Services Group
24
Summary
• DBI can implement powerful debugger features
• API allows Pin tools to extend debugger easily
• Multi-platform
– Linux: GDB
– Windows: Microsoft Visual Studio 11 (soon)
• Works with off-the-shelf debuggers
http://pintool.org
25
Software & Services Group