Part 5: Anti-Reverse-Engineering
Chapter 15: Anti-Disassembly
Chapter 16: Anti-Debugging
Chapter 17: Anti-Virtual Machine Techniques
Chapter 18: Packing and Unpacking
Chapter 15: Anti-Disassembly
Anti-Disassembly
1. Understanding Anti-Disassembly
2. Defeating Disassembly Algorithms
3. Anti-Disassembly Techniques
4. Obscuring Flow Control
5. Thwarting Stack-Frame Analysis
1. Understanding Anti-Disassembly
Special code to cause disassembly analysis to produce
incorrect program listings


Goal is to delay or prevent analysis of malicious code
Thwart automated and manual analysis
Tricking disassembly at an incorrect offset

Examples on p. 328 and 329
2. Defeating Disassembly Algorithms
Two types of algorithms
Linear disassembly



Iterate over a block of code, disassembling one instruction at a
time linearly
Decode blindly from start to end, ignores flow-control instructions
that cause only a part of the buffer to execute (Examples on p.
331)
Opcode 0xE8 assumed to be “call”, next 4 bytes assumed to be
target (can instead contain malicious code)
Flow-oriented disassembly



Builds list of locations to assemble by examining code from entry
p. 332: After unconditional jmp, decoding stops
Arbitrary results based on the order in which conditional branches
and calls are followed by disassembler (p. 333, 334)
3. Anti-Disassembly Techniques
Jump instructions with the Same Target



Back-to-back conditional jumps with the same target
jz followed by jnz should be treated as unconditional jmp
(Example on p. 335)
Toggle bytes from code to data using “C” and “U”
Jump instruction with a constant condition


XOR reg,reg followed by jz (Example on p. 336)
Note: Both methods use a “rogue” byte (0xE9 or 0xE8)
Impossible disassembly




Using a single byte in two instructions
Disassembler limited to picking one interpretation, but
processor can use both
Inward jump (Figure 15-4)
More complex case (Figure 15-5)
4. Obscuring Flow Control
Function pointers


Locations resolved at run-time
Hard to statically reverse engineer
Return pointer abuse

Modify return value on stack at run-time (return-oriented
programming)
• call $+5 on Example on p. 342
Misusing structured exception handlers



SEH allows program to handle error conditions intelligently
Uses a stack to manage (FS segment register)
Example on p. 346: Push pointer to exception routine onto
she stack, then trigger exception (divide-by-zero). Routine is
not disassembled
5. Thwarting Stack-Frame Analysis
Stack-frame analysis dependent upon compiler used



Calling conventions vary
Custom management also possible such as management
using esp directly
Listing 15-1 does not use ebp, breaking IDA Pro analysis
• cmp instruction is more or less predictable, but
IDA Pro traces incorrect branch
• Misses “add esp, 104h” and shows esp getting
into an incorrect range (at -F8)
In-class exercise
Lab 15-01


In IDA Pro, what anti-disassembly technique is used and
how many times is it used? Undo the anti-disassembly in
IDA Pro
What order is the input checked?
In-class exercise
Lab 15-02






Explain the false conditional at 0x0040115A. Patch it.
(cfg/idagui.cfg , ENABLE_PATCH_SUBMENU NO)
Explain the false conditional at 0x004011D0. Patch it.
Explain the technique being used at 0x00401215. Patch it.
Explain the technique being used at 0x00401269. Patch it.
Explain the technique being used at 0x004012E6. Which two
methods does it combine? Patch it to reveal Listing 15-7L.
Step through analysis of the malicious code
• What do sub_40130F and sub_401386 do?
•
Show how the first downloaded file is used to
generate the second downloaded file
•
Show how the second downloaded file is used
Chapter 16: Anti-Debugging
Anti-Debugging
Anti-analysis technique for malware to recognize when it is
under the control of a debugger


Slow down analysis as much as possible to increase window
of vulnerability
Hundreds of techniques
Anti-Debugging
1. Windows Debugger Detection
2. Identifying Debugger Behavior
3. Interfering with Debugger Functionality
4. Debugger Vulnerabilities
1. Windows Debugger Detection
Using the Windows API




IsDebuggerPresent() returns 0 if no debugger attached by
searching the Process Environment Block for field
IsDebugged
CheckRemoteDebuggerPresent() allows one to check the
IsDebugged flag on other processes
NTQueryInformationProcess using value ProcessDebugPort
OutputDebugString (Listing 16-1)
1. Windows Debugger Detection
Manual checks



Bypass Windows API to check memory locations directly
Preferred by malware since calls can be hooked by anti-virus
BeingDebugged flag
• Loading of PEB structure address fs:[30h] (Listing 162)
•

Followed by access of BeingDebugged flag at offset
0x2 (Table 16-1)
Debugger heap check
• Get address of first ProcessHeap by loading value at
0x18 into PEB structure, then access flag field at
0x10 (XP) or 0x44 (Win7) (Listing 16-3)
1. Windows Debugger Detection
Manual checks

NtGlobalFlag check
• Heap management different for debugged programs
•




Specified at 0x68 offset in PEB. Set to 0x70 if
debugged (Listing 16-4)
Registry values used by debuggers (HKLM\....\AeDebug)
Window names (e.g. OLLYDBG)
File system
Debugging services, API hooks (OllyDbg detour of OpenProcess),
well-known fixed values in memory (e.g. OllyDbg stores some
strings at 0x004B064B)
2. Identifying Debugger Behavior
INT scanning


INT 3 inserted by debugger to temporarily replace an
instruction so that debug exception handler can run when
software breakpoints are hit (Opcode 0xCC)
Search for 0xCC in code (Listing 16-6)
Performing code checksums

Malware performs checksum on its code pages and exits if
tampering detected
Timing checks



Malware take timestamps and exits if there is a lag
Especially effective when taken before and after an
exception
Implemented via rdtsc instruction (Listing 16-7),
QueryPerformanceCounter, or GetTickCount (Listing 16-8)
2. Identifying Debugger Behavior
Debugger artifacts

INT 1 overwrites 6 bytes below current ESP with return
values for IP, CS, and Flags
PUSH AX
POP AX
DEC SP
DEC SP
POP BX
CMP AX,BX
JNE CODE_IS_TRACED

Force INT 1/INT 3 tracing to disable essential
• Use a canary similar to StackGuard
• Hide critical value (e.g. decryption key) on
stack directly without modifying stack pointer
• Debugger overwrites value if it runs
2. Identifying Debugger Behavior
Debugger artifacts

Check registers/flags saved by debugger such as DR0-DR7
• Set handler and force exception (divide by zero)
• Debug registers saved on stack on context
switch
• Read and write values directly

Execute exception with Trap flag set
• No debugger = SEH occurs
• Debugger attached = SEH will not occur
3. Interfering with Debugger
Functionality
Using TLS (thread local storage) callbacks



Debuggers pause at program entry point defined by the PE
header
TLS implemented in an executable contains a .tls section
that is initialized before program entry point
Most debuggers can be configured to pause before TLS
callback code if a .tls section is present in malware
Using exceptions


Debuggers can be configured to either trap exceptions or
pass them through automatically to application
Malware probes to ensure exceptions are passed through
quickly
3. Interfering with Debugger
Functionality
Inserting interrupts





Inserting a long loop of INT 3 instructions
Inserting 0xCD03 (STATUS_BREAKPOINT) to generate an INT 3.
Inserting INT 2D (kernel debugger breakpoint)
Running line of code
• Hook INT 1
•
Decrypt next instruction, encrypt previous one
•
Only one instruction decrypted in memory at a time
•
Hard to analyze
Side-effects of having a debugger attached result in malware
changing how it executes
• Have malicious code be a part of an SEH handler
•
INT 3 without debugger returns exception directly
back into program to handle
•
INT 3 with debugger goes elsewhere (Listing 16-9)
3. Interfering with Debugger
Functionality
Modifying expected interrupt behavior


Continually overwrite Interrupt Vector of INT 1/3 instructions
to point to garbage code to crash debugger
Turning off keyboard interrupts
IN AL, 20h
OR AL, 02
OUT AL, 20
<virus code>
IN AL, 20
AND AL, NOT 2
OUT AL,20
4. Debugger Vulnerabilities
PE header vulnerabilities

OllyDbg follows specifications of PE headers more strictly
than Windows. Crashes on malformed headers that will run
without debugger
Code vulnerabilities



OutputDebugString vulnerable to format string vulnerability in
OllyDbg v. 1.1. Pass malformed string to crash debugger
Exploit instructions that OllyDbg handles differently than
CPU to crash debugger
Exploit exceptions that OllyDbg handles differently than CPU
to crash debugger (memory handling)
In-class exercise
Lab 16-01



Load the binary in IDA Pro. Bring up Figure 16-1L and explain what the
three jz checks are doing
Bring up sub_401000 and Listing 16-1L. What does this code do?
Load the binary in OllyDbg
• Set a breakpoint at 0x00403554. What is the value of eax?
Step over several instructions. What happens?
•
Bring up Figure 16-2L or Figure 16-3L (via the Command
Line plug-in or Phant0m plug-in) to reset the flag
•
Re-run the first OllyDbg step. What happens?
•
Explain the second anti-debugging check at 0x00403573
and how to bypass it
•
Explain the third anti-debugging check at 0x00403594 and
how to bypass it
•
Set the argument to “-in” and single-step to reach
0x004035D5.
Chapter 17: Anti-Virtual Machine Techniques
Anti-Virtual Machine Techniques
Virtual machines initially used only by malware analysts


Malware benefited from detecting VM (especially VMware)
and shutting down to escape analysis
Method is increasingly uncommon as a result of the
prevalent use of VMs by normal users
• Rollback recovery easy
• Portability
Anti-Virtual Machine Techniques
1. VMware Artifacts
2. Vulnerable Instructions
3. Tweaking Settings
4. Escaping the Virtual Machine
1. VMware Artifacts
Filesystem (e.g. C:\Program Files\VMware\VMware Tools)
Registry

p. 371
Process listing

Figure 17-1
Memory (invariant strings in VMware virtual machine)
Networking

MAC addresses assigned for use by IEEE for VMware NICs
begin with 00:0C:29
1. VMware Artifacts
Example code to check

Listing 17-1
Circumventing checks



Patch condition on branch to bypass in debugger
Use hex editor to modify VMware string
Uninstall VMware tool being checked
2. Vulnerable Instructions
Instructions exposing host machine




For performance, virtual machine monitors allow certain
instructions to execute directly
Instructions such as sidt, sgdt, sldt access hardware registers
directly without generating an interrupt
Can expose inconsistency within guest VM
Must NOP out the check
Querying the I/O communication port (Phatbot, Storm)




VMware virtualizes I/O ports
Port can be queried to detect presence of VMware
Obtaining VMware version via IO port (Listing 17-3)
Must NOP out the check
Common Anti-VM instructions


sidt, sgdt, sldt, smsw, str, in, cpuid
20 instructions designated by VMware as “not virtualizable”
3. Tweaking Settings
VMware provides options to hide itself from malware



Listing 17-5
Protects against all checks implemented by ScoopyNG, a
free VMware detection tool
Last-resort since performance will crater if used
4. Escaping the Virtual Machine
Exploiting VMware bugs to crash host or run code in it



Shared folder feature
Drag-and-drop functionality in VMware Tools
VM display function
In-class exercise
Lab 17-01


In IDA Pro, show and explain the three anti-VM checks being performed
Run the code
• Break before the first anti-VM check. Does this check
succeed? If so, NOP or skip the check and run again.
•
Break before the second anti-VM check. Does this check
succeed? If so, NOP or skip the check and run again.
•
Break before the third anti-VM check by setting a
breakpoint at 0x004012CB and stepping into sub_401100.
Does this check succeed? If so, NOP or skip the check
and run again.
•
Reach the beginning of malware code at 0x004012DF and
generate Listing 17-5L
Chapter 18: Packers and Unpacking
Packers
Used to shrink malware and thwart detection by antivirus



Thwarts static analysis since malware must be unpacked
before it can be analyzed
Original executable transformed to a new self-extracting one
via compression, encryption, or obfuscation making it harder
to recognize and reverse-engineer
Typically employs anti-disassembly, anti-debugging, and
anti-VM techniques to prevent unpacking on an analyst
machine
Packers and Unpacking
1. Packer Anatomy
2. Identifying Packed Programs
3. Unpacking Options
4. Tips and Tricks for Common Packers
5. Packed DLLs
1. Packer Anatomy
Unpacking Stub



Small piece of code loaded by the operating system just as a
normal program
Unpacking stub then loads original program
Step #1: Unpacking original executable into memory
• Loader reads PE header and copies sections into
allocated memory normally
•

Step #2: Resolve imports of original executable
• Loader reads PE header to find library functions to
import and their addresses
•

Unpacking code does the same for packed code
Unless packed code's imports included in unpacking
code's import section, unpacker must resolve imports
manually using LoadLibrary and GetProcAddress
Step #3: Transfers execution to original execution point
• Tail jump to entry point
1. Packer Anatomy
Unpacking samples

Save flags and all registers (PUSHFD, PUSHAD), call unpacking routine
0040AC44 FFFF
INVALID
0040AC4C 9C
PUSHFD
0040AC4D 60
PUSHAD
0040AC4E E802000000
CALL
0040AC55
**If you step over this CALL using F10, the program will run.
Thus, reload the program and step into this CALL using F8 next time.

Within unpacking routine
aaaaaaaa
...
wwwwwwww
xxxxxxxx JNZ zzzzzzzz
yyyyyyyy JMP aaaaaaaa
zzzzzzzz New Instructions

<-- Loop back to aaaaaaaa
Jump to OEP (original execution point)
• Tail jump (POPAD/POPFD restoration, PUSH followed by a
RET!)
0040CA83
0040CA89
0040CA8E
0040CA8F
0040CA90
0040CA91
0040CA96
•
8BBD2E744000
E85E040000
61
9D
50
68CC104000
C20400
MOV
CALL
POPAD
POPFD
PUSH
PUSH
RET
EDI,[EBP+0040742E]
0040CEEC
EAX
004010CC
0004
Set breakpoint at 0x0040CA96 and dump memory image
1. Packer Anatomy
Unpacking samples

Alternate jump to OEP
015F:01017554
015F:01017558
015F:01017559
015F:01017563
of EAX!
015F:01017564

MOV
POPAD
JNZ
PUSH
RET
[ESP+1C],EAX
01017563
EAX *** Take note of the value
(JUMP
)
*** Stop here!!!
Can also use a jmp (Listing 18-1)
• Note: empty bytes after JMP and huge offset
•
IDA Pro can identify JMP goes to garbage and flags it red
(Figure 18-5)
2. Identifying Packed Programs
Simple indicators





Program with few imports and imports are LoadLibrary and
GetProcAddress
IDA Pro recognizes a small amount of code
Presence of UPX0 section (a specific packer)
Abnormal section sizes
Used by tools such as PEiD to determine if code is packed
Entropy calculation

Disorder in a program much larger in encrypted and
compressed payloads
3. Unpacking Options
Automated static unpacking



Decompress and decrypt executable to restore original code
Specific to a packer (i.e. you must know which packer was
used)
PE Explorer
• Supports NSPack, UPack, and UPX
Automated dynamic unpacking




Program is run and unpacker stub is allowed to unpack
original executable
Once tail jump is reached, memory is dumped and original
program written to disk
Fails if the end of unpacking stub is not identified properly
Not many publicly available tools for this
3. Unpacking Options
Manual dynamic unpacking



Option #1: Discover packing algorithm and write a program
to run it in reverse
Option #2: Run packed program so unpacking stub does the
work
• Break and dump the process out of memory
(Listing 18-2 and 18-3)
• Manually fix up PE header so program is
complete
Helpful tools
• OllyDump plug-in for OllyDbg (performs OEP
identification, import table reconstruction, entry
point patching)
• ImpRec (Import Reconstructor) when OllyDump
fails to build a proper import table
3. Unpacking Options
Manual dynamic unpacking

Finding OEP via stack trace
• Upon entry into unpacking stub, registers often pushed
•
Set a breakpoint for esp accessing those stack locations
again
•
Indicates unpacking code is finished and a jump to original
entry point forthcoming

Finding OEP via iteration
• Break at the end of each loop and iterate until tail jump
identified

Manual import table patching
• Two tables: table of function names, table of addresses
•
Listing 18-4 when import table broken
•
Cross-reference between OllyDbg and IDA Pro to patch
import table with function name
4. Tips and Tricks for Common
Packers
UPX (Ultimate Packer for eXecutables)



Open-source
Designed for compression not for security
OllyDump finds easily using heuristics previously described
PECompact

Similar to UPX, uses a tail jump of jmp *eax
ASPack

Uses self-modifying code to thwart analysis
4. Tips and Tricks for Common
Packers
Petite


Uses single-step exceptions to break into debugger
Must pass single-step exceptions back to Petite or employ
hardware breakpoints to find OEP
WinUpack

Uses PUSH followed by RET for tail jump

Placed in the middle of stub (Listing 18-5)
Themida




Secure packer employing anti-debugging, anti-analysis, and
anti-VM techniques
Contains a kernel component making it difficult to follow
Runs code continuously
Use ProcDump to dump memory without attaching debugger
5. Packed DLLs
Similar to executables



Unpacking stub contained in DllMain
DllMain unpacks original DLL
Some debuggers execute DllMain before breaking
•
Can set IMAGE_FILE_HEADER values to cause DLL to be
interpreted as executable
In-class exercise
Lab 18-1





Load executable in IDA Pro to identify packed code
Run PEiD on binary and find section UPX2. Perform a “deep
scan”. What does PEiD return?
In OllyDbg, locate the jump to the unpacking stub by finding
the register save instruction
Set a breakpoint at this location and execute unpacking
code. Single-step to the OEP.
Use OllyDump to dump the program into a new executable
and load the new executable in IDA Pro