Anti-Unpacking techniques
Peter Ferrie
Senior Anti-virus Researcher
1 May, 2008
1
Classes of unpackers
Memory dumper
Debugger
Emulator
Code-buffer
W-X interceptor
Peter Ferrie, Microsoft Corporation
2
Anti-dumping
SizeOfImage

Field in PEB

When altered, prevents debugger from attaching

Causes incorrect number of pages to be dumped
Erasing the header

Removes section information

Not obvious how many pages should be dumped

Used by Yoda’s Crypter
Nanomites

Introduced in Armadillo

Replace branches with int 3 and encoded pointers
Peter Ferrie, Microsoft Corporation
3
Anti-dumping (cont.)
Stolen bytes

Introduced in ASProtect

Copy bytes elsewhere and execute from there

Optionally destroy original bytes
Guard pages

Allows on-demand decryption or decompression

Used by Armadillo (decryption)

Used by Shrinker (decompression)
Imports

Table is often removed, encrypted or compressed

Sometimes erased after first use

Functions can be redirected to another location
Peter Ferrie, Microsoft Corporation
4
Anti-dumping (cont.)
Virtual machines

Uses p-code instead of native code

P-code can be polymorphic (Themida)

P-code can contain anti-debugging (HyperUnpackMe2)

Interpreter can be obfuscated (Virtual CPU)
Peter Ferrie, Microsoft Corporation
5
Anti-debugging
NtGlobalFlag

Field in PEB

Debugger typically causes certain flags to be set

Very common method

Very commonly incorrect (cmp instead of test)

Very unreliable (several ways to set/clear flags)

Used by ExeCryptor
Heap flags

Pointer is field in PEB

Two sets of flags

Debugger typically causes certain flags to be set

Also very unreliable (flags controllable like NtGlobalFlag)
Peter Ferrie, Microsoft Corporation
6
Anti-debugging (cont.)
Heap

Tail markers appear if certain flag is set

Debugger typically causes flag to be set

Marker is constant and easy to find

Used by Themida
IsDebuggerPresent()

Introduced in Windows 95

Reads field in PEB

One of the most common methods

Returns true if a debugger is present
Peter Ferrie, Microsoft Corporation
7
Anti-debugging (cont.)
CheckRemoteDebuggerPresent()

Introduced in Windows XP SP1

Wrapper around NtQueryInformationProcess (ProcessDebugPort)

Returns true if a debugger is present
NtQueryInformationProcess()

Can be called with ProcessDebugPort class

Returns EPROCESS->DebugPort value

Non-zero if a debugger is present

Used by MSLRH
Peter Ferrie, Microsoft Corporation
8
Anti-debugging (cont.)
Debug objects

Another use for NtQueryInformationProcess()

Introduced in Windows XP

Can be called with ProcessDebugObjectHandle class

Returns non-zero if a debugger is present

Used by HyperUnpackMe2
More debug objects

Yet another use for NtQueryInformationProcess()

Also introduced in Windows XP

Can be called with ProcessDebugFlags class

Returns inverse of EPROCESS->NoDebugInherit bit

Returns zero if a debugger is present

Used by HyperUnpackMe2
Peter Ferrie, Microsoft Corporation
9
Anti-debugging (cont.)
Kernel debugger

Final use for NtQueryInformationProcess()

Can be called with SystemKernelDebuggerInformation class

Not supported by any version of Windows(?)

Returns non-zero in AH register if a debugger is present

Used by SafeDisc
NtQueryObject()

Supported since Windows NT

Extended in Windows XP to include debug object

Request all objects

Search for "DebugObject"

Check associated handle count

Non-zero if a debugger is present
Peter Ferrie, Microsoft Corporation
10
Anti-debugging (cont.)
Thread hiding

NtSetInformationThread() method

Introduced in Windows 2000

Can be called with HideThreadFromDebugger class

Debugger receives no further events for that thread

Used by HyperUnpackMe2
OpenProcess()

Csrss.exe process can be opened

If debug privilege is obtained first

Even though it's a system process

Process can be opened with full access rights

Allows thread injection and system-wide DoS
Peter Ferrie, Microsoft Corporation
11
Anti-debugging (cont.)
CloseHandle()

Different behaviour if debugger is present

Kernel raises EXCEPTION_INVALID_HANDLE

For invalid handle if debugger is present

Exception can be intercepted by debugged process
OutputDebugString()

Different behaviour if debugger is present

Kernel clears thread error code

If debugger is present
ReadFile()

Indirect breakpoint removal

Read data from file over location of breakpoint
Peter Ferrie, Microsoft Corporation
12
Anti-debugging (cont.)
WriteProcessMemory()

Alternative breakpoint removal

Write data from memory over location of breakpoint

Used by NsAnti
UnhandledExceptionFilter()

Called as last resort if no other handlers succeed

Not called if a debugger is present

Exception is passed to debugger instead
BlockInput()

Blocks mouse and keyboard events

Used by Yoda's Protector
Peter Ferrie, Microsoft Corporation
13
Anti-debugging (cont.)
SuspendThread()

Can be used on remote processes

Such as debuggers

Used by Yoda's Protector
Guard pages

Certain debuggers might not support them properly

Access should cause EXCEPTION_GUARD_PAGE

OllyDbg uses guard pages for memory breakpoints

Consumes breakpoint and allows execution

Used by PC Guard
Peter Ferrie, Microsoft Corporation
14
Anti-debugging (cont.)
Alternative desktop

Sessions support multiple desktops

A new desktop can be created and made active

No obvious way to switch back

Used by HyperUnpackMe2
Prefetch queue

Queue is not flushed when rep instruction is used

Allows for some old self-overwriting tricks

Two known variations of the trick

One is used by Invius

The other is used by Obsidium
Peter Ferrie, Microsoft Corporation
15
Anti-debugging (cont.)
Hardware breakpoints

Exception handler receives context structure

Structure contains all CPU debug registers

Non-zero values suggest presence of debugger

Handler can also remove breakpoints and resume
Instruction counting

Handler can insert own hardware breakpoints

Then count number of exceptions that occur

Debugger might skip some breakpoints
Execution timing

Single-stepping through code take a long time

Delay can be detected

Several time sources available
Peter Ferrie, Microsoft Corporation
16
Anti-debugging (cont.)
Exceptions

Obfuscated method to move EIP around

Infinite variety of possible implementations
Header entrypoint

Some debuggers don't use VirtualProtectEx()

So header remains write-protected

Cannot set initial breakpoint

Code runs freely
Parent process

Should be shell process if launched by clicking

Usually Explorer.exe, unless debugger is involved

Used by Yoda's Protector
Peter Ferrie, Microsoft Corporation
17
Anti-debugging (cont.)
Self-execution

Simple method to escape from debugger

Synchronisation object prevents infinite executions

Used by MSLRH
Process name

Process names can be enumerated

Names can be compared to "watch list"

List commonly includes anti-malware software
Peter Ferrie, Microsoft Corporation
18
Anti-debugging (cont.)
Threads

Multiple threads are hard to trace

Some anti-malware emulators don't support them

One thread is often devoted to debugger detection

Can also be used to detect code changes

Used by PE-Crypt32
Self-debugging

Effective anti-debugging method

Only one debugger can attach to a process

Defeated by zeroing EPROCESS->DebugPort

Or call DebugActiveProcessStop() on Windows XP

Used by Armadillo
Peter Ferrie, Microsoft Corporation
19
Anti-debugging (cont.)
Disassembly

Can be used to steal instructions

Safer method to detect breakpoints

Can also be used to detect detours

Problematic if routine is hot-patched

Since that will be a detour
TLS callback

Relatively little-known method to execute code

Code executes before main entrypoint

Also executes before inserted breakpoint

Debuggers typically don't handle it

Used by ExeCryptor
Peter Ferrie, Microsoft Corporation
20
Anti-debugging (cont.)
Device names

A way to communicate with drivers

Also a way to detect presence of drivers

List of names often target SoftICE, RegMon and FileMon
SoftICE-specific

Driver names can be enumerated

Version information can be queried

Product and copyright strings can be compared to "watch list"

Harder to change than renaming the driver

List of strings are usually SoftICE-related
Peter Ferrie, Microsoft Corporation
21
Anti-debugging (cont.)
SoftICE-specific (cont.)

Interrupt 1 is normally not invokable from ring 3

SoftICE hooks interrupt 1 and allows ring 3 access

So wrong exception when SoftICE is running

Used by SafeDisc
OllyDbg-specific

Cannot handle unusual NumberOfRvaAndSizes value

Some unchecked fields allow memory allocation DoS

Initial ESI register value is -1 on Windows XP

Looks like a detection method

It's just a coincidence
Peter Ferrie, Microsoft Corporation
22
Anti-debugging (cont.)
OllyDbg-specific (cont.)

Passes user-defined data directly to _vsprintf()

Leads to DoS condition

Debugger window can be found by calling FindWindow("OLLYDBG")
HideDebugger-specific

Plug-in for OllyDbg

Detectable by far jump at OpenProcess()+6
ImmunityDebugger-specific

Based on OllyDbg

Shares many of the same vulnerabilities
WinDbg-specific

Debugger window can be found by calling FindWindow("WinDbgFrameClass")
Peter Ferrie, Microsoft Corporation
23
Anti-emulating
Interrupt 3

Size of instruction is not checked

Using "cd 03" instead of "cc" instruction -> EIP points to "03"

Prefixed "cc" instruction works, too

Except perhaps in some emulators

Used by TryGames
Time-locks

Effective anti-emulation technique

Causes all iterations to be consumed before anything happens

Some loops can be skipped arbitrarily

Some loops cannot (results might be used for decryption keys)

Used by Tibs
Peter Ferrie, Microsoft Corporation
24
Anti-emulating (cont.)
Invalid API parameters

Some emulators do not support invalid parameters in APIs

An expected error code can be used for decryption keys

No error code means no decryption

Used by Tibs
GetProcAddress

There are too many possible APIs for emulators to handle

So some emulators always return a value for GetProcAddress()

Sometimes GetProcAddress() should return 0

A non-zero value reveals the emulator

Used by NsAnti
Peter Ferrie, Microsoft Corporation
25
Anti-emulating (cont.)
"Modern" CPU instructions

Some emulators do not handle modern CPU instructions

Some malware use FPU and MMX as do-nothing instructions

Emulators might give up early as a result

Used by NsAnti
Undocumented instructions

Some emulators do not handle undocumented CPU instructions

Some instructions more undocumented than others

Example: multi-byte NOP

Example: fully allocated PREFETCH
Peter Ferrie, Microsoft Corporation
26
Anti-emulating (cont.)
Selector verification

Some emulators pretend to be Windows NT-based

While using Windows 9x-based selector values

A CS selector value larger than 0xff reveals this

Used by MSLRH
Memory layout

Certain in-memory structures are at predictable memory locations

Example: RTL_USER_PROCESS_PARAMETERS at 0x20000

If they are missing, the program might not run

Used by TryGames
Peter Ferrie, Microsoft Corporation
27
Anti-emulating (cont.)
File-format tricks

Non-aligned SizeOfImage

Windows will silently round up the value

Overlapping structures

Tools such as IDA have a problem with this

Non-standard NumberOfRvaAndSizes

SoftICE and OllyDbg have a problem with this

Non-aligned SizeOfRawData

Windows will silently round up the value

Non-aligned PointerToRawData

Windows will silently round down the value

No section table

Allowed when SectionAlignment is less than 4kb

Header becomes writable and executable
Peter Ferrie, Microsoft Corporation
28
Anti-intercepting
Write->Exec

Some interceptors watch for write-then-exec

Executing dummy just-written instruction can fool them

Used by ASPack, but probably for multi-processor support
Write^Exec

Some interceptors prevent write and exec on the same page

Change can be detected indirectly

Kernel functions return error when writing to read-only pages

VirtualQuery() and VirtualProtect() return old page attributes
Peter Ferrie, Microsoft Corporation
29
Thank you
More techniques in the paper
http://pferrie.tripod.com
Questions?
Peter Ferrie, Microsoft Corporation
30