A generic buffer overflow takes place when a buffer that has been assigned a specific storage
space contains more data than it can handle. Programs and applications are vulnerable because
boundary checks are not completely done or they are skipped completely in most cases.
Programming languages such as C are vulnerable. As some functions, such as strcat(), strcpy(),
sprint(), vsprintf(), bcopy(), gets(), and scanf(), in C language do not check for buffer size, they
can be exploited.
In this chapter, we will learn about buffer overflows, types of buffer overflows, and buffer
overflow steps. We will also learn about smashing the stack, mutating a buffer overflow exploit,
and identifying buffer overflow detection tools and buffer overflow countermeasures tools.
17.1 Understand buffer overflows (BoF)
Exam Focus: Understand buffer overflows (BoF). Objective includes:






Understand buffer overflows (BoF) and examples of buffer overflows.
Understand stack-based buffer overflow.
Know heap-based buffer overflow.
Understand stack operations.
Identify buffer overflow steps.
Analyze attacking a real program.
Buffer overflow
Buffer overflow is a condition in which an application receives more data than it is configured to
accept. It helps attackers not only to execute a malicious code on the target system, but also to
install backdoors on the target system for further attacks. Buffer overflow attacks are almost
always due to sloppy programming or poor memory management by the application developers.
A generic buffer overflow takes place when a buffer that has been assigned a specific storage
space contains more data than it can handle. Programs and applications are vulnerable because
boundary checks are not completely done or they are skipped completely in most cases.
Programming languages such as C are particularly vulnerable, since functions, such as strcat(),
strcpy(), sprint(), vsprintf(), bcopy(), gets(), and scanf(),do not check for buffer size, and can
therefore be exploited.
Types of buffer overflow
The primary types of buffer overflow are:




Stack-based buffer overflow
Format string overflow
Heap-based buffer overflow
Integer overflow
Stack-based buffer overflow
A stack is the Last in First Out (LIFO) abstract data type and data structure. It can have any
abstract data type as an element. It is like a buffer that holds all the information that is needed by
the function.
A stack-based buffer overflow attack uses a memory object known as a stack. The hacker
develops the code which reserves a specific amount of space for the stack. If the input of the user
is longer than the amount of space reserved for it within the stack, the stack will overflow. An
attacker injects malicious code on the stack and overflows the stack to overwrite the return
pointer to switch the flow of control to the malicious code.
Stack buffer overflow attacks
Buffer overflows, also known as 'smash the stack' are a favorite exploit for hackers. Both
commercial and in-house applications are susceptible to buffer overflow attacks. Here are
common ways for a stack buffer overflow attack to occur:



Overwriting a local variable that is near the buffer in memory on the stack to change the
behavior of the program which may benefit the attacker.
Overwriting the return address in a stack frame. Once the function returns, execution will
resume at the return address as specified by the attacker, usually a user input filled buffer.
Overwriting a function pointer, or exception handler, which is subsequently executed.
It is very important for developers, and QA professionals to understand how buffer overflows
work so they can plan for vulnerability testing on home-grown applications prior to deployment.
Format string overflow
A format string overflow attack is caused by the use of unfiltered user input. It can be used to
crash a program or to execute harmful code. It modifies the flow of an application by using string
formatting library features to access other memory space. Vulnerabilities occur due to the direct
use of user-supplied data as formatting string input for certain C/C++ functions. A format string
attack is related to the buffer overflow and integer overflow attacks.
Heap-based buffer overflow attack
Heap is an area of memory used by an application. Memory on the heap is dynamically allocated
by the application at run-time and typically contains program data. Exploitation is performed by
corrupting this data in specific ways to cause the application to overwrite internal structures such
as linked list pointers.
In a heap-based buffer overflow attack, an attacker overflows a buffer that is placed on the lower
part of heap, overwriting other dynamic variables, which can have unexpected and unwanted
effects.
If an application copies data without first checking whether it fits into the target destination, the
attacker could supply the application with a piece of data that is large, overwriting heap
management information. In most environments, this may allow the attacker to control over the
execution of the program.
Integer overflow
Integer overflow, or integer wrapping, is a potential problem in a program. It works upon the fact
that the value that can be held in a numeric datatype is limited by the data type's size in bytes.
When a value is placed into a data type that is too small to hold it, the high-order bits are
dropped, and only the low-order bits are stored.
How hackers exploit buffer overflow
In a nutshell, here's how hackers exploit buffer overflows
1. Determine the presence and location of buffer overflow vulnerability.
2. Write more data into the buffer than it can handle and overwrite the return address of a
function.
3. Change the execution flow to the hacker code.
Example:
An attacker assumes that a string function is exploited and sends a long string as the input. This
string causes a segmentation error by overflowing the buffer. The return pointer of the function is
overwritten, and the attacker becomes successful in changing the flow of the execution. A user
should know the exact address and size of the stack and make the return pointer point to his code
for execution if he has to insert code in the input.
Other ways to exploit code



Shellcode:is a small piece of code that is used as payload when software vulnerability is
exploited. As buffers overflow easily if the conditions match, they are considered as soft
targets for attackers. Buffer overflow shellcodes are written in assembly language. They
exploit vulnerabilities in stack and heap memory management.
No operation (NOP) sleds: There are many CPUs that have a No Operation (NOP)
instruction. Essentially, they are set of instructions that effectively do nothing at all. A
NOP-sled is the oldest and most widely known technique for successfully exploiting a
stack buffer overflow. It solves the problem of finding the exact address of the buffer by
effectively increasing the size of the target area. To do this, much larger sections of the
stack are corrupted with the HYPERLINK "http://en.wikipedia.org/wiki/No-op" \o "Noop" no-op machine instruction.
Because of the popularity of this technique, many vendors of HYPERLINK
"http://en.wikipedia.org/wiki/Intrusion_prevention_system" \o "Intrusion prevention
system" intrusion prevention systems will search for this pattern of no-op machine
instructions in an attempt to detect shellcode in use. In addition, because exploits using
this technique still must rely on some amount of luck that they will guess offsets on the
stack that are within the NOP-sled region, they have a higher risk of being detected. An
incorrect guess will usually result in the target program crashing and alert the application
or system administration team to a possible NOP attack.
Canaries: Canaries or canary words are known values that are placed between a buffer
and control data on the stack to monitor buffer overflows. When the buffer overflows, the
first data to be corrupted will be the canary, and a failed verification of the canary data is
therefore an alert of an overflow, which can then be handled, for example, by invalidating
the corrupted data.
17.2 Reasons for buffer overflow attacks, and skills required to program buffer overflow exploits
Exam Focus: Reasons for buffer overflow attacks, and skills required to program buffer overflow
exploits. Objective includes:



Examine smashing the stack.
Understand how to mutate a buffer overflow exploit.
Learn how to identify buffer overflows.
Smash the stack
Recall that a 'smash the stack' is a buffer overflow situation. On many C implementations, it is
possible to corrupt the execution stack by writing past the end of an array declared auto in a
routine. Code that does this is said to smash the stack, and can cause return from the routine to
jump to a random address. The function will then jump to whatever address is on the stack when
the function has been performed. Buffer overflow permits a user to change the return address of
a function. This almost always results in corruption of adjacent data on the stack, and in cases
where the overflow was triggered by mistake, will often cause the program to crash or operate
incorrectly.
If the affected program is running with special privileges, or accepts data from untrusted network
hosts such as a "http://en.wikipedia.org/wiki/Webserver" \o "Webserver" webserver, then the bug
is a potential security vulnerability. If the stack buffer is filled with data supplied from an
untrusted user then that user can corrupt the stack in such a way as to inject executable code into
the running program and take control of the process. Once the stack is smashed, an attacker can
create a backdoor using tools like inetd, Trivial FTP (TFTP), or Netcat in order to make raw and
interactive connections. This is one of the oldest and more reliable methods for
"http://en.wikipedia.org/wiki/Hacker_(computer_security)" \l "Classifications" \o "Hacker
(computer security)" black hat hackers to gain unauthorized access to a computer.
Reasons for buffer overflow attacks
Buffer overflow attacks depend on the lack of boundary testing and a machine that can execute a
code that resides in the data/stack segment. The lack of boundary is common and the program
usually ends with the segmentation fault or bus error. The offender must create the data to be fed
to the application to exploit buffer overflow for gaining access to or escalate privileges. Random
data will produce a segmentation fault or bus error, never a remote shell or the execution of a
command.
C library functions that cause buffer overflow attacks
C programs are critically vulnerable to buffer overflow attacks. C library functions, such as
gets(), strcpy(), strcat(), sprintf(), vsprintf(), etc. are responsible for buffer overflow in most
cases. The scanf() function can also be the reason of a buffer overflow attack.
Identifying buffer overflows
Take the following steps to identify buffer overflows:
1. Run a web server on the local machine and issue requests with long tags-all long tags end
with "$$$$$".
2. Search core dump for "$$$$$" in order to determine the overflow location if the web
server crashes.
3. Use automated tools, such as codeBlocker, eEye Retina, etc., and disassemblers and
debuggers.
4. Construct an exploit by using IDA-Pro.
Detect buffer overflows in a program
There are at least two ways to detect buffer overflows:
1. Source code: In this case, the hacker can verify the presence of boundary checks after
looking for strings declared as local variables in functions or methods. Improper use of
standard functions should be checked, especially those related to strings and input/output.
2. Feed the application with huge amounts of data and check for the abnormal behavior.
SmashGuard
SmashGuard is a hardware-based solution for stack-smashing buffer overflow. At each function
call, SmashGuard keeps a copy of the function Return Address written to the program stack in a
LIFO buffer on the CPU. Just before the function is completed, the Return Address of the
function is compared with the address at the top of the stack. If both addresses match, the
program flow continues without any interruption. However, if both addresses do not match each
other, an error is displayed.
Skills required to program buffer overflow exploits
The following skills are required to program buffer overflow attacks:





Understanding of stack and heap memory processes
Understanding of the working of system calls at the machine code level
Understanding of assembly and machine language
Understanding of C and Perl programming language
Understanding of compiling and debugging tools such as gdb
Testing for format string conditions
Format string vulnerabilities manifest in the Web server, application servers, Web application
using C/C++ based code, and CGI scripts written in C. Input parameters are manipulated by an
attacker to include %x or %n. The attacker checks instances of code (assembly fragments) to
identify the presence of a format string vulnerability. The address of a format type specifier
being pushed on the stack is clearly visible before a call to print is made when the disassembly is
examined using IDA Pro.
17.3 Testing for heap overflow conditions: heap.exe, and understand OllyDbg debugger
Exam Focus: Testing for heap overflow conditions: heap.exe, and understand OllyDbg debugger.
Objective includes:




Test for heap overflow conditions: heap.exe.
Understand steps for testing stack overflow in OllyDbg debugger.
Identify buffer overflow detection tools.
Understand defense against buffer overflows.
Testing for heap overflow conditions
The variants of the heap overflow (heap corruption) vulnerability including those that permit
overwriting function pointers and exploit memory management structures for arbitrary code
execution are identified. Longer input strings than expected are supplied to test for heap
overflows. When debugging a windows program, a heap overflow can appear in several different
forms. The most common one being a pointer exchange taking place after the heap management
routine comes into action.
Testing a stack overflow in OllyDbg debugger
To test a stack overflow in OllyDbg debugger:
1. Attach a debugger to the target application or process, and generate malformed input for
the application.
2. Subject the application to malformed input, and inspect responses in a debugger.
For example, an attacker can take the following steps to overwrite the instruction pointer and
control program execution:
1. Test "sample.exe" for stack overflow.
2. Launch "sample.exe" in a debugger. A large sequence of characters such as "A" can be
provided in the argument field.
3. Open the executable with the supplied arguments and continue execution. EIP contains
the value "41414141". The value represents the hexadecimal "AAAA".
BoF detection tools


BOU (Buffer Overflow Utility): An attacker can use the BOU tool to test Web apps for
buffer overflow conditions. The tool requires to input the "request" file that is to be tested
and how much of the code that should be attacked. The tool takes a request file to be
tested and outputs all of the activity to STDOUT depending on the level of verbosity
specified.
Flawfinder: Flawfinder is a buffer overflow detection tool written in Python by David
Wheeler. It uses a built-in database for C/C++ functions that are vulnerable to buffer
overflow attacks. Flawfinder is used to check vulnerabilities for buffer overflow risks as




well as format string problems, race conditions, potential shell metacharacter, and poor
random number acquisition.
OllyDbg: OllyDbg is a 32-bit assembler level debugger for Microsoft Windows, which is
used to debug a stack overflow attack. It is used to debug multithread applications. It can
be attached to running programs and configurable disassembler. It is used to If you are
running OllyDbg on Windows Vista or above, you will need to ensure that you run it
using the "Run as Administrator" option, available when you right-click on the
executable or shortcut. If you don't run the program with Administrative privileges, you
won't be able to properly perform your debugging tasks.
RATS (Rough Auditing Tool for Security): RATS (Rough Auditing Tools for
Security) is a buffer overflow detection tool that is used to find out potentially dangerous
function calls. It is used to differentiate between heap- and stack-allocated buffers. RATS
can be used in C, C++, Python, Perl, and PHP programming languages.
ITS4: ITS4 is a buffer overflow detection tool developed by Cigital to check C/C++
code. It is a command-line tool that is used to check for potentially dangerous function
calls. ITS4 also generates a report describing potential problems and providing possible
suggestions. It also checks Time-of-Check-Time-of-Use problems.
BufferShield: is used to detect and prevent the exploitation of buffer overflows. It is
used to detect code execution on the stack, default heap, dynamic heap, virtual memory,
and data segments.
Tools that can prevent buffer overflow vulnerabilities:


Data Execution Prevention (DEP): is a security feature included in Microsoft Windows
operating systems, which is intended to prevent an application or service from executing
code from a non-executable memory region. This helps prevent certain exploits that store
code via a buffer overflow; for example, DEP runs in two modes: hardware-enforced
DEP for CPUs that can mark memory pages as non-executable, and software-enforced
DEP with a limited prevention for CPUs that do not have hardware support. Softwareenforced DEP does not protect from execution of code in data pages, but instead from
another type of attack (SEH overwrite).
Enhanced Mitigation Experience Toolkit (EMET): is used so that it becomes more
difficult for an attacker to exploit the vulnerabilities of software and access the system. It
supports mitigation techniques that prevent common attack techniques that primarily
relate to stack overflow and involve use of malware for interacting with operating
systems.
Defense against buffer overflows
The following serve as defense against buffer overflows:




Performing manual auditing of the code
Using compiler techniques
Using safer C library support
Disabling stack execution
Preventing BoF attacks
The following measures are taken to prevent BoF attacks:






Using type safe languages such as Java and ML
Implementing run-time checking
Using address obfuscation
Randomizing location of functions in libc
Performing static source code analysis
Marking stack as non-execute, random stack location
Programming countermeasures
The following are programming countermeasures:













Programs should be designed with security in mind.
Stack execution should be disabled.
The code should be tested and debugged in order to find errors.
The use of dangerous functions, such as gets, strcpy, etc. should be prevented.
Using "safer" compilers such as StackGuard should be considered.
Return addresses should be prevented from being overwritten.
Arguments should be validated and the amount of code that runs with the root privilege
should be reduced.
All sensitive information should be prevented from being overwritten.
Static or dynamic source code analyzers should be used at the source code level in order
to check the code for the overflow problems.
The compiler should be changed at the compiler level. The compiler level bounds
checking or protects addresses from overwriting.
The rules should be changed at the operating system level for which memory pages are
permitted to hold executable data.
Safe libraries should be used.
Tools that can detect buffer overflow vulnerabilities should be used.
17.4 Understand buffer overflow countermeasures tools and buffer overflow pen testing
Exam Focus: Understand buffer overflow countermeasures tools and buffer overflow pen testing.
Objective includes:


Identify buffer overflow countermeasures tools.
Understand buffer overflow pen testing.
Countermeasures against buffer overflow
The countermeasures against buffer overflow are as follows:

Perform manual auditing of the code.




Stack execution should be disabled.
Take the support of the functions which are not the cause of the buffer overflow.
Take compiler support. For example, Java automatically checks if an array index is
within the proper bounds. Use compilers, such as Java instead of C to avoid buffer
overflow attacks.
Use tools, such as StackGuard and Immunix System to avoid buffer overflows.
Countermeasures against IIS buffer overflow attacks
A network administrator should take the following steps to protect a Web server from IIS buffer
overflow attacks:





Conduct frequent scans for server vulnerabilities.
Install the upgrades of Microsoft service packs.
Implement effective firewalls.
Apply URLScan and IISLockdown utilities.
Remove the IPP printing capability.
Buffer overflow pen testing
A buffer overflow takes place when a program tries to store more data in a static buffer than
intended. The additional data overwrites and corrupts adjacent blocks of memory, and can allow
an attacker to take control of the flow of execution and inject arbitrary instructions. Overflow
vulnerabilities are more commonly found in applications developed in the C/C++ language;
newer languages, such as C# provide additional stack protection for the careless developer.
Recent examples of overflows in web applications include mnoGoSearch and Oracle E-Business
Suite.
Buffer overflows can often be located through black-box testing by feeding increasingly larger
values into form inputs, header and cookie fields. In the case of ISAPI applications, a 500 error
message (or time-out) in response to a large input may indicate a segmentation fault at the server
side. The environment should first be fingerprinted to determine if the development language is
prone to overflow attacks as overflows are more common to compiled executable files than
scripted applications. Note that most of the popular web development languages (Java, PHP,
Perl, Python) are interpreted languages in which the interpreter handles all memory allocation.
Format string attacks occur when certain C functions process inputs containing formatting
characters (%). The printf/fprint/sprintf, syslog() and setproctitle() functions are known to
misbehave when dealing with formatting characters. In some cases, format string bugs can lead
to an attacker gaining control over the flow of execution of a program.
Skills of a penetration tester
The following are skills of a penetration tester:


Understand working of a buffer overflow attack.
Understand memory management in various operating environments.


Understand programming languages.
Proficiently run debuggers, disassemblers, and fuzzers.
Steps for buffer overflow penetration testing
The following steps are taken for buffer overflow penetration testing:
1. Search for call to insecure library functions that may result in buffer overflow if not used
properly.
2. Use tools such as RATS and Flawfinder to perform static code analysis.
3. Use disassemblers such as IDA Pro and OllyDbg to reverse engineer the application.
These tools are required to analyze code of compiled software to recognize buffer
overflow condition.
4. Identify the buffer overflow condition by attaching a debugger to the target application,
supplying a large input data, and inspecting responses in a debugger.
5. Repeat the last step with different inputs of variable length.
6. Supply format type specifiers such as %x or %s in the input.
7. Use fuzzing techniques that gives invalid, unexpected, or random data to the application
inputs and observing application behavior.
8. Use fuzzing tools such as Spike and Brute Force Binary Tester (BFB) for automated
fuzzing testing.
Chapter Summary
In this chapter, we learned about buffer overflows, types of buffer overflow, and buffer overflow
steps. This chapter focused on smashing the stack, mutating a buffer overflow exploit, and
identifying buffer overflow detection tools and buffer overflow countermeasures tools.
Glossary
BoF
Buffer Overflows
Buffer overflow
Buffer overflow is a condition in which an application receives more data than it is configured to
accept.
BufferShield
BufferShield is used to detect and prevent the exploitation of buffer overflows.
Flawfinder
Flawfinder is a buffer overflow detection tool written in Python by David Wheeler. It uses a
built-in database for C/C++ functions that are vulnerable to buffer overflow attacks.
Format string overflow attack
A format string overflow attack is caused by the use of an unfiltered user input. It can be used to
crash a program or to execute harmful code.
Heap
Heap is an area of memory used by an application. It is assigned dynamically at the run time
with functions, such as malloc().
ITS4
ITS4 is a buffer overflow detection tool developed by Cigital to check C/C++ code. It is a
command-line tool that is used to check for potentially dangerous function calls.
.
OllyDbg
OllyDbg is a 32-bit assembler level debugger for Microsoft Windows, which is used to debug a
stack overflow attack. It is used to debug multithread applications.
OllyDbg
32-bit assembler level debugger for Microsoft Windows
RATS
RATS (Rough Auditing Tools for Security) is a buffer overflow detection tool that is used to find
out potentially dangerous function calls.
Shellcode
Shellcode is a small piece of code that is used as payload when software vulnerability is
exploited.