Dean Carlson and Beth Anne Byrd
CpSc 420




What is reverse engineering?
Brief History
Usefulness
The process
 Bagle Virus example


“[T]he process of analyzing a subject system
to create representations of the system at a
higher level of abstraction” (Chikofsky, 1990).
Going through the software development
cycle backwards



Started as analyzing hardware in an attempt
to gain an advantage.
The first time this was applied to a piece of
malware was in 1987.
Bernt Fix disassembled and neutralized the
Charlie virus.





Analysis of a product
Recreating lost or nonexistent
documentation
Academic use
Curiosity
With Malware
 Contain it
 Remove it
 Prevent it

Diffuse “time bombs”
 Conficker virus
▪ The Y2K of today
Set up a controlled, isolated laboratory
Perform behavioral analysis to examine the
specimen’s interactions with its environment.
3. Perform static code analysis to further
understand the specimen’s inner-workings.
4. Perform dynamic code analysis to understand
the more difficult aspects of the code.
5. If necessary, unpack the specimen.
6. Repeat steps 2, 3, and 4 (order may vary) until
sufficient analysis objectives are met.
7. Document findings and clean-up the laboratory
for future analysis.
1.
2.

HOST:
 Windows XP in Virtual Machine
 DataRescue IDA Pro
 Microsoft Visual C++
▪ Dumpbin
 UltraEdit

SERVER





Solaris 9 (SPARC)
Snoop
BIND (DNS)
GCC
GDB

The Email and DNS programs on the server
were setup to log all of their activity and
network traffic in order to see the virus
interact with the server.

Open in IDA Pro
 Breaks it down into assembly and hex

Open in dumpbin to determine type
 PE (Portable Executable)

Walk through the virus step by step with a
debugger and look at register values.
Especially EAX, EIP, ZF bit of EFLAGS
 EAX = return values from functions
 ZF = flag used for comparisons and decisions
 EIP = useful for thread usage

Use IDA to chart subroutines

Use IDA to identify function parameters and
variables
 arg_8 can be accessed by adding “10h” to the EBP
Register

Multiple Thread
 Extended Instruction Pointer (EIP) doesn’t follow
new threads unless specified

The Bagle virus was not packed
 Compressed or encrypted

It also was not polymorphic
 Changing the assembly, usually by inserting
“noop” thus changing the virus signature but not
changing the effectiveness

The Bagle virus has many removal tools
Reverse engineering malware started in 1987
It is good to contain, remove, and prevent
malware
 7 steps
1. Set up lab
2. Behavioral analysis
3. Static code analysis
4. Dynamic code analysis
5. Unpack
6. Repeat steps 2, 3, and 4
7. Document and clean-up

