Malware Detection based on Application
Behavior Modeling
NWMTD’11
Jun 20–21, 2011
Mrs P.R.Lakshmi Eswari
C-DAC, Hyderabad
Evolution of Malware Attacks
Era
Who is Leading
Initially
Hobbyists
Late 90s
Criminals
Early of this
decade
Terrorists (more dangerous criminals)
Now
Spies
Malware Definition (Wikipedia)
• A software which is designed to infiltrate a computer
system without the owner’s informed consent
• Refers to a variety of forms of hostile, intrusive,
annoying software code
• MALicious softWARE
Threat from the Malware
• A code
– which collects the credit card number or any other
personal info
– Which makes an application do the buffer overflow
and crash
– Loosing the private and sensitive information
– which shows annoying advertisements without your
consent
– Which encrypts the data and asks for money to
decrypt it
Malware Categories
Category
Description
Virus
Attaches itself to a file (preferably binary)
Trojan
Look as if useful program but invites attacker
Worm
Same as virus but got the capability of spreading at its own
Exploit
Malware code which exploits a vulnerability in the app
Root-kit
To hide the actual malware from system information
Spyware
Spies on the user habits and data and sends it
Phishing
A website made to mimic an existing website
Spam
Sending unwanted emails
Bots
Code in command and control network to launch DDOS
Attacks and other malicious operations
A Typical Malware
Exploit Logic
Motivational Logic
Protection Logic
• Spam
• Data theft
• Ransom
• Disrupt the routine
• Packing
• Anti Debugging
• Anti Virtualization
Propagation
Logic
Mails
USBs
Attacks - Classified
• Untargeted attacks
– Attacking websites
– Infecting portable storage devices
– Attacking social networking websites
– Wild malware (worms etc)
• Botnets
• Targeted Attacks
Targeted Attacks
A Typical Attack
Originally a executable
Doc file
Whenever updates windows, also
downloads the malware, sends
the data out etc.
Opens the file,
and executes the malware
Malware
Changes the
windows update
program
Botnet
4. Attacker will also join this channel
(preferably through a program) and
issue commands (for e.g. update)
IRC
Server
3. Join a channel on IRC
Receives the command (update)
1. Exploit / Attack
Victim
2. Download malware (bot)
Botnet
• DDoS (distributed denial of service attacks)
• Collecting lot of bank related data
• Spidering attacks (on websites)
• Spams
• Using victim for other sensitive attack
• Shutdown the computer etc
Motivation and Business
Motivation and Business
14
Vulnerability, Exploit and Race
Vulnerability, Exploit and Race
Malware Detection Techniques
• Black listing
– Anti Virus
– Intrusion Detection System
– Behavior Based Malware Detection
• White listing
– Specification Based Detection
– Anomaly Detection
Commercial Solutions
Desktop security software Behavior based anti malware
(major anti malware
solutions
products)
AVG
Avir
McAfee
Norton
F-Secure
ESET
Bit Defender
Zone Alarm
Trend Micro
Sunbelt
Sana Security Primary Response
Malware Defender
Mamutu
Malware Resist[C-DAC Hyderabad]
Nova Shield
PC Tools - Threat Fire
End System Security Suites
• Centralized configuration on all clients
• Centrally controlled
– Firewall
– Encryption
– Device Control
– Anti Malware
– Security policies
White listing Solutions
•
•
•
•
•
Core Trace Bouncer
Bit9 Parity
Robot Genius
Microsoft App Locker
McAfee Application Control
Don’t want to pay ? !
•
•
•
•
•
•
•
•
•
•
Free Anti Virus [AVG, AVIRA, AVAST]
Free Firewall [Zone Alarm]
URL Scanner [AVG, WOT, RG Guard]
Trend Micro Web Protection Add on
Disable Auto runs
Returnil Virtual System / Windows Steady State
Wehn-Trust HIPS [MUST for Windows XP – ASLR Tool]
Win-pooch HIPS [Windows XP]
OSSEC HIDS
WinPatrol [BillP Studios]
How anti malware works?
Behavior Based Engine
(On Process Activities)
Basic Activity
Scanning *
Anti Virus Scanning
(On file content)
White listing
(On process
creation)
( * Process activity, file read or write )
Behaviors
database
Malware
Signature
database
Known
Applications
database
Malware Prevention System (MPS)
MPS - Approach
• Each application makes sequence of system calls for accessing
various OS resources through multiple control paths (normal
behaviour)
• When the application is infected with malware, its behaviour
changes
User
Process 1
User
Process 2
……………..
User
Process n
User Space
Operating System
System Calls
Kernel Space
Detects malicious activity before it causes damage to end system
i.e. before the system calls are executed by the operating system
MPS - Architecture
Flowchart
Malware Prevention System
Protection against
overall threats Process Execution
Control
1. Application Profiling and Model
Generation Process in a Sandbox
Model
Enforceme
nt Module
4. Client
Server communication
module
2. Server Manages the models and
admin can set the policies here
3. Based on the policies the model
gets pushed to clients
27
Model Generation
Optimization of the representation
of the profiled data
Considers the system calls that are
made on to a resource
Resource specific clustering Model
Each cluster can be defined as a
2-tuple<R,S>
–R : Resource
–S : System call
Example:
–Cluster 1- <A, {1,2,4}>
–Cluster 2- <B, {1,3,4,2}>
–Cluster 3- <C, {1,2,4}>
It is platform independent
implementation
Resource - A
System calls :
{1,2,4}
Resource - C
System calls:
{1,2,4}
Resource - B
System
calls:
{1,3,4,2}
Operations Hooked in MPS




File System Calls
Process hooks
Network Calls
Registry Calls
Deployment Scenario
System Architecture
Database Structure @ Server
Database Structure @ Client
Index File @ Server
Update Request
MPS Server
MPS Client
Major No,
Minor No,
OS type,
ModelUpdate,
Db Major No,
Db Minor No
UPDATE_REQUEST
UPDATE_RESPONSE
No.of Model Files,
Model File names,
ModelFile Path
File Transfer Request
MPS Server
MPS Client
Model File
Name with
path
TRANSFER_REQUEST
TRANSFER_RESPONSE
Contents of
the Model File
Log Message Request
Application name,
OS type,
Date,
IP,
Operation,
Path
Success
or
Fail
Client and Server – Technologies used
Server on Linux
– Apache Server 2.2
– Virtual Machine
– Windows XP, Vista and 7 images
– Linux 2.6.23 kernel image
– Java runtime environment
– PHP
– HTTP message format
– XML, OpenSSL
Windows Client
– Mini Filter Driver
– Call out Drivers
– Win32 programming
– C, C++ programming
– PE Executable format Open SSL
Linux Client
– Linux Security Modules
– C, C++ programming
– Qt Programming
– OpenSSL
Server GUI
Client GUI
Malicious Pdf
•
•
•
•
Creation of Axsle.dll
Creation of Icucnv34.dll
Write file on cvs.exe
The malware repeatedly tries to write cvs.exe
file and it gets blocked. The document doesn’t
open until the write file operation on cvs.exe is
completed.
Malicious Pdf
Stuxnet
• Behaviors Detected
– Hides view of system files
– Hidden image file
– File has system attribute
– Creates logon entry
– Unsigned binary
– Drops executable
– Modifies internet settings
– Spawns process
Stuxnet
Stuxnet
ATT27390 doc file
• Activities blocked
– Dropping of zipfldr.dll in system32 folder
– Dropping of wuaueng.dll in system32 folder
Field Testing Report
• MPS is compared with similar best commercial tools available in
the market like NovaShield, Mamutu, Malware Defender, Sana
Security Primary Response, Safe Connect, Threat fire etc.
properties claimed for Malware Prevention System
protects from the malware before they do any harm to your system.
assurance level
(mark to 5)
Remarks
3.75
is a very effective and low cost anti malware solution
4
has the capability to detect unknown malware.
4
is able to detect malware using its unique heuristic technology to
detect malicious behaviors.
Database can be expanded and we can update you with new
malicious behaviors.
3.5
-Not checked-
is easy to use. Even if your antivirus hasn’t detected a malware, you
can quarantine a process
4
Enforcement model applied
3
False positive generation
5
It doesn’t use any sort of malware signature database.
5
Field Testing Report
• MPS is found sensitive against blended MS office and PDF
documents wherein the MPS solution alone identified the
malicious activity as the other industry product remain silent
• Application has a tendency to raise false alarm against
benign documents as it might match the enforcement
policies defined
• Overall it is felt that the solution is detecting high level
targeted malware behaviours, but there is a need to improve
the capabilities by suppressing the false alarms.
Malware Resist
Simplifying and Strengthening Security
 Detection Based on Runtime Behaviour. All running programs
are monitored for a set of critical behaviors that could affect
the normal functioning
Salient Features
Detection Based on Runtime Behavior
Small memory footprint and high
detection rate
Co-exists with Anti Virus Solutions
Low False Positive Rate
Easy to Deploy and Use
Malware Prevention System (MPS)
 Behavior modeling of application
 Verification of application against critical resource
access
 Process Execution Control
 Enforcing the model at run time
 Guard from application exploits and implicit
malicious activity
 Fine grained monitoring of file, process, network
and registry access
 Co-existence with other antivirus solutions
Ongoing Research @ C-DAC Hyderabad
Design and Development of Anti
Malware Solution for Web
Applications and Mobiles
Malware Analysis
The approach to analyze the Malware
• Run the malware in isolated lab
• Monitor network and system connections
• Understand the program’s code
• Repeat until satisfied with gathered info
How to?
• Manual
– Dedicated system (ready to be compromised)
– Virtualized System
• Automated Analysis
Automated Analysis
Anubis [analyzing unknown binaries]
• http://anubis.iseclab.org/
Virus total [analyze suspicious file]
• http://www.virustotal.com/
Bit-Blaze [Malware Analysis Service]
• https://aerie.cs.berkeley.edu/
Norman Sandbox
Joe Box Sandbox
Sunbelt CWSandBox
Comodo [Comodo Instant Malware Analysis]
• http://camas.comodo.com/
Two Steps / Phases
• Behavioral (Dynamic) Analysis
• Code (Static) Analysis
• Gather as much as from behavioral analysis
• Fill the gaps from the code analysis
Analysis
Dynamic analysis
• Involves loading the file onto a test bed
system and launching it, while monitoring
it to determine what effect it has on the
system
Static analysis
• Involves examining and analyzing the
contents of the file without launching it,
either as a standalone executable or
through an application
Malware Analysis
• To analyze malware, we requires basic and advanced
knowledge in Windows and Linux concepts (depends)
• For example: while doing behavioral analysis of the
malware, we find malware modifies file A. – To get
more out of it, we must know what is the significance
of file A
Prepare the System
• Use VMWare and use the snapshot feature
to restore state after malware execution
• Use Virtual PC – execute the malware –
Close and Delete changes
• Physical System State Restore
– Returnil Virtual System
– Windows Steady State
Behavioral Analysis
• Activate various monitoring tools
• Execute the malware
• Terminate / suspend the malware process
– Sometimes malware process comes again and again
• Observe the results of monitoring tools
Process Explorer
• Free from Microsoft TechNet
• Super Task Manager
• Shows process tree
– We can know if malware created the new
processes
• Also shows files which a process is using
• Can see the strings also
Process Monitor
• Free from Microsoft TechNet
• Monitors the following activities
– Process creation
– File related
– Registry
– Network related
• Captures for all the process
– Best is to do it for all and then apply the filters
Regshot
Using IDAPro
• Can reveal a lot of information
• Great tool if user can reverse the C/C++ code
Use OllyDbg
• OllyDbg is a great debugger
• Open the sample using OllyDbg
Snort
• Either use snort in a separate virtual machine to
monitor its network activity
• Or use tools like wire shark
• Find
– IRC server to whom this sample connects
– Web servers?
• May notice DNS queries
Packed Malicious Executables
• Packers compress / encrypt the executable
• This is used
– Difficult to analyze
– Smaller size on hard disk
• However runs unpacked and original in memory
How it executes?
Executable
Decryptor
Packed
program
stored as data
Small Decryptor extracts
the packed code and
executes the code
Unpacked
program in
memory
PE Format
IMAGE_DOS_HEADER
MS-DOS Stub Program
Signature
IMAGE_NT_HEADERS
IMAGE_FILE_HEADER
IMAGE_OPTIONAL_HEADER
IMAGE_SECTION_HEADER
IMAGE_SECTION_HEADER
SECTION
SECTION
If it is packed
IMAGE_DOS_HEADER
MS-DOS Stub Program
IMAGE_DOS_HEADER
This is
Decryptor
code
IMAGE_NT_HEADERS
IMAGE_NT_HEADERS
IMAGE_SECTION_HEADE
R
IMAGE_SECTION_HEADE
R
IMAGE_SECTION_HEADE
R
IMAGE_SECTION_HEADE
R
SECTION
SECTION
MS-DOS Stub Program
SECTION
Original PE
SECTION
Packers Availiable
•
•
•
•
•
UPX
ASPack
Themida
Petite
VMProtect
PEiD
Process dumping with LordPE
• LordPE shows all the processes and can dump there
images from memory
• We can run the process from packed executable
– Anyways it has to unpack itself in the memory
• We can dump from memory using LordPE
Thank You