1
Detection of Trojan horse by Analysis of
System Behavior and Data Packets
Vamshi Krishna Gudipati, Aayush Vetwal, Varun Kumar, Anjorin Adeniyi, and Abdelshakour Abuzneid
University of Bridgeport, Bridgeport, CT 06604
(E1, E2, E3,E4)@my.bridgeport.edu and abuzneid@bridgeport.edu

Abstract— Trojan horse is said to be one of the most serious
threats to computer security. A Trojan horse is an executable file
in the Windows operating system. These executable files will have
certain static and runtime characteristics. Multiple Windows
system process will be called whenever a Trojan horse tries to
execute any operation on the system. In this paper, a new Trojan
Horse detection method by using Windows dynamic link libraries
to identify system calls from a Trojan Horses is explicated. Process
explorer is used to identify the malicious executable and to
determine whether it is a Trojan or not. Further, an attempt is
made to study the network behavior after a Trojan Horse is
executed using wire shark.
Index Terms—process explorer, executable, wireshark
I.
INTRODUCTION
Attacks on computers and networks are growing at an alarming
rate nowadays. Numerous attacks are being seen today and each
attack has a different motive and uses different strategy to
exploit systems. This makes the detection and prevention of the
attacks extremely difficult. Even though we have several type
of attacks on computers like malwares, virus, worms, Trojan
Horses are most widely being used and their popularity in the
field of security is increasing every day. Trojan Horses are
similar to any computer program that runs on our computer.
They pretend to do an action that is asked or requested by the
user, but usually it carries out actions that are specified by the
hacker or in other words, the one who created the Trojan Horse.
Trojan Horse basically gives remote access to the computer in
which the Trojan Horse is deployed. A Trojan horse cannot run
without the user of the system giving permission to it for the
first time. As it is an executable file, one must run it on his
system in order for it start working. So, the creator of Trojan
horse creates it in such a way that the user completely believes
it as the legitimate software, so that he will download and install
it in his system. If he did not run this on his system, there is no
way the hacker gets access to the system.
The mainly used program in most of the anti-viruses today is
signature based technology (Shugang 2009). Using this
technology, it is hard to detect the Trojan horse because of the
polymorphism property.
The Trojan detection algorithms can mainly be classified into
two categories. One is by using the Trojan signatures, but this
will not be efficient because the Trojans are not identical and it
is quite difficult to distinguish between the potentially harmful
files and the legitimate files. Furthermore, different Trojans are
being scripted every day, and their signatures will be different
from the existing. So this is observed to be an inefficient method
to detect the Trojan horses. Acquiring the signatures of all the
Trojan horses and keeping the anti-virus signature directory
updated all the time is both difficult and unmanageable. The
other way is by dynamic monitoring of ports, registries and
system configuration files.
Trojan can be described as a simple executable file in windows
operating system. But it has some of its properties very different
from the general executable files. We can use these properties
to detect the presence of a Trojan horse. The Trojans are always
not active on the client system. For a Trojan to work, the client
should run it at least once on his system. From (Cong, XiaoYan et al. 2010) then the Trojan starts doing the work for the
exploiter like sending the data to the listener, and providing
remote access.
In the below section, we have reviewed the related work in
section 2, we have given steps for creation of Trojan Horse in
section 3,and in sections 4 and 5, we have explained our
detection methods and finally we concluded in section 6.
II. RELATED WORK
During the few past years, many methods have been proposed
in the detection of Trojan horses. However, most of the methods
are focused on hardware Trojan Horses, worms and malwares.
Few works has specifically targeted software Trojan
horses(Shumei and Yanru 2010). Yu-Feng Liu proposed Trojan
horse detection based on system behavior using machine
learning method(Yu-Feng, Li-Wei et al. 2010). These machine
learning methods comprise of using the Instance based learner
(KNN), Naïve Bayes, decision tree and feature selection. This
involves collecting a few samples of data and storing them in a
database and analyze them through these machine learning
methods. The disadvantage of this technique is that the new
signatures are not detected. Chen Qin-Zhang et al, also
proposed a method of classification algorithms for Trojan horse
detection based on behavior(Qin-Zhang, Rong et al. 2009). This
method is basically implemented by creating an anti-Trojan
classification algorithm using the fuzzy classification which
2
includes data formalization, design of classification algorithm
which classifies sets of Trojans based on their behavior.
Jie Qin et al. also proposed a method of detecting Trojan horses
based on behavior analysis. This was basically done by
collection of different Trojan horses and analyzing their
behavior based on where they reside on the computer, what
change they effect on the registry, and the typical kind of
processes that are being called by the Trojan horses. (Jie,
Huijuan et al. 2010).
III.
TROJAN HORSE CREATION
Trojan horses are classified as below:
Remote access Trojan
Data sending Trojan
Destructive Trojan
Security software disabler Trojan
Denial-of-Service attack Trojan
Remote Access Trojan gives the remote access to system as if
the exploiter has physical access to the system. It is a piece of
code that gives an operator the remote access to the system
(NaiQi, Yanming et al. 2006). It basically provides the hacker
with unlimited access to infected endpoints. Using the prey’s
access privileges, they can access, modify, destroy and steal
sensitive business and private data including intellectual
property, personally identifiable information. While automated
cyber-attacks allow the exploiter to attack browser-based access
to sensitive applications, the remote access Trojans are used to
steal secure information through manual operation of the end
entity on behalf of the prey.
security aimed at digital forensics and penetration testing.
With the basic Commands on the Metasploit Framework
(Command line Interface), we have exploited a remote
computer. BackTrack is based on Linux environment. It is a
penetration testing platform that supports penetration testers,
bug hunters, and security professionals to perform assessments
in a purely native environment dedicated to hacking.
Irrespective of how it is being using BackTrack, one may install
BackTrack, boot it from a Live DVD, flash drive, the
penetration distribution has been customized down to kernel
configuration, every package, every assessment tool, scripts
used for exploitation and patch solely for the purpose of the
penetration tester. BackTrack is intended for all kinds of users
from the most savvy security professionals to early rookies to
the information security field. It promotes a robust and efficient
way to find and update the largest database of security tools
collection to-date. The user community range from highly
skilled penetration testers in the field of information security
field, government entities, information technology, security
enthusiasts, and individuals who are very new to the security
community. Whilst it is so easy to use and carry out
exploitation, it has been a blind folded job for anyone with the
minimum knowledge on how to use it and carry out an exploit.
We have created a backdoor by injecting a reverse meterpreter
payload onto an application that we want to use for exploitation.
Here, an exe file is used to exploit into the target computer. This
is a reverse TCP protocol for creation of the Trojan horse.
Below are the steps we used to establish a connection with the
remote computer.
Figure 1: give a description for this figure and make sure you call it from he
text
Data sending Trojan will be designed in such a way that it can
transmit sensitive data on a system like passwords, credit card
details, bank account details, security logs to the creator of the
Trojan. Destructive Trojan as the name suggests, is used for
destroying or deleting some files from the system. Anti-virus
may not be able to detect these virus. Security software disabler
Trojan disables all the security services like firewalls and
antiviruses that are deployed on the system. This makes the
system vulnerable to exploit and gain access to the computer
without any restriction. Denial-of-Service Trojan makes the
server unable to perform the user requests. It keeps the server
very busy, so that the server may not be able to serve any further
requests. All these attacks can be done using different payloads
and different approaches. Here, we show you all the
characteristics of the Trojan as stated above and eventually get
complete access over the remote computer. So, we first
considered to exploit a remote computer using the Backtrack r3
operating system, an Ubuntu Linux distribution that focuses on
Figure 2: give information for this figure.
root@bt:~# msfpayload windows/meterpreter/reverse_tcp
LHOST=192.168.2.11 LPORT=444 R | msfencode -e
x86/shikata_ga_nai -c 10 -t exe -x
/root/Desktop/IEXPLORE.EXE -o
/root/Desktop/IEXPLORE2.EXE
msf > use exploit/multi/handler
msf exploit(handler) > set PAYLOAD
windows/meterpreter/reverse_tcp
3
PAYLOAD => windows/meterpreter/reverse_tcp
msf exploit(handler) > set LHOST 192.168.2.11
LHOST => 192.168.2.11
msf exploit(handler) > set LPORT 444
LPORT => 444
msf exploit(handler)> exploit
setpayload windows/meterpreter/reverse_tcp
run getgui –u <<username>> -p <<password>>
Run multi_console_command –rc
/root/.msf4/logs/scripts/getgui/clean_up_20110112.2448.r
c
Figure 4: Process Explorer by SysInternals
Figure 3: give explanation for this figure.
IV. DETECTION BY PROCESS EXPLORER
We have proposed several methods to detect a Trojan in the
system. Whatever the process is, the principal goal is to
segregate a suspicious process or program out of several others,
based on the behaviors that Trojan or suspicious file relatively
shows. Before analyzing or detecting a Trojan horse, it is
necessary to figure out the objects of Trojan horse operation.
Trojan horses usually operate on registry, file, port, process,
system service and other I/O interfaces like keyboard, webcam
etc. Based on these objects that Trojans act upon, we now know
where to monitor the activity.
We could use various tools to monitor process activities in a
system. We can use Wireshark to analyze packets through the
network or do some form of (dll) injection in a system process
so that we are notified whenever a foreign process tries to take
control over native system processes. A Trojan horse consists
of section of program codes. If it runs on the target computer, it
must call different API functions. So, we can use an unorthodox
API Hook technology methodology to monitor and intercept
Trojans. The basic motto is to code a function that is invoked
every time a certain system process in windows is started.
Hooks were distributed by Microsoft predominantly to help
programmers to straighten out the errors of their applications,
but they can be put to use in many different ways. However,
using API hooking and DLL injection to detect what a certain
foreign harmful process is doing in our system is a complex
matter, because every time we inject (dll) to a process, we are
inflicting with the system memory that is otherwise always
reserved for that particular process. This could bring several
problems while using the system simultaneously(Shicong,
Xiaochun et al. 2012). The simplest way to monitor the system
process is to use the process explorer by SysInternals. It is
basically a tool similar to Windows Task Manager with more
freedom in obtaining information. It can be very useful in
analyzing and detecting any malicious process/code running in
the system. It displays any current process running on the
system, its child, process ID, description and various other
useful information associated. It can be a great tool to detect
Trojans and Malwares on system.
Whenever we are looking for any malicious process running on
our system, we are usually after the processes that do not have
genuine digital signatures, icon/description or company name,
that lives in user directory or user profile, that are packed, that
include strange URLs in their strings and have open TCP/IP
end-points. Therefore, there are mainly the following things to
look at using the Process Explorer tool: whether the process is
packed or not, if the process resides in auto start location, the
process-timeline, whether the process is digitally verified and
whether the process contains any (dll) injected into core system
processes.
Here, “Packed” means whether compressed or encrypted.
Malicious programs usually use packing (with common
techniques such as UPX) to make antivirus signature more
difficult to match. The indication of whether a process is packed
or not is given by a certain color highlight in Process Explorer.
Usually, purple highlighting means that a process image is
packed.
There are several other colors too, for example, pink process
signifies hosting windows processes, and blue highlighting
signifies that the process is running in the same security context
as windows processes, white color signifies system processes
or some processes running on a different user account.
However, our main focus here is on determining whether a
process is packed or not, which is signified by purple
highlighting because it is extremely common for
Trojans/Malwares to get compressed or encrypted.
Figure 5: suspicious process TSServ.exe highlighted in purple.
It is also common for Trojans and Malwares to reside in the
system path that is used for auto-start of applications/processes
or they attach themselves in processes that start as soon as
4
system boots. They often hide behind Svchost, Rundll32 and
DLLHost. Any suspicious process can be checked if it resides
in an auto-start location using Process Explorer.
Process Explorer also shows the process lifetime of a process
since the booting of the system. Some of the processes have
Natural Lifetime. They start easily from when system boots. So
using the process explorer, we can find the ones that have
launched later. In most cases, the backdoor and malicious codes
are the ones that start later.
Image verification is one of the important tools that can assist
in the detection of malwares and Trojans. Image Verification is
the process of checking digital signature on file. Most
Legitimate software are digitally signed i.e. there is tamperproof sealed image that gives the identification of product and
company. Any process running on the system can be verified
by clicking verify button to check for signatures. All Microsoft
codes are digitally signed i.e. hash of file is signed with
Microsoft’s private key. Signature is checked by decrypting the
signed hash with public key(Shumei and Yanru 2010). The
programs aren’t genuine can’t be verified and they can be put
under suspicion.
activities and properly understand the activities on the network.
Wireshark is proposed for the analysis because it is one of the
best if not the best packet analysis tool as it supports over 850
protocols which has the highest numbers of protocols
supported. It is an open source tool which is readily available to
all without a charged cost and also supports all modern
operating system, including windows, Mac OS X, and Linux
based platforms(Cong, Xiao-Yan et al. 2010).
Wireshark is a graphical user interface based, packet analysis
tool which goes through the phase of collecting, converting and
analyzing of captured data from the network.
Collection Phase: In this phase, the packet analysis tool
assembles the raw binary data from the wire. Generally, this is
carried out by switching the selected network interface into
promiscuous mode. In this mode, the network card can listen to
all the network segment, not only the traffic that is addressed to
it.
Conversion Phase: During this phase, the captured binary data
is converted into a readable form. This is where the most
advanced command-line packet sniffers stop. At this point, the
network data is in a form that can be interpreted only on a very
basic level, leaving the majority of the analysis to the end user
Using different filters for the analysis depending on what the
end result of the end users are.
Analysis Phase: This is the third and the final phase which
implicates the analysis of the readable form data. This is by far
the most important phase which helps in better understanding
of the network activities. The packet analyzer takes the captured
network data, verifies its protocol based on the material
extracted and begins to analyze the protocol’s specific features
in accordance with the filters that are applied in the analysis.
VI. CONCLUSION
Figure 6: the suspicious file has no digital signature and resides in auto start
location (highlighted in black).
By analyzing the behavior of the system injected with Trojan
Horse using process explorer and packet analysis by wire shark
this paper proposes a new detection algorithm for detecting the
Trojan horses. Analysis shows that this method is more
advantageous than using the static methods using digital
signatures.
REFERENCES
Another feature that Process Explorer provides is the DLL
view. Malwares and Trojans can hide inside any legitimate
process. Typically such processes load via auto-start. Malwares
attach themselves to such processes via dll injection so that
whenever the process runs its associated dll, the malicious
codes get to start. We can also see the dll associated to processes
and detect any unusual behavior.
V. PACKET ANALYSIS BY WIRESHARK
Packet analysis, often referred to as packet sniffing or protocol
analysis, entails the process of capturing and interpreting live
data as it flows across a network in order to understand better
what is happening on that network. Packet analysis is
performed by a packet sniffer, a tool used to capture raw
network data going across the wire or a network. Wireshark is
the packet analysis tool which we propose to detect malicious
Cong, J., et al. (2010). Dynamic Attack Tree and Its Applications on Trojan
Horse Detection. Multimedia and Information Technology (MMIT), 2010
Second International Conference on.
Jie, Q., et al. (2010). A Trojan Horse Detection Technology Based on Behavior
Analysis. Wireless Communications Networking and Mobile Computing
(WiCOM), 2010 6th International Conference on.
NaiQi, W., et al. (2006). A Novel Approach to Trojan Horse Detection by
Process Tracing. Networking, Sensing and Control, 2006. ICNSC '06.
Proceedings of the 2006 IEEE International Conference on.
Qin-Zhang, C., et al. (2009). Classification Algorithms of Trojan Horse
Detection Based on Behavior. Multimedia Information Networking and
Security, 2009. MINES '09. International Conference on.
Shicong, L., et al. (2012). A General Framework of Trojan Communication
Detection Based on Network Traces. Networking, Architecture and Storage
(NAS), 2012 IEEE 7th International Conference on.
5
Shugang, T. (2009). The Detection of Trojan Horse Based on the Data Mining.
Fuzzy Systems and Knowledge Discovery, 2009. FSKD '09. Sixth International
Conference on.
Shumei, Z. and J. Yanru (2010). The Model of Trojan Horse Detection System
Based on Behavior Analysis. Multimedia Technology (ICMT), 2010
International Conference on.
Yu-Feng, L., et al. (2010). Detecting Trojan horses based on system behavior
using machine learning method. Machine Learning and Cybernetics (ICMLC),
2010 International Conference on.