Signature-Builder
Testing
Document
1 Testing functional requirements ............................................................................... 3
1.1 SignatureBuilder ..................................................................................................... 3
1.2 Evaluation Envoironment and CFL ....................... Error! Bookmark not defined.
2 Testing non-functional requirements...................................................................... 10
2.1 Speed, Capacity & Throughput ............................................................................ 10
2.2 Reliability: ............................................................................................................. 10
2.3 Safety & Security: ................................................................................................. 11
2.4 Usability ................................................................................................................. 10
3 Test-Driven development ......................................................................................... 11
4 Random & automatically-generated tests ............................................................... 13
5 Testing the user interface ........................................................................................ 14
6 Testing build, integration & deployment................................................................. 15
1 Testing functional requirements
1.1 Signature Builder – server
Requirement #1 in the ARD: Extract Functions
Testing IDAPRO extractor
Test No.
Test Description
Expected Result
1
Success scenario – trying to extract
functions from file – while file exists
A list of functions extracted
2
failure scenario – trying to extract
functions but file doesn’t exist
Error message
Testing dataMiningExtractor
Test No.
Test Description
Expected Result
1
Success scenario – trying to extract
functions from file – while file exists
A list of functions extracted
2
failure scenario – trying to extract
functions but file doesn’t exist
Error message
Requirement #2 in the ARD: Filter Functions
Test No.
Test Description
Expected Result
1
Success scenario – filtering a list of
functions and the list exists
A list of non-common functions
2
Failure scenario – trying filter a list
while there is no list of functions
Empty List
Requirement #3 in the ARD: Generate Signatures
Test No.
Test Description
Expected Result
1
Success scenario – expanding the
function while the attributes are not
null
The function is expanded
2
failure scenario – trying to expand
the function while the attributes are
null
Error message
Requirement #4 in the ARD: Select Candidate
Testing RandomSelctor
Test No.
Test Description
Expected Result
1
Success scenario – Select a
signature from the candidates and
the list exists
A random signature selected
2
failure scenario – trying to Select a
signature while the list doesn’t exist
Error message
Testing ClusterSelctor
Test No.
Test Description
Expected Result
1
Success scenario – Select a
signature from the candidates and
the list exists
A signature with maximum score of
clusters selected
2
failure scenario – trying to Select a
signature while the list doesn’t exist
Error message
Testing MinEntropySelctor
Test No.
Test Description
Expected Result
1
Success scenario – Select a signature
from the candidates and the list exists
A signature with minimum
entropy selected
2
failure scenario – trying to Select a
signature while the list doesn’t exist
Error message
Testing probabilitySelector
Test No.
Test Description
Expected Result
1
Success scenario – Select a
signature from the candidates and
the list exists
A signature with minimum
probability of existence in a benign
file.
2
failure scenario – trying to Select a
signature while the list doesn’t exist
Error message
Requirement #5 in the ARD: create Signature
Test No.
Test Description
Expected Result
1
Success scenario – trying to create
signature after initialize all the
fields.
Returns a signature.
2
Failure scenario: trying to create
signature while selector is not
initialize.
Error message
3
Failure scenario: trying to create
signature while extractor is not
initialize.
Error message
4
Failure scenario: trying to create
signature while CFL is empty
Error message
1.2 Evaluation Envoironment and CFL
Requirement #1 in the ARD: CFL
Testing Create CFL
Test No.
Test Description
Expected Result
1
Success scenario – creating CFL and
there is an input file
The cfl is initialized
2
Failure scenario – trying to create cfl
while there is no input file
Error Message
Testing Reset CFL
Test No.
Test Description
Expected Result
1
Success scenario – success clearing
the CFL while the CFL exists
The CFL is Empty
2
Failure scenario – trying to reset the
cfl while there is no CFL
Error Message
Testing Save CFL
Test No.
Test Description
Expected Result
1
Success scenario – success saving
the CFL while the CFL exists
The CFL is saved as a file
2
Failure scenario – trying to save the
cfl while there is no CFL
Error Message
Testing Update CFL
Test No.
Test Description
Expected Result
1
Success scenario – the Functions
were added to the cfl
The CFL includes the new functions
while the CFL exists
2
Failure scenario – update the cfl
while there is no CFL
AND
CFL.size>0
Error Message
Requirement #2 in the ARD: Signature Malwares
Test No.
Test Description
Expected Result
1
Success scenario – trying to produce Returns a set of signatures.
signatures from each malware after
initialize all the fields.
2
Failure scenario: trying to create
signatures while selector is not
initialize.
Error message
3
Failure scenario: trying to create
signatures while extractor is not
initialize.
Error message
4
Failure scenario: trying to create
signatures while CFL is empty
Error message
Requirement #3 in the ARD: Signature Benign files
Test No.
Test Description
Expected Result
1
Success scenario – trying to produce Returns a set of signatures.
signatures from each benign file
after initialize all the fields.
2
Failure scenario: trying to create
signatures while selector is not
initialize.
Error message
3
Failure scenario: trying to create
signatures while extractor is not
initialize.
Error message
4
Failure scenario: trying to create
signatures while CFL is empty
Error message
Requirement #4 in the ARD: Show Statistics
Test No.
Test Description
Expected Result
1
Success scenario – Compare each
malware's signature with the set of
benign file's signatures.
Return the probability of the
uniqueness for the malware's.
2
Failure scenario: there is no set of
benign file's signatures
Error message
3
Failure scenario: there is no
malwares signature
Error message
2 Testing non-functional requirements
2.1 Speed, Capacity & Throughput
Test No.
Test Description
Expected Result
1
Loading CFL
created from 1GB of files should
take no more than 10mins
2
Generating CFL
Generating CFL from 1GB of files
should take no more than 2hrs
3
Generating signature for a malware
file
1MB should not take more than 5sec
4
Searching a 48byte signature in GB
of files
should take less than 8sec
5
Generating 3grams and 2-grams
repository from the CFL files
created from 1 GB of file, should
take no more than 2hrs.
2.2 Reliability:
Test No.
Test Description
Expected Result
1
generate signatures up to 5Gb files
The system will have an output
without memory problems.
We will perform many tests on the system with different sizes of cfl, benign files,
malicious files and measure the system's reliability by calculating the false alarms
percentage.
2.3 Safety & Security:
The system domain expert will be authorized to configure the system and all the
components in it.
Test No.
Test Description
Expected Result
1
Change configuration of the
signature builder
Check if the output is corresponding
to the input and the configuration of
the system
2.4 Usability:
Testing usability will be by letting someone to try the system, doing the follows:
Installing the evaluation environment and the signature builder.
Change configuration of the system
Create cfl, add files and delete files.
Create signature to malicious files
Use the evaluation environment and get statistics for a set of files.
Check the response of the system.
At last get his opinion of his understanding and using the system.
3 Test-Driven developments
We didn't use the TDD approach.
We think we don’t have enough experience in software developing to use this
approach for such a big project.
In addition, as part of the course we had to present a full design of the
system, this software developing approach contradicts the TDD approach,
which the design should be created through the tests.
4 Random & automatically-generated
tests
Random and automatically tests for the signature builder are done by the
evaluation environment. The signature builder is a server and the evaluation
environment is actually a client that tests the signature builder according to different
configurations. The main idea is to help deciding on a good signature: unique and
optimal.
Each configuration consists the following input:
-
CFL size in MB
maximum signature length in byte
Function similarity threshold
Offset size in byte
Function Extractor
function selection.
The output will consist the following:
-Processed - The number of malware files that the system managed to generate a
signature for them.
-Processed (%) - Processed / Total Malware Files.
-Signature Hits - The number of malware files that gives at least one False Alarm,
which means the number of unique malware files that produced False Alarm.
-Signature Hits (%) - Signature Hits / Processed.
-Hits In Files - The number of the signature's hits in the Control Group for any pair
(signature, file)
-Unique Signature - The number of unique signatures produced FA.
-Different Files - The number of distinct files in the CG that has at least one hit.
-Different Files (%) – Different Files / Total Control Group Files.
5 Testing the user interface
We have two components with GUI to test – one is the signature builder itself and
the other one is evaluation environment.
Testing the configuration in the signature builder – the test is simple, for every
change in the configuration screen, all we have to check if there was a change in the
configuration.txt file.
Cfl manager – for any change of the cfl we will test the new cfl, when we delete it –
we check if the size of it is 0, if we add/delete function – check new size of the cfl.
For the other screens in both the signature builder and the evaluation environment
all the tests of their functionality were mention on the other parts of this document.
6 Testing build, integration &
deployment
The signature builder and the evaluation environment package will be a standard
windows installation executable, thus our project doesn’t need any special needs for
the building and deployment stages. The use of this standard is different from a
GNU sources packages, therefore, the installation and the deployment for the project
is easy to test – such an installer has already been tested in the past.