International Journal on Engineering Applications (I.R.E.A.), Vol. 10, N. 5
ISSN 2281-2881
September 2022
A Comparative Study of Current Dataset
Used to Evaluate Intrusion Detection System
Wasim A. Ali1, P. Sandhya2, Michele Roccotelli1, Maria Pia Fanti1
Abstract – Today, since the world increasingly depends on computers and automation, securing
T
applications and systems has become critical. The possibility of threat and violence is expanding
rapidly due to the growing complexity of modern networks and services. This study focuses on
current datasets, the gaps and limitations of these datasets, and their effects on creating Intrusion
Detection Systems (IDS). The study outlines the existing datasets that show an apparent absence of
real-world threats, attack representation, and outdated threats, limiting the efficiency of machine
learning IDS techniques. This work presents the most Machine Learning techniques for
implementing IDS and a comprehensive review of the available benchmark datasets. We believe
this review will enable future researchers in this field to understand better different kinds of IDS
datasets, advantages, and limitations. Copyright © 2022 Praise Worthy Prize S.r.l. - All rights
reserved.
I.
Introduction
datasets used to test IDS must be updated continuously.
The dataset contains normal and abnormal network
traffic, which allows the model to classify data with
enough examples using labels derived from this dataset;
therefore, these data sets don't perform well when new
attacks occur, such as zero-day exploits. Several
techniques have been used effectively for IDS; however,
each IDS method trains and tests the dataset in a different
manner. For example, on NSL KDD, KDD CUP 99 [5],
or DARPA [6] dataset, different algorithms used for IDS
implementation and based on numerous criteria, such as
the variability in a dataset, feature and parameter
optimization, the model produced can be evaluated and
compared [7]. Choosing a suitable metric for
performance is a crucial factor that should be considered
and techniques used for IDS. Several metrics are used for
this purpose, but the most frequently used is accuracy,
the confusion matrix. Precision and recall are derived
from the confusion matrix, which can evaluate the
algorithm efficiency and classification quality [8]. Even
while these metrics can accurately measure the
algorithms' efficiency, they are not sufficiently detailed
to estimate the IDS algorithms' efficiency. Even if falsepositive and false-negative issues remain, the classifier's
accuracy varies across attacks. The existing surveys
focus on reviewing the latest IDS techniques regarding
the efficiency of ML algorithms and the best evaluation
metrics. However, there is less coverage of the datasets
generated for evaluating and examining the proposed
models for IDS. IDS dataset can be obtained by
combining data from various sources, including network
traffic flows [9], which contain information about the
host, user activity, and system configurations. This data
is needed to investigate the network attack patterns and
R
EP
R
With the rapid development of the network's speed,
the wide use of computer technology, and an immense
online data revolution, systems encountered numerous
problems such as stability, privacy, and security.
Significant improvement has been made to strengthen the
reliability of information systems and networks.
Nonetheless, stability, privacy, and the secrecy of
devices in computer networks are potentially significant
problems. Intrusion Detection Systems (IDS) were
developed to be the first line of defence for any system
on the network [1]. In conjunction with the security
measures, such as electronic access control systems,
authentication, Identification systems, and encryption
techniques, IDS can be developed to secure the systems
better from any cyber attack [2]. There are two major
types of current IDS: misuse-based detection (or
signature-based detection) and anomaly-based detection.
Both technologies have limitations due to the
acceleration and massive inflation in the volume of data
and its sources [3]. Detection of network anomalies is
difficult due to the complex complexity of network
traffic. Anomalies occur due to intrusions and attacks in
the network and malfunctioning equipment or network
overloads. Various Machine Learning (ML) approaches
were developed to detect anomalies and intrusions in
devices used by network users. Where a bulk amount of
datasets are available online, models are trained and
tested using those datasets. Training data is the best way
to prove the effectiveness of proposed approaches, but
the difficult task is to evaluate these techniques and
address the effects on different datasets [4]. The type of
network threats has evolved over the years; thus, the
IN
Keywords: Anomaly, Intrusion Detection System, Cybersecurity, Machine Learning, Datasets
Copyright © 2022 Praise Worthy Prize S.r.l. - All rights reserved
https://doi.org/10.15866/irea.v10i5.21030
336
Wasim A. Ali et al.
unusual activities. Hence, updating IDS datasets and
classifying them according to their advantages and
limitations is necessary. Accordingly, we conducted a
comprehensive survey, highlighting the current datasets
used in evaluating the IDS and showing their advantages
and disadvantages. In addition, the study presents recent
advances in IDS datasets that research communities can
use to develop efficient ML-based IDS. The paper also
discusses the need of real-time datasets to obtain results
that are more relevant to reality and proof the efficiency
of the performance of any proposed IDS in the future.
The paper structure can be summarized as follows:
Section II is an overview of IDS and network attacks.
Section III is an overview of ML approaches used for
IDS. Section IV presents the current IDS datasets.
Section V discusses limitations andchallenges. Finally,
Section VI presents the concluding remarks.
Detectionand false-positive rate are key parameters
that should be noted during IDS evaluation to be
considered successful. For an IDS evaluation, various
metrics can be used. There are few standard metrics in
most of the papers surveyed that evaluate all proposed
approaches. Table II presents the most common
evaluation metrics. This article reviews different ML
approaches used in IDS. Besides, we will discuss the
current dataset's characteristics, advantages, and
disadvantages to indicate the best dataset that researchers
can use for their work.
V.
Machine Learning Techniques
T
Machine learning helps systems learn and increase
automatic abilities without manual manipulation or
programming. For IDS, various ML methods are utilized.
Several ML algorithms were applied in the
construction of IDS, as shown in Figure 2. Authors in
[15] gave a thorough review study describing the
differences between unsupervised and supervised
learning applied on IDS.
Intrusion Detection System
EP
R
IDS can be defined as software or tool developed to
detect malicious activity and unauthorized transactions
on the network. Every other operation is reported to the
Database Manager. An IDS works by tracking system
activities by detecting system bugs, file integrity, and
performing pattern analysis. It scans the systems and
watches the latest vulnerabilities that might lead to an
attack [10]. IDS aims to help computer systems to cope
with threats. Moreover, IDS collects data from many
sources inside networks and systems and compares it to
previously established discriminatory patterns to
determine whether threats or vulnerabilities exist [11].
According to the detection location, IDS may be split
into two main categories: Host-based IDS (HIDS) and
Network-based IDS (NIDS). HIDS detects suspicious
user activity on the local host machine, while NIDS
detects the attack primarily by network traffic. Each of
them will look at the log file and the database to compare
results. The detection approach can be categorized as
signature-based and anomaly-based. Signature-based,
also known as misuse-based, gathers malware signatures
continually and then creates a malware signature
database. When a network traffic flow or activity fits the
malware database's signature, it is marked as abnormal.
In comparison, anomaly-based detection and network
packet analysis were compared to a standard baseline
[12]. Figure 1 describes the IDS classification according
to data collection methods and detection techniques.
IDS Evaluation Metrics
IN
II.
IV.
Fig. 1. Classification of intrusion detection systems
R
Main category
Infection
Exploding
Prop
III. Network Attacks
Cheat
A network attack can be defined as unauthorized
actions on the network devices or network flaws
attempting to break system protections to change, destroy
or steal confidential data. In the early decades [13], the
attacks were classified into two types: external and
internal. Whereas the authors in [14] categorized the
network attacks into seven categories based on
implementing such attacks, as seen in Table I.
Traverse
Concurrency
Copyright © 2022 Praise Worthy Prize S.r.l. - All rights reserved
TABLE I
ATTACKS C ATEGORIES
Definition
Aim to infect the target system
either by tampering or by
installing malicious files in the
system.
Seek to explode or overflow the
target system with bugs.
Gather information about the
target system through tools.
Examples
Viruses, Worms,
Trojans.
Buffer Overflow.
Sniffing, Port
sweep, IPsweep.
IP Spoofing, MAC
Spoofing, DNS
Typical examples of this
Spoofing, Session
category include attempts to use
Hijacking, XSS
fake identity
Attacks, Hidden
Area Operation.
Attempts to crack a victim
Brute Force,
system through a simple match Dictionary Attacks,
against all possible keys.
Doorknob Attacks.
Victimize a system or a service
by sending a mass of identical
Flooding, DDoS
requests which exceed the
(Distributed Denial
capacity that the system or the
of Service).
service could supply.
International Journal on Engineering Applications, Vol. 10, N. 5
337
Wasim A. Ali et al.
TABLE II
THE MOST C OMMON EVALUATION METRICS
True positive rate
(TP)
True Negative
rate (TN)
False-positive
rate (FP)
False-negative
rate (FN)
Number of intrusions correctly detected
= 1–
(2)
Number of non- intrusions incorrectly detected
=
(3)
+
= 1–
Number of intrusions incorrectly detected
The percentage of correctly predicted instances in the testing dataset
ErrorRate (ER)
The percentage of all predictions that were incorrectly classified
The number of true positives divided by the total of true positives and false
positives
The number of true positives divided by the total of true positives and false
negatives
The harmonic means of recall and precision knew as f-value orf-score
Matthews
Correlation
Coefficient
(MCC)
Measures the correlation between the predicted results and the real data
Legitimate
FN
TN
(5)
)
(6)
(7)
+
=
+
precision × recall
=2×
precision + recall
=
(
(
+
×
)(
MCC =
) − (
×
)
+
+
)(
)(
(8)
(9)
(10)
+
)
IN
F-measure (FM)
=
T
Accuracy (Acc)
(4)
Prediction
Intrusion
Intrusion
TP
legitimate
FP
+
=
( +
+
+
= 1–
Actual
A type of matrix allowing easy comparison between predicted and actual
classes
Recall
(1)
Number of non-intrusions correctly detected
Confusion
Matrix (CM)
Precision
TP
+
=
EP
R
selection method with an SVM classifier. In this work,
the authors found that combining feature selection and
parameter optimization for SVM decreases training time
and improves the classifier's performance. Furthermore,
the proposed FS PSO-SVM model provides a high
detection rate and minimal false-positive rate. They
evaluated the proposed model's efficiency using the
NSL-KDD Dataset, containing four different network
attacks: DoS, R2L, and U2Rand Prob.
K-Means Algorithm
K-means or distance-based algorithm is the most
common unsupervised technique. The k-means clustering
attempts to reduce the distance between points in a
cluster and their centroid. Clustering is based on the
measurement of distances between classification objects.
This distance measurement compounds the
observations into homogenous groups [17]. The K-means
algorithm is primarily used in time series data to obtain
patterns for matching and clustering connections in
datasets. In the K-Mean algorithm, k items are used as
cluster centres, the distance between each centre and an
item is calculated, and then the item is placed near the
nearest centre. After the clusters have been updated, the
operation will be repeated, with all clusters being
supplied. Due to the rise in the importance of stream
mining, researchers are paying more attention to it as a
technique to mine big datasets. However, it has many
challenges, including anomaly and outlier detection,
fraud detection, and other complications [18].
Unsupervised methodssuch as the K-means algorithm,
although the method is fast and efficient but not widely
used. As shown in [19], the authors developed a model
that uses the CaliniskiHarabasz indicator to improve the
K-means approach for determining the number of
Fig. 2. Machine learning techniques
R
The most common MLmethods used in IDS include
SVM, K-means, Logistic Regression, the Naive Bayes
method, Artificial Neural Networks (ANN), and
Principal Component Analysis (PCA). These approaches
successfully and efficiently detect abnormalities and
intrusions in the network. In the following sections, some
MLmethodsutilized to identify such threats are described.
V.1.
V.2.
Support Vector Machines
Support Vector Machines (SVM) algorithm is a
supervised ML approach that is always used for
regression as well as classification. The primary idea of
the SVM is to develop an optimal hyper-plane that can
classify two classes. Authors in [16] introduced a novel
ID model based on Particle Swarm Optimization (PSO)
algorithm that combines an information gain feature
Copyright © 2022 Praise Worthy Prize S.r.l. - All rights reserved
International Journal on Engineering Applications, Vol. 10, N. 5
338
Wasim A. Ali et al.
clusters necessary for clustering. The proposed model is
evaluated on two datasets: NSL-KDD and the
CICIDS2017. Compared to the SVM classifier, a
supervised classifier, the experimental findings reveal
that the suggested model was extremely efficient in both
detection and time consumption. To develop an optimal
classification, the proposed work in [20] combined two
clustering and optimization approaches, the K-means
algorithm and Simulated Annealing, in their work. The
proposed method showed satisfactory accuracy; however,
it was evaluated on the NSL-KDD dataset, and tests
showed that it improved IDS efficiency and misdetection
rates.
V.3.
The probability (P) of one entity does not affect the
probability of another entity, i.e., the attributes are
independent [24]. First, this algorithm calculates the total
output cases. The algorithm then determines the
conditional probability for each data class. As a result,
the Naïve Bayes algorithm has a short implementation
time, can work with continuous and discreet attributes,
and learns quickly.
Many recent papers, such as [25], proposed a novel
paradigm for implementing an IDS based on the Naïve
Bayes algorithm. The model is intended to secure the
Internet of Things (IoT) infrastructure from the most
frequent attacks generated by intruders on the network,
such as DDoS. This attack type was highly disseminated
in the IoT network because of the heterogeneous
resources that were useful to attackers. The authors built
and implemented an IDS consisting of many agents (NBMAIDS). MA agent was deployed over the network for
NB classifier implementation. The IoT system gained
access to sensors, which gather data on network nodes
displaying abnormal behaviour. Using the suggested
multi-agent NB classifier methodology, we demonstrated
that the classifier is helpful for quickly blocking assaults
with a lower execution cost. The NSL-KDD dataset was
utilized to evaluate the effectiveness of the suggested
classifier.
Logistic Regression
1
1+
(11)
( )
V.5.
Artificial Neural Networks
R
F( ) is a value between 0 and 1, and is the input
function. SPLR-based feature selection and intrusion
classification are proposed in [22] for IDS. This strategy,
called sparse regularized optimization and sometimes
known as SPLR, selects a subset of features from the
original feature set and uses them to model data for
classification. In a linear SPLR model, characteristics
from a dataset repository are used to build a linear
classifier, and then coefficients of the Classifier are
discovered. Filtering and wrapper techniques, like the
filter (ranking) and wrapper (classification) techniques,
differ from feature selection techniques as the feature
selection and classification are separate. At the same
time, SPLR incorporates both into a cohesive framework.
Binary classification methods such as Logistic
Regression can be used separately or with different data
mining techniques to identify abnormal traffic from the
normal ones in the network. Using the UNSW-NB15
dataset, authors in [23] developed a two-stage anomalybased N-IDS. They use Recursive Feature Elimination
and Random Forests, among other techniques, to select
the best dataset features for ML; then, they use the
Logistic Regression method with different data mining
techniques to distinguish intrusive from normal traffic.
IN
( )=
T
A supervised ML approach called logistic regression
identifies discrete class sets. The logistics functions use
sigmoid functions in their algorithms, which transfer
various predictions into probabilities [21]. In this
equation, F represents the sigmoid function, which has
the following definition:
R
EP
The human brain is a neural network in which several
neurons are connected to form several parallel pathways.
Neurons usually interconnect and interact in a highly
complex way.
The nodes of ANNs are bound together utilizing
multiple interconnecting nodes. Authors in [26]
employed a sort of neural network model known as CNN
in their study. The authors note that utilizing CNNs to
identify intrusions could be beneficial. Traffic
characteristics are included in the raw information, and
CNN adopts a certain kind of NN in response. The most
significant obstacle to contend with was the availability
of an unbalanced dataset. Accuracy is improved, and a
lower false alarm rate is also achieved. The study also
devised a way to lower the computational cost of raw
traffic vector transformation to picture format.
Additionally, the researchers developed a technique to
transform raw traffic data into an image format, lowering
processing costs. In [27], the proposed model was for
detecting network attacks based on Neural Networks and
Deep Learning (DL). The model has been evaluated on
the CICIDS2017 dataset. The proposed neural network
algorithm has shown to be more accurate in detecting
various types of network attacks. Another work [28]
introduced a new technique for NIDS that uses a Genetic
Algorithm (GA) to identify an enhanced feature subset
(FCM). The proposed method significantly improves the
final detection performance. The NSL-KDD dataset is
used in this work to determine how the model is working
effectively.
V.4.
Naïve Bayes Algorithm
The Naïve Bayes algorithm is based on the Bayes
theorem, which is about probability updating depending
on provided evidence:
( | )=
( \ ) ( )
( )
(12)
Copyright © 2022 Praise Worthy Prize S.r.l. - All rights reserved
International Journal on Engineering Applications, Vol. 10, N. 5
339
Wasim A. Ali et al.
V.6.
VI.3. NSL-KDD
Principal Component Analysis
The NSL-KDD dataset was built in [32] to come up
with solutions for the KDD CUP 99 dataset. The number
of records in this dataset is reasonable, making the
dataset cost-effective in running experiments on a whole
set instead of a small selected group. The main advantage
of the NSL_KDD dataset is that it excludes redundant
data records; therefore, classification is not based on
frequently occurring data records.The NSL-KDD train
dataset has 125,973 records, while the test dataset
includes 22,544 records. Incorporating the full NSLKDD dataset is practicable since the NSL-KDD dataset is
large enough to utilize the entire dataset without the need
to sample it randomly. The findings are repeatable and
similar, as shown in many studies.
The Principal Component Analysis (PCA) is a
statistical technique used to reduce dimensions of
datasets, including many variables, all of which have
relationships with each other. Moreover, labelled data is
not required when employing PCA on any training set.
PCA is one of the unsupervised learning approaches
that reduce the unlabeled training set. A novel anomaly
detection technique, Robust PCA (RPCA), was
developed in [29].
Network packet capture data illustrates the influence
on various network attack detection systems in the
proposed technique RPCA. In DARPA's tests, the
DARPA dataset has been utilized in several attack
scenarios, including Distributed Denial of Service
(DDoS), assaults, IP sweeps, and probing and breaking.
The model achieved the lowest false positives while
ensuring a high positive rate. Furthermore, this model
could identify network threats. the approach could
identify packet streams even in the case of an assault that
had not been previously experienced or trained.
The Current IDS Datasets
VI.1. DARPA
T
R
Today, numerous organizations use various data
formats for their own goals and contribute to public
repositories. Therefore, the relevant pre-existing dataset
will be explained in this section.
The UNSW-NB15 was introduced in [33]. This
dataset has 42 features; three are categorical, while 39 are
numeric. The dataset is divided into UNSW-NB15TRAIN, used in training different models, and UNSWNB15-TEST used to test trained models. This dataset has
different network attack categories, including Shell code,
worms, DoS, and Backdoor [34]. Newly developed
techniques are used to produce the UNSWNB15 data set
features. This dataset is accessible for research use and is
also available online.
IN
VI.
VI.4. UNSW-NB15
VI.5. ISCX2012
Due to the scarcity of datasets and the ever-changing
patterns and evolution of intrusions, dynamic datasets are
essential. The ISCX2012 was built to address the need
for dynamically generated datasets with a comprehensive
description of intrusion models. Some of the traffic
captured in this dataset includes SMSTP, SSH, FTP, and
HTML. This dataset has many characteristics: Realistic
network and traffic, Labeled dataset, total interaction
capture, complete capture, and diverse intrusion
scenarios [35]. ISCX IDS 2012 dataset includes entire
packet payloads in pcap format and pertinent profiles,
which are freely available for researchers.
VI.2. KDD CUP 99
VI.6. CIC-DDoS2019
A public dataset known as KDD 99 datasets is made
accessible to researchers in the area ofIDS. This dataset
is accessible solely because it is based on DARPA 98
dataset [31]. The KDD Cup has 41 attributes assigned to
each connection instance (38 numerical and three
symbolic attributes). Every incident is tagged with either
"normal" or "abnormal," which can be one of 4 attack
types. That attack falls into one of the four categories:
DoS, Probe, R2L, and U2R. The IDS research
community still uses KDD99 as a standard, as there is a
lack of datasets available to the public which can be
freely accessible.
CICDDoS2019 [36] is a new dataset that overcomes
all of the present limitations. This dataset is used for the
evaluation of IDS algorithms and systems on DDoS
attacks.
Using this dataset, they proposed a new detection and
classification approach based on a set of network flow
features. In addition, they include the important features
required to identify various forms of Network threats
with their corresponding weights.The CIC-DDoS2019
dataset has 11 DDoS attacks collected over a real-time
network, including MSSQL, UDP, NTP, DNS, and
SNMP [37].
R
EP
The Defense Advanced Research Project Agency
(DARPA) was one of the earliest IDS datasets. The
DARPA dataset had two parts, i.e., a real-time evaluation
and an offline evaluation.
By modelling a US Air force based on restricted
personnel, data was collected using several computers
connected to the internet. The kind of data collected was
host log files or network data packets.
The collected data had 41 attributes that were
characterized as either normal or abnormal. The collected
data is a number of TCP sessions where data flows to a
specified address with several simulated attacks [30].
Copyright © 2022 Praise Worthy Prize S.r.l. - All rights reserved
International Journal on Engineering Applications, Vol. 10, N. 5
340
Wasim A. Ali et al.
VI.7. CIC-IDS-2017 & CSE-CIC-IDS-2018
-
The CIC-IDS-2017 [38] and CSE-CIC-IDS-2018 [39]
datasets are created with crucial criteria in mind, such as
attack variety, access protocols, feature set, labelled data
samples, whole network traffic capture, heterogeneity,
anonymity, and metadata. These datasets were generated
in a well-ordered manner using the profile concept.
These datasets include in-depth information on attacks
and a conceptual understanding of various application
models, protocols, and devices on the network. The
attacks contained in the CIC-2017 attacks have a similar
sort of signature to that of the PCAP. Multiple research
has suggested the use of a CIC-2017 intrusion detection
model, which utilizes ML techniques. The latest CIC2018 dataset includes the most current malware detection
methods. The data used to produce CIC-2018 was not
based on KDD and consisted of 7 attack types labelled
data, including brute-force, denial of service(DoS), and
Botnet. Finally, CIC-2018 includes extensive network
traffic and system logs [40].
Limitations and Challenges
T
-
-
-
Data Set
DARPA
Developed By
MIT Laboratory
University of California
R
KDD CUP 99
EP
R
It is not easy to define or provide proof of any
suggested IDS completeness and correctness without
evaluating the model on the current datasets. One of the
study results showed that it is exceedingly difficult to
create an accurate, scalable, robust, and protective IDS.
In the future, researchers will encounter several
limitations and obstacles in using datasets,as listed
below:
- Openly available datasets provide a small percentage
of attacks that are known. According to many
researchers, this is one of the most significant barriers
-
IN
VII.
-
to IDS implementation in the real world;
Zero-Day Attacks: Although attack speeds are
increasing, it's not keeping pace with data collection.
Additional methods of dataset production are
required, and these datasets no longer detect zero-day
attacks. The faster we can produce datasets and make
them available to the public, the quicker we can keep
up with new IDS model versions;
Only a restricted number of datasets are available
when selecting the appropriate data for a specialpurpose IDS. For example, SCADA, IoT, and Tor
network datasets are currently lacking;
A fresh malware dataset is required since most of the
existing approaches are trained and evaluated on the
assumption that the current datasets don't include new
malicious activity;
The absence of documentation linked with newly
available datasets is a significant problem for
researchers developing outdated data sets.
Additionally, it is also necessary to share raw packet
data instead of only calculated characteristics to
utilize datasets better;
Privacy is cited as a significant challenge to
collecting new attack data. This point concerns how
researchers are required to employ their custom
techniques to perform tasks related to data collection,
documentation, anonymization, and publication
because of the absence of standard tools;
Due to the lack of comprehensive data
documentation, it is more difficult to map one to
another, making it unfeasible to add new attacks to
current datasets. Additionally, even if a standard way
for exporting realistic traffic exists, there is
insufficient information about inserting freshly
gathered data into old ones.
NSL-KDD
University of California
UNSW-NB15
Lab of UNSW Canberra
ISCX2012
University of New
Brunswick
CIC-DDoS2019
University of New
Brunswick
CIC-IDS-2017 &
2018
University of New
Brunswick
TABLE III
A C OMPARISON OF THE VARIOUS D ATASETS
Features
Attack Type
Volume of data
Description
Absence of false-positive instances,
No represent of real network traffic,
41
Dos,U2R, R2L, Probe
Attack data patterns with differing
consistency.
Contains redundant and duplicate
41
Dos, U2R, R2L, Probe
5M points
data samples.
An optimized dataset of KDD CUP
41
Dos, U2R, R2L, Probe
150,000 points
99 but has a restricted number of
attack types.
The training set is 175,341 records,
and the testing set is 82,332 records,
49
DoS, worm, Backdoors Fuzzers
2M points
some issues like over-fitting,
dimensionality, and imbalance in the
dataset.
consist of network scenarios with
DoS, DDoS, Brute-force,
IP flows
2M flows
intrusive activities and labelled data
Infiltration
instances.
containing 12 different DDoS
DDoS attacks: LDAP, PortMap,
attacks, and a completely labelled
80
More than 10GB
UDP-Lag, NetBIOS,etc.
dataset can be executed using
TCP/UDP.
solve the issue of high-class
Brute Force FTP, Botnet
4.6 GB for 2017 & imbalance, have a wide range of
Heartbleed, Brute Force SSH,
80
more than 50GB for attack categories, have limitations in
Infiltration, DoS, Web Attack,
2018
data samples, and analyze files
Infiltration, and DDoS.
created by network flow.
Copyright © 2022 Praise Worthy Prize S.r.l. - All rights reserved
International Journal on Engineering Applications, Vol. 10, N. 5
341
Wasim A. Ali et al.
considering the network updates in real-time. We believe
this benchmark could fill the requirement of building
IDS with a realistic attack scenario on the network.
In the future, we will focus on examining the
updateddatasets' performance with various ML
techniques in the research area of securing Vehicular ad
hoc networks (VANET) for autonomous vehicles.
Discussion
References
Pawlicki, M., Choraś, M., &Kozik, R. (2020). Defending network
intrusion detection systems against adversarial evasion attacks.
Future Generation Computer Systems, 110, 148-154.
[2] Ferrag, M. A., Maglaras, L., Moschoyiannis, S., &Janicke, H.
(2020). Deep learning for cybersecurity intrusion detection:
Approaches, datasets, and comparative study. Journal of
Information Security and Applications, 50, 102419.
[3] Min, E., Long, J., Liu, Q., Cui, J., & Chen, W. (2018). TR-IDS:
Anomaly-based intrusion detection through text-convolutional
neural network and random forest. Security and Communication
Networks, 2018.
[4] Li, Z., Rios, A. L. G., Xu, G., &Trajković, L. (2019, May).
Machine learning techniques for classifying network anomalies
and intrusions. In 2019 IEEE international symposium on circuits
and systems (ISCAS) (pp. 1-5). IEEE.
[5] Tavallaee, M., Bagheri, E., Lu, W., &Ghorbani, A. A. (2009,
July). A detailed analysis of the KDD CUP 99 data set. In 2009
IEEE symposium on computational intelligence for security and
defense applications (pp. 1-6). IEEE.
[6] McHugh, J. (2000). Testing intrusion detection systems: a critique
of the 1998 and 1999 DARPA intrusion detection system
evaluations as performed by the lincoln laboratory. ACM
Transactions on Information and System Security (TISSEC), 3(4),
262-294.
[7] Thakkar, A., &Lohiya, R. (2020). A Review of the Advancement
in Intrusion Detection Datasets. Procedia Computer Science, 167,
636-645.
[8] Kovács, G., Sebestyen, G., &Hangan, A. (2019). Evaluation
metrics for anomaly detection algorithms in time-series. Acta
Universitatis Sapientiae, Informatica, 11(2), 113-130.
[9] Hindy, H., Brosset, D., Bayne, E., Seeam, A. K., Tachtatzis, C.,
Atkinson, R., &Bellekens, X. (2020). A taxonomy of network
threats and the effect of current datasets on intrusion detection
systems. IEEE Access, 8, 104650-104675.
[10] Scarfone, K., & Mell, P. (2007). Guide to intrusion detection and
prevention systems (idps). NIST special publication, 800(2007),
94.
[11] Ashoor, A. S., & Gore, S. (2011). Importance of intrusion
detection system (IDS). International Journal of Scientific and
Engineering Research, 2(1), 1-4.
[12] Liu, I. H., Lo, C. H., Liu, T. C., Li, J. S., Liu, C. G., & Li, C. F.
(2020). IDS Malicious Flow Classification. Journal of Robotics,
Networking and Artificial Life, 7(2), 103-106.
[13] Anderson, J. P. (1980). Computer security threat monitoring and
surveillance. Technical Report, James P. Anderson Company.
[14] Bhattacharyya, D. K., &Kalita, J. K. (2013). Network anomaly
detection: A machine learning perspective. CRC Press.
[15] Ali, W. A., Manasa, K. N., Aljunid, M., Bendechache, M.,
&Sandhya, P. (2020). A Review of Current Machine Learning
Approaches for Anomaly Detection in Network Traffic. Journal
of Telecommunications and the Digital Economy, 8(4), 64-95.
[16] Chakir, E. M., Moughit, M., &Khamlichi, Y. I. (2018). An
Effective Intrusion Detection Model Based On SVM With Feature
Selection And Parameters Optimization. Journal of Theoretical &
Applied Information Technology, 96(12).
[17] M. M. Min and Y. Y. Aung, An Analysis of K-means Algorithm
Based Network Intrusion Detection System. Advances in Science
Technology and Engineering Systems Journal, vol 3, pp 496-501.
[18] Thakare, Y. S., &Bagal, S. B. (2015). Performance evaluation of
K-means clustering algorithm with various distance metrics.
International Journal of Computer Applications, 110(11), 12-16.
T
[1]
EP
R
In a dynamic context, the behavioural patterns of
network cyber attacks change over time, necessitating the
update of the available datasets. This will assist in
developing a variety of network traffic scenarios and new
patterns of attack that are simple to learn and redefine
[41]. Moreover, selecting the right dataset is crucial.
Some datasets are generated by individual organizations
and labs for particular research purposes and are not
publicly available. In addition, the publicly available
datasets may contain records that do not fit current
technological requirements. These publicly available data
sets are also statistically insufficient; thus, the lack of an
ideal data set is an issue to be considered [42]. In this
review, we investigated the current datasets used to
evaluate the developed models in network IDS. We
observed that an optimum dataset should contain all
communication using various protocols, whether normal
or attacks. The study showed that The dataset should
include a wide variety of up-to-date malware and attack
categories that could help the new IDS models protect
the systems in an updated context. In addition, datasets
should contain sufficient documentation describing the
testing environment, attack system infrastructure, victim
system infrastructure, and attack scenarios. As a result of
this study, we discovered that CIC-IDS-2017 and CSECIC-IDS-2018 datasets are designed to consider these
factors. Moreover, these datasets contain seven attack
categories describing the current attack scenarios [43].
Therefore, the characteristics of CIC-IDS-2017/2018
attracted many developers and researchers to build their
IDS using these benchmark datasets [44]-[46].
IN
VIII.
IX. Conclusion
R
This study started with a detailed survey of IDS
techniques, types of network attacks, and current IDS
datasets with the aim of helping cyber security
researchers understand the nature of this dynamic
research field and the current datasets. The datasets
discussed above have been used as performance
evaluators for IDS. The study reports the need to update
the existing datasets to identify and regulate current
attacks with higher performances. Most of the data used
for testing purposes does not match real-world properties,
is unreliable, and will affect IDS performance. An IDS
gathers information and uses it to make observations on
patterns. Most of the current IDS systems can detect
anomalies and present outstanding results from the
evaluation but use outdated datasets. For this reason, the
results arenot representative of real-world systems and
attack scenarios. With the creation of new datasets, the
dataset creators should be able to adapt to extreme
network traffic systems and also to detect zero-day
attacks and classify them as vulnerabilities. We conclude
that the CIC-IDS-2017 and CSE-CIC-IDS-2018 datasets
are built and designed to overcome all limitations stated
in the discussion and could enhance the IDS performance
Copyright © 2022 Praise Worthy Prize S.r.l. - All rights reserved
International Journal on Engineering Applications, Vol. 10, N. 5
342
Wasim A. Ali et al.
IN
T
Multitopic Conference (INMIC) (pp. 1-6). IEEE.
[38] Intrusion Detection Evaluation Dataset (CICIDS2017).
https://www.unb.ca/cic/datasets/ids2017.html
[39] CSE-CIC-IDS2018 on AWS.
https://www.unb.ca /cic/datasets/ids-2018.html
[40] Kim, J., Shin, Y., & Choi, E. (2019). An intrusion detection
model based on a convolutional neural network. Journal of
Multimedia Information System, 6(4), 165-172.
[41] Sharafaldin, I., Gharib, A., Lashkari, A. H., &Ghorbani, A. A.
(2018). Towards a reliable intrusion detection benchmark dataset.
Software Networking, 2018(1), 177-200.
[42] Koch, R., Golling, M., &Rodosek, G. D. (2014, May). Towards
comparability of intrusion detection systems: New data sets. In
TERENA Networking Conference (Vol. 7).
[43] Sharafaldin, I., Lashkari, A. H., &Ghorbani, A. A. (2018).
Toward generating a new intrusion detection dataset and intrusion
traffic characterization. ICISS, p, 1, 108-116.
[44] Vijayanand, R., Devaraj, D., &Kannapiran, B. (2018). Intrusion
detection system for wireless mesh network using multiple
support vector machine classifiers with genetic-algorithm-based
feature selection. Computers & Security, 77, 304-314.
[45] Ullah, S., Khan, M. A., Ahmad, J., Jamal, S. S., e Huma, Z.,
Hassan, M. T., ... & Buchanan, W. J. (2022). HDL-IDS: a hybrid
deep learning architecture for intrusion detection in the Internet of
Vehicles. Sensors, 22(4), 1340.
[46] Seth, S., Chahal, K. K., & Singh, G. (2021). A novel ensemble
framework for an intelligent intrusion detection system. IEEE
Access, 9, 138451-138467.
Authors’ information
1Department of Electrical and Information Engineering, Politecnico di
Bari, Bari, Italy
2Department
of Computer Science and Engineering, PG Centre,
Visvesvaraya Technological University, Mysore, Karnataka, India.
R
[19] Kherbache, M., Espes, D., &Amroun, K. (2021, July). An
Enhanced approach of the K-means clustering for Anomaly-based
intrusion detection systems. In 2021 International Conference on
Computing, Computational Modelling and Applications (ICCMA)
(pp. 78-83). IEEE.
[20] Kamal Idrissi, H., Kartit, Z., Kartit, A., El Marraki, M., CKMSA:
an Anomaly Detection Process Based on K-Means and Simulated
Annealing Algorithms, (2016) International Review on
Computers and Software (IRECOS), 11 (1), pp. 42-48.
doi: https://doi.org/10.15866/irecos.v11i1.8272
[21] E. Besharati, M. Naderan, and E. Namjoo, LR-HIDS: logistic
regression host-based intrusion detection system for cloud
environments, Journal of Ambient Intelligence and Humanized
Computing volume, vol. 10, p. 3669–3692, 2019.
[22] Shah, R. A., Qian, Y., Kumar, D., Ali, M., &Alvi, M. B. (2017).
Network intrusion detection through discriminative feature
selection by using sparse logistic regression. Future Internet, 9(4),
81.
[23] Meftah, S., Rachidi, T., &Assem, N. (2019). Network based
intrusion detection using the UNSW-NB15 dataset. International
Journal of Computing and Digital Systems, 8(5), 478-487.
[24] B. S. Sharmila and R. Nagapadma, Intrusion Detection System
using Naive Bayes algorithm. In IEEE International WIE
Conference on Electrical and Computer Engineering (WIECONECE), 2019.
[25] Mehmood, A., Mukherjee, M., Ahmed, S. H., Song, H., & Malik,
K. M. (2018). NBC-MAIDS: Naïve Bayesian classification
technique in multi-agent system-enriched IDS for securing IoT
against DDoS attacks. The Journal of Supercomputing, 74(10),
5156-5170.
[26] Wu, K., Chen, Z., & Li, W. (2018). A novel intrusion detection
model for a massive network using convolutional neural
networks. IEEE Access, 6, 50850-50859
[27] Shatnawi, M., Bani Yassein, M., Aljawarneh, S., Alodibat, S.,
Meqdadi, O., Hmeidi, I., Al Zoubi, O., An Improvement of Neural
Network Algorithm for Anomaly Intrusion Detection System,
(2020) International Journal on Communications Antenna and
Propagation (IRECAP), 10 (2), pp. 84-93.
doi: https://doi.org/10.15866/irecap.v10i2.18735
[28] Nguyen, M. T., & Kim, K. (2020). Genetic convolutional neural
network for intrusion detection systems. Future Generation
Computer Systems, 113, 418-427.
[29] Paffenroth, R., Kay, K., &Servi, L. (2018). Robust PCA for
anomaly detection in cyber networks. arXiv preprint
arXiv:1801.01571.
[30] A. Khraisat, I. Gondal, P. Vamplewand J. Kamruzzaman, Survey
of intrusion detection systems: techniques, datasets, and
challenges, Cybersecurity, vol. 20, 2019.
[31] Brugger, T. (2007). KDD Cup’99 dataset (Network Intrusion)
considered harmful. KDnuggets newsletter, 7(18), 15.
[32] M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, A detailed
analysis of the KDD CUP 99 data set, in IEEE Symposium on
Computational Intelligence for Security and Defense
Applications, 2009, pp. 1–6.
[33] Moustafa, N., & Slay, J. (2015, November). UNSW-NB15: a
comprehensive data set for network intrusion detection systems
(UNSW-NB15 network data set). In 2015 military
communications and information systems conference (MilCIS)
(pp. 1-6). IEEE.
[34] S. M. Kasongo and Y. Sun, Performance Analysis of Intrusion
Detection Systems Using a Feature Selection Method on the
UNSW-NB15 Dataset, Journal of Big Data, p. 105, 2020.
[35] Shiravi, A., Shiravi, H., Tavallaee, M., &Ghorbani, A. A. (2012).
Toward developing a systematic approach to generate benchmark
datasets for intrusion detection. Computers & Security, 31(3),
357-374.
[36] Sharafaldin, I., Lashkari, A. H., Hakak, S., &Ghorbani, A. A.
(2019, October). Developing realistic distributed denial of service
(DDoS) attack dataset and taxonomy. In 2019 International
Carnahan Conference on Security Technology (ICCST) (pp. 1-8).
IEEE.
[37] Hussain, F., Abbas, S. G., Husnain, M., Fayyaz, U. U., Shahzad,
F., & Shah, G. A. (2020, November). IoT DoS and DDoS attack
detection using ResNet. In 2020 IEEE 23rd International
EP
Wasim A. Ali received his B.E. Information
technology from University of science and
technology, Taiz,Yemen (2014). Obtained
master's degree in Computer Science from St.
Philomena's College, University ofMysore, India
(2017). Currently, pursuing a PhD in Electrical
and Information Engineering at Politecnico di
Bari, Italy. His research interests include Big
data analytics, IoT application, and Cybersecurity. working with the
Public Corporation of Telecommunication(PTC), Yemen, since 2003.
Mr. Wasim is amember of IEEE.
E-mail: wasim.ali@poliba.it
R
Dr. P. Sandhya is an associate Professor,
Department of Computer Applications, Centre
for
Postgraduate
Studies,
Visvesvaraya
technological University, Mysore, India. She
completed her post graduation in computer
applications from Visveswaraiah Technological
University, Belgaum, Karnataka, India and
Recieved PhD degree in Computer Applications
from Dr. M.G.R. Educational and Research Institute, India (2011). Her
Research areas of Biomedical Data mining, CDSS, Soft Computing,
Neural networks, and Management of Neurodegenerative disorders.
She has published many research publications in International Journals,
and International and National Conferences. Dr. Sandhya has a
professional Memberships in CSI( Life member), IEEE, and ARDSI.
E-mail: sanjoshi17@yahoo.com
Copyright © 2022 Praise Worthy Prize S.r.l. - All rights reserved
International Journal on Engineering Applications, Vol. 10, N. 5
343
Wasim A. Ali et al.
IN
R
EP
R
Maria Pia Fanti has been with the Department
of Electrical and Information Engineering of the
Polytechnic of Bari, Italy since 1983, and she is
currently a Full Professor of system and control
engineering and Chair of the Laboratory of
Automation and Control. Her research interests
include management and modeling of complex
systems, such as transportation, logistics and
manufacturing systems; discrete event systems; Petri nets; consensus
protocols. Prof. Fanti has published more than 315 papers and two
textbooks on her research topics. She was senior editor of the IEEE
Trans. on ASE and she is Associate Editor of the IEEE Trans. on SMC:
Systems. She is member of the AdCom of the IEEE RAS, and chair of
the Technical Committee on Automation in Logistics of the IEEE RAS.
E-mail: mariapia.fanti@poliba.it
T
Michele Roccotelli received the M.Sc.Eng.
degree in Automation Engineering from the
Polytechnic of Bari in 2010. He received the
Ph.D. in Electrical and Information Engineering
from the Polytechnic of Bari, Italy (Apr. 2016).
He has been a visiting scholar (Jan. 2015-Jul.
2015) at the École Normale Supérieure de
Cachan, Paris, France. He is currently an
Assistant professor at Polytechnic of Bari, at the Department of
Electrical and Information Engineering. His research interests include
energy efficiency systems, models and strategies for smart grids and
smart buildings, electric mobility systems and services. Dr Roccotelli
has been member of the program committees of several international
conferences such as IEEE SOLI 2017, IEEE CoDIT 2019-2022, IFAC
Wodes 2020 and in the organizing committee of the IEEE SMC 2019
and he is reviewer of IEEE, Wiley and MDPI journals. He has
coauthored more than 40 publications in international conference and
journals and one editorial.
E-mail: michele.roccotelli@poliba.it
Copyright © 2022 Praise Worthy Prize S.r.l. - All rights reserved
International Journal on Engineering Applications, Vol. 10, N. 5
344