Cyber Security: Evaluating the Effects of Attack Strategy and Base Rate through
Instance-Based Learning
Aman Arora (aman@students.iitmandi.ac.in)
School of Computing and Electrical Engineering
Indian Institute of Technology, Mandi, India
Varun Dutt (varun@iitmandi.ac.in)
School of Computing and Electrical Engineering
School of Humanities and Social Sciences
Indian Institute of Technology, Mandi, India
cyber-world like ours, we must have defenses to keep us
safe. In order to create appropriate defenses against cyber
attacks, the role security analysts, a human decision maker
whose main role is to protect computer networks from cyber
attacks, is becoming indispensable (Jajodia, Liu, Swarup, &
Wang, 2010). Given the growing demand, companies have
started employing security analyst. However, currently not
much emphasis has been laid on how different
environmental factors (e.g., attack strategy and base rate)
would affect the analyst’s accurate and timely detection of
cyber attacks.
One of the researches done in this field indicates that the
analyst’s accurate and timely threat detection is likely to be
a function of prior experiences with cyber attacks and
tolerance to perceived threats (Dutt et al., in press;
McCumber, 2004; Salter, Saydjari, Schneier, & Wallner,
1998). Dutt et al. (in press) have given primary predictions
about a simulated analyst’s performance according to her
experience and tolerance to threats (or risk-taking). These
authors created a cognitive model of an analyst’s decisions,
based upon Instance-Based Learning Theory (IBLT;
Gonzalez & Dutt, 2011), and populated the model’s
memory with cyber threat and cyber non-threat experiences.
The model’s tolerance was determined by the number of
events it perceived as threats before it declared the sequence
of network events to be a cyber attack. Accordingly, a
model with a greater proportion of threat experiences was
more accurate and timely in detecting threats compared with
one with a smaller proportion of such experiences; whereas,
the tolerance did not influence model’s accuracy and
timeliness.
Although Dutt et al. (in press) have highlighted the role of
prior experience and risk-taking on cyber threat detection, it
is unclear how their results would vary due to the nature of
adversarial behaviors (an external factor). Thus, here we
simulate an analyst’s decision process through the
computational model developed by Dutt et al. (in press) and
derive predictions about analyst’s decisions in scenarios that
differ in adversarial behaviors. One characteristic of
adversary’s behavior is the attack strategy (patient or
impatient) being followed to infiltrate the network. An
impatient strategy is the strategy in which attacker passes all
threat events in the beginning of a sequence of network
events; whereas, a patient strategy will make attacker to
Abstract
Cyber attacks, the disruption of normal operations in a
computer network due to malicious events called cyber
threats, are becoming widespread. In order to check the
prevalence of cyber attacks the role of security analysts,
human decision makers whose job is to prevent cyber attacks,
is becoming extremely important. However, currently very
little is known on how security analysts might respond to
different attack strategies of an attacker in cyber attacks. Also,
little is known on how the proportion of threats (i.e., base
rate) in an attack scenario influences the analyst’s timely and
accurate detection of such attacks. In this paper, we use an
existing cognitive model of the security analyst, based upon
Instance-Based Learning Theory, and we evaluate the effects
of attack strategy and base rate on the model’s accurate and
timely detection of cyber-attacks in a simulated scenario. The
attack strategy was manipulated as: impatient (attacker injects
all threats in the beginning of the scenario) and patient
(attacker waits till the end of the scenario to inject threats);
and, base rate was manipulated as: common (13 out of 25
scenario events (52%) were threats) and rare (3 out of 25
scenario events (12%) were threats). Results reveal that the
attack strategy influences only the analyst’s accuracy and not
her timeliness; however, the base rate influences her
timeliness and not the accuracy. We discuss the implications
of our results for training analyst in their job.
Keywords: — cyber attacks; base rate; attack strategy;
Instance-based Learning Theory; accuracy; timeliness.
Introduction
According to the U.S. White House, “Cyberspace touches
nearly every part of our daily lives. It is the broadband
networks beneath us and the wireless signals around us, the
local networks in schools, hospitals, work-places and
business...Today’s world is more interconnected than ever
before.” These lines explicitly describe the inevitableness of
Internet. Yet, for all its advantages, increased connectivity
brings increased risk of theft, fraud, and abuse. As we are
becoming more reliant on modern technology, we are also
becoming more vulnerable to cyber attacks such as
Corporate Security Breaches, Spear Phishing, and Social
Media Fraud. With the prevalence of “Anonymous” and
“LulzSec” hacking groups and other threats to corporate and
national security, guarding against cyber attacks is
becoming a significant part of IT governance, especially
because most government agencies and private companies
have moved to online systems (Sideman, 2011). Thus, in a
336
wait till the end to inject the threats in a sequence of
network events. We also consider proportion of threat
events (i.e., base rate; common or rare) that an attacker uses
to attack a network. A common base rate is when the
attacker uses a large number of threat events in a cyber
attack; however, a rare base rate is when the analyst uses
only a smaller number of threat events in a cyber attack.
Next, a cyber-infrastructure and an IBL model of
analyst’s cyber threat detection are detailed. Then, we
describe the manipulations in the attack scenarios and detail
the results of our manipulations. We close this paper by
presenting a discussion of our findings and their
implications to training human analysts in their job.
Cyber Infrastructure
A corporate network may consist of different types of
servers and multiple layers of firewalls. We used a
simplified network configuration consisting of a web server,
a fileserver, and two firewalls (Ou, Boyer, & McQueen,
2006; Xie, Li, Ou, & Levy, 2010). An external firewall
(‘firewall 1’ in Figure 1) controls the flow of traffic between
the Internet and the Demilitarized zone (DMZ; a subnetwork that separates the Internet from the company’s
internal LAN network). Second firewall (‘firewall 2’ in
Figure 1) controls the traffic between the webServer and the
fileServer, a company’s internal LAN network. The
webServer interacts with outside world and it is behind the
DMZ. FileServer contains necessary information that
internal users (employees) use to do their daily operations.
FileServer is connected to workstations, which employees
use as a medium to interact with the fileServer and are
authorized to run executable files resting on the fileServer.
Generally, an attacker is identified as a computer on the
Internet that is trying to gain access to the internal corporate
servers and workStations. For this cyber-infrastructure,
attackers follow a pattern of “island-hopping” attack
(Jajodia et al., 2010; pp. 30), where the webServer is
compromised first, and then it is used to originate attacks on
the fileServer and other company workStations.
An attacker, in order to gain access to fileserver, will
need to pass at least 3 events (to get to webServer,
fileserver, and then to execute binaries inside fileServer).
Because of this reason, we define the rare base rate to
contain 3 threats out of total 25 network events (as it is
minimum an attacker may require).
The model discussed in this paper is presented with
different cyber events in a sequence, where these events
follow an attacker’s strategy, i.e., the sequence of events are
a combination of a base rate and timing strategy. Attack
sequences may be classified as rare-impatient, rare-patient,
common-impatient and common-patient depending upon the
combination of attack strategies being followed by attacker.
Rare-impatient attack sequence contains 3 out of total 25
network events as threats and are all being injected in the
beginning. Rare-patient also contains 3 threats but here
these are being injected in the end of sequence of 25
network events.
Figure 1: An Example Attack Scenario.
In common-impatient attack sequence there are 13
threats out of total 25 network events and these are being
injected with impatient strategy, i.e., all of them are being
injected in the beginning of the sequence; however, in
common-patient attack sequence these 13 threats are being
injected at the end of sequence. Nature of these attack
sequences is not known to the model; however, the model is
able to get alerts corresponding to some network events
(that are regarded as threats) generated from the intrusiondetection system (IDS) (Jajodia et al., 2010). Out of 25
events, some are threats that are initiated by an attacker (the
rest of the events are initiated by inside users). The model
does not know which events are generated by the attacker
and which are generated by corporate employees. By
perceiving network events in a sequence as threats or nonthreats, the model needs to identify, as early and accurately
as possible, whether the sequence constitutes a cyber attack.
Instance-Based Learning Theory (IBLT)
IBLT is a theory of how people make decisions from
experience in dynamic environments (Gonzalez & Dutt,
2011). Computational models based on IBLT have been
shown to generate accurate predictions of human behavior
in many dynamic decision-making situations similar to
those faced by analysts (Dutt et al., in press; Dutt &
Gonzalez, 2012; Gonzalez & Dutt, 2011; Gonzalez et al.,
2011). IBLT proposes that every decision situation is
represented as an experience called an instance that is stored
in memory. Each instance in memory is composed of two
parts: situation (S) (the knowledge of attributes that describe
an event), a Decision (D) (the action taken in such
situation), and utility (U) (a measure of expected result of a
decision that is to be made for an event). For a situation
337
involving securing a network from threats, the situation
attributes are those that can discriminate between threat and
non-threat events: the IP address of a computer (web-server,
file-server, or workstation, etc.) where the event occurred,
the directory location in which the event occurred, whether
the IDS raised an alert corresponding to the event, and
whether the operation carried out as part of the event(e.g., a
file execution) by a user of the network (which could be an
attacker) succeeded or failed. In the IBL model of a analyst,
an instance’s S part refers to the situation attributes defined
above; and the U slot refers to the expectation in memory
that a network event is a threat or not. For example, an
instance could be defined as [webserver, c:\, malicious code,
success; threat], where “webserver,” “c:\,” “malicious
code,” and “success” constitutes the instance’s S part; and
“threat” is the instance’s U part (the decision being binary:
threat or not, is not included in this model).
An instance is retrieved in the recognition phase from
memory according to an activation mechanism (Gonzalez et
al., 2003; Lejarraga et al., in press). The activation of an
instance i in memory is defined using a simplified version of
ACT-R’s activation equation:
Directory
Alert
Operation
∑
is the similarity component and represents
the mismatch between a situation's attributes and the
situation (S) part of an instance i in memory. The k is the
total number of attributes for a situation event that are used
to retrieve the instance i from memory. The value of k=4 as
there are 4 attributes (IP, directory, alert, and operation)
that characterize a situation in the network. The match scale
( ) reflects the amount of weighting given to the similarity
between an instance i’s situation part l and the
corresponding situation event’s attribute.
is generally a
negative integer with a common value of -1.0 for all
situation slots k of an instance i, and we assume this value
for the . The
or match similarities represents the
similarity between the value l of a situation event’s attribute
and the value in the corresponding situation part of an
instance i in memory. Typically,
is defined using a
squared distance between the situation event’s attributes and
the corresponding instance's situation slots (Shepard, 1962).
Thus,
is equal to the sum of squared differences
between a situation event’s attributes and the corresponding
instance's S part. In order to find the sum of these squared
differences, the situation events’ attributes and the values in
the corresponding S part of instances in memory were coded
using numeric codes. Table 1 shows the codes assigned to
the S part of instances and the situation events’ attributes.
The noise value (Anderson & Lebiere, 1998; Gonzalez
& Dutt, 2011) is defined as
The Bi equation is given by:
∑
{
)
}
In this equation, the frequency effect is provided by
,
the number of retrievals of the ith instance from memory in
the past. The recency effect is provided by
, the time
since the th past retrieval of the ith instance (in equation 2, t
denotes the current event number in the scenario). The d is
the decay parameter and has a default value of 0.5 in the
ACT-R architecture, and this is the value we assume for the
IBL model.
refers to the similarity between the attributes of the
situation and the attributes of the ith instance.
is defined
as,
(
Values
Codes
IP
Webserver
1
)
where,
is a random draw from a uniform distribution
bounded in [0, 1] for an instance i in memory. We set the
parameter s in an IBL model to make it a part of the
activation equation (equation 1). The s parameter has a
default value of 0.25 in the ACT-R architecture, and we
assume this default value in the IBL model.
We used IBLT to study the accurate and timely detection
of threats by cyber analysts because IBLT is known to make
better decisions compared to other models and techniques.
Basava, Ramakrishna and Varun in there paper “Cyber
Situation Awareness: Rational Methods versus InstanceBased Learning Theory for Cyber Threat Detection; ICCM”
Table 1: The coded values in the S part of instances in
memory and attributes of a situation event.
Attributes
2
3
-1001
1
1
0
1
0
∑
where i refers to the ith instance that is pre-populated in
memory, and i = 1,2, … constitutes the total number of prepopulated instances in memory; Bi is the base-level learning
mechanism and reflects both the recency and frequency of
use for the ith instance since the time it was created; and
is the noise value that is computed and added to an instance
i’s activation at the time of its retrieval attempt from
memory.
(
Fileserver
Workstation
Missing value
File X
Present
Absent
Successful
Unsuccessful
338
in this conference compare IBLT with Naïve Bayes
classifier which is a rational approach to make decisions.
There results depict that Naïve Bayes approach is poor in
terms of timeliness and accuracy as compared to the IBLT.
Hence, we ran our experiments with IBLT to study the
interaction of base rate and attack strategy in timely and
accurate detection of cyber attacks.
Accuracy
As shown in Figure 2, it was seen that d’ corresponding to
common base rate (-0.026) was greater than that for the rare
base rate (-0.75). So, the base rate did influence the model’s
accuracy (d’). Furthermore, as shown in Figure 3, the
accuracy for the impatient strategy (d’ = 3.48) was greater
than that for the patient strategy (d’ = -4.25). Thus, attack
strategy also influenced the accuracy.
However, the base rate did not play as big a role as that
played by strategy (see Figure 4). Strategy was found to
influence accuracy and its effect was irrespective of the base
rate. From Figure 4, an impatient strategy had a greater d’
(common-impatient = 4.23, rare-impatient = 2.72) compared
to a patient strategy (common-patient = -4.28, rare-patient =
-4.23).
Experiments
The IBL model used here has been taken from Dutt et al.
(in press). This model is presented with sequences of
network events that represent four strategies (commonpatient, common-impatient, rare-patient, rare-impatient) of
the attacker with tolerance fixed at 50% of base rate. All
sequences contained 25 network events. Model’s memory
was pre-populated with instances that represent analysts
with different experiences and the model was fixed to use
tolerance level of 0.5. The IBL model retrieved instance
with highest activation and made a decision about an event
being a threat or a non-threat.
We use only 500 simulations of the model as they were
sufficient for generating stable model results (Dutt et al., in
press). We ran 500 simulations (each simulation consisting
of 25 network events) and the model’s effectiveness was
evaluated using its accuracy and detection timing in four
groups defined by: strategy (patient and impatient) and base
rate (common and rare). Accuracy was determined by
computing the d’(Z(hitrate) – Z(false-alarmrate)), hitrate
(hits/(hits + misses)), and false-alarm rate = (falsealarms/(false-alarms + correct-rejections)) (Wickens, 2001)
over the course of 25 network events and averaged across
the 500 simulations. The decision of the model for each
network event was marked as a hit if an instance with its U
slot indicated a threat for an actual threat event in the
sequence. Similarly, the model’s decision was marked as a
false-alarm if an instance with its U slot indicated a threat
for an actual non-threat event in the sequence. Hits and
false-alarms were calculated for all events before model
declared a cyber attack and stopped, or when all the 25
events had occurred (whichever came first).
Furthermore, detection timing was calculated in each
simulation as the “proportion of attack steps,” defined as
100% - the percentage of threat events out of a total 25 that
have occurred after which the model classifies the event
sequence as a cyber attack and stops. Therefore, higher
percentages of attacks steps would indicate the model to be
timelier in detecting cyber attacks.
0.0
-0.1
-0.026 d'
-0.2
-0.3
Common
-0.4
Rare
-0.5
-0.6
-0.7
-0.8
-0.75
Figure 2: Accuracy(d’) as affected by base rate.
3.47
4.0
2.0
0.0
Impatient
d'
Patient
-2.0
-4.0
-4.26
-6.0
Figure 3: Accuracy(d’) as affected by strategy.
Results
We will be explaining the results obtained with the help of
figures presented in this paper from Figure 2 to Figure 7.
Each of these figures is a histogram where y axis
corresponds to accuracy or timeliness as mentioned across
respective figure and the bars represents attack scenario,
those can be decoded by the legend in each figure.
339
Common Patient
6
4.22
Common Impatient
4
2.72
2
Rare Patient
Rare Impatient
0
d'
-2
-4
-4.23
-4.28
-6
42
45
40
35
30
25
20
15
10
5
0
41
Impatient
Patient
Timeliness (%)
Figure 4: Accuracy(d’) as affected by different
combinations of base rate and strategy
Figure 6: Timeliness as affected by strategy.
60
Timeliness
After running the model it was observed that the
timeliness (Figure 5) for the rare scenario (36%) was higher
than that for the common scenario (47%). Thus, timeliness
was influenced by base rate. However, strategy did not
influence timeliness, as can be seen from Figure 6 both
strategies have almost same timeliness (impatient strategy
with timeliness of 42% and patient strategy with timeliness
of 41%). This means that strategy plays no role in the time it
takes to determine if there is an attack. Interaction of base
rate and strategy in determining the timeliness was
evaluated in Figure 7. Timeliness for common scenarios
(common-patient = 36%, common-impatient = 35%) were
lower than that in the rare scenarios (rare-patient = 46%,
rare-impatient = 49%). Also, timeliness did not vary much
for both strategies in both scenarios. Thus, there is only an
influence base rate on timeliness and not of strategy on
timeliness: The rare base rate caused model to be timelier.
Thus, to conclude, timeliness was affected by base rate.
40
Common Impatient
Rare Patient
Rare Impatient
10
0
Timeliness (%)
Figure 7: Timeliness as affected by different combinations
of base rate and strategy.
Discussion and Conclusions
Common
20
35
Common Patient
20
36
30
36
49
30
47
50
40
46
50
Rare
10
0
Timeliness (%)
Figure 5: Timeliness as affected by base rate
340
In this paper, we studied the effects of base rate and attack
strategy on the model’s accurate and timely detection of
cyber-attacks. Such an analysis is important as unlike the
cognitive factors (recency and tolerance) that are under the
direct control of analyst, the environmental factors (attack
strategy and base rate) are controlled by the attacker and
outside the direct control of the analyst. We find that both
these environmental factors, being outside the control of the
analyst, influence the analyst’s accuracy or timeliness.
First, the analyst’s accuracy was influenced by the strategy.
The model was more accurate when the strategy was
impatient compared to when it was patient. The likely
reason for this result is that an impatient strategy’s early
threats increase the activation of threat instances in the
model’s memory early on. Therefore, the increase in
activation is likely to make the model perform more
accurately against an impatient strategy compared to a
patient strategy.
Second, the timeliness was influenced by the base rate:
The model was timelier for the rare base rate compared to
the common base rate. This observation can be explained
based upon the definition of base rate, i.e., the proportion of
threats being passed among the 25 network events. If the
Jajodia, S., Liu, P., Swarup, V., & Wang, C. (2010). Cyber situational
awareness. New York, NY: Springer.
McCumber, J. (2004). Assessing and managing security risk in IT systems: A
structured methodology. Boca Raton, FL: Auerbach Publications.
Ou, X., Boyer, W. F., & McQueen, M. A. (2006). A scalable approach to attack
graph generation. In Proceedings of the 13th ACM Conference on
Computer and Communications Security (pp. 336-345). Alexandria,
VA: ACM. doi: 10.1145/1180405.1180446
Salter, C., Saydjari, O., Schneier, B., & Wallner, J. (1998). Toward a secure
system engineering methodology. In Proceedings of New Security
Paradigms Workshop (pp. 2-10). Charlottesville, VA: ACM. doi:
10.1145/310889.310900
Shepard, R .N. (1962). The analysis of proximities: multidimensional scaling
with an unknown distance function. Part I. Psychometrika, 27, 125–
140.
Sideman, A. (2011). Agencies must determine computer security teams in face
of
potential
federal
shutdown.
Retrieved
from
http://fcw.com/articles/2011/02/23/agencies-must-determinecomputer-security-teams-in-face-of-shutdown.aspx
Wickens, T. D. (2001). Elementary signal detection theory. New York, NY:
Oxford University Press, USA.
Xie, P., Li, J. H., Ou X., Liu, P., & Levy, R. (2010). Using Bayesian networks
for cyber security analysis. In Proceedings of the 2010 IEEE/IFIP
International Conference on Dependable Systems and Networks
(DSN) (pp. 211-220). Hong Kong, China: IEEE Press. doi:
10.1109/DSN.2010.5544924
proportion of threats is less (i.e., in rare scenario), it will
take less time for the model to declare a sequence of
network events as an attack compared to when the
proportion of threats is more (i.e., in the common scenario).
The activation of threat instances is likely increase faster for
the scenario with the rare base rate compared with the
scenario with the common base rate (because the instances
with the rare base rate are fewer in number and easy to
identify). This increase in activation in the rare scenario
would cause these activated instances to be retrieved from
memory often, causing the model to stop early.
Our results have important implications for training
analysts in their job. First, as both the base rate and strategy
influence the cyber threat detection, it is prudent to train
analysts on scenarios that differ in both these environmental
factors. Second, it is expected that analyst should be trained
in the common scenario and for a patient attack strategy as
in these cases the model’s performance was the poorest.
In these experiments, we overlooked the effects of
tolerance (risk-taking) by setting it up at 50% of base rate.
However, one expects that the model’s tolerance would
likely vary from one individual to another. More
specifically, the model’s ability would likely be influenced
by its risk-taking and varying the tolerance may have
significant effects on the model’s accurate and timely
detection of cyber-attacks. Thus, the next step in this
research would be to introduce tolerance as another
parameter and see its interaction with base rate and strategy.
We plan to undertake this idea as part of our ongoing
research on this topic.
Acknowledgements
We are very thankful to Dr. Varun Dutt, Assistant Professor,
Indian Institute of Technology, Mandi for guiding us in
difficult times on this project. Also, we are grateful to the
Indian Institute of Technology, Mandi for providing the
necessary capital and financial resources that made this
project possible in the first place. This research was also
partially supported by the Multidisciplinary University
Research Initiative Award on Cyber Situation Awareness
(MURI; #W911NF-09-1-0525) from Army Research Office
to Cleotilde Gonzalez, Carnegie Mellon University, USA.
References
Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought.
Hillsdale, NJ: Lawrence Erlbaum Associates.
Dutt, V., Ahn, Y. S., & Gonzalez, C. (in press). Cyber situation awareness:
modeling detection of cyber attacks with instance-based learning
theory. Human Factors.
Dutt, V., & Gonzalez, C. (2012). Cyber Situation Awareness through InstanceBased Learning: Modeling the Security Analyst in a Cyber-Attack
Scenario. In C. Onwubiko & T. Owens (Eds.), Situational
Awareness in Computer Network Defense: Principles, Methods and
Applications (pp. 125-140). Hershey, PA: IGI Global.
Gonzalez, C., & Dutt, V. (2011). Instance-based learning: Integrating decisions
from experience in sampling and repeated choice paradigms.
Psychological Review, 118(4), 523-551. doi: 10.1037/a0024558
Gonzalez, C., Dutt, V., & Lejarraga, T. (2011). A loser can be a winner:
Comparison of two instance-based learning models in a market entry
competition. Games, 2(1), 136-162. doi: 10.3390/g2010136
341