Computer-Aided Human Centric Cyber
Situation Awareness
M. McNeese, D. Hall, N. Giacobe,
V. Mancuso, D. Minotra, and E. McMillan
Objectives:
• Understand cognitive/contextual elements of
situation awareness in cyber-security domains
• Implement a systems perspective to research
linking real-world analysts with theory and human
in the loop experiments
• Utilize multi-modal research methodology
• Focus on the human and team elements within real
context applications
Scientific/Technical Approach
Living laboratory framework involving;
• Ethnographic studies
• Knowledge elicitation of domain experts
• Development of cognitive and process
frameworks and theories
• Implementation of a scaled world prototype
• Conduct of human in the loop experiments
• Analysis and transition to real-world
environments
Accomplishments
• Developed framework /process for studying SA in cyber security
•
•
•
•
via a Living Laboratory framework
Collected interview/survey data from practicing analysts
Implemented a simulation toolset for cyber SA to support human
in loop experiments
Conducted experiments in transactive memory, dynamic task
prioritization and visualization aids
Developed new SA evaluation metrics framework
Challenges
• Rapid evolution of cyber threats and threat environments
• Access to domain experts and state of the art practice
• Modeling adversarial threats – integration in simulation
Main Scientific/Technical Accomplishments
Motivation
•
Improvement in Cyber SA requires focus on the ultimate limited resource: the
human cyber analyst. This in turn requires understanding of the cognitive
processes, the context, limitations and issues associated with perception, cognition
and decision making for cyber SA
Summary of Accomplishments
•
•
•
•
•
We conducted three experiments on; i) transactive memory, ii) task prioritization
and iii) visual analytics in cyber-security and completed analysis efforts
We completed three PhD dissertations and one Master’s thesis
Developed new concepts for computer assisted SA using Complex Event Processing
and Coherence Net Processing
Designed a new, general visual analytics workbench for cyber SA
We disseminated our findings via peer-reviewed journal articles, conference
papers, edited book chapters, and presentations.
2
Task Statistics and Summary
Students supported:
- Four graduates/undergraduate students: Nicklaus Giacobe (50%), Vincent Mancuso (5 %), Dev Minotra ( 5 %),
Eric McMillan (5 %), Tristan Endsley (10 %) and Erin Johnson ( 50 %)
- Two faculty (D. Hall, M. McNeese) – Note: funding for all faculty provided by Penn State
- Degrees awarded: (MS, PhD): E. McMillan (M.S.), V. Mancuso (PhD), D. Minotra (PhD) N. Giacobe (PhD)
- Degrees in progress: E. Johnson (M.S.), Tristan Endsley (PhD)
Publications:
-
Refereed journal papers - 2
Conference papers – 3
Conference presentations - 3
Dissertations and Theses – 4
Technology Transitions:
• Interactions with industry
– Ethnographic studies/knowledge elicitation with network analysts working in education, military, government, and industry
domains.
– Briefings provided to several companies including: Deloitte, Lockheed Martin, Raytheon Corporation, GE, MITRE, Computer
Sciences Corporation, and MIT Lincoln Laboratory
• Interactions with other government agencies
– Briefings presented to representatives from the National Security Agency (NSA), Defense Threat Reduction Agency (DTRA),
Office of Naval Research (ONR), Department of Homeland Security (DHS), Department of Defense Intelligence Information
Systems (DoDIIS), and Air Force Research Laboratory – AFRL – 711 Performance Wing
3
Publications, Honors and Awards
Year 4
Peer-Reviewed Journals
• Tyworth, M., Giacobe, N.A., Mancuso, V.F., McNeese, M.D. and Hall, D.L. (2013). A Human-in-the-loop Approach to Understanding
Situation Awareness in Cyber Defense Analysis. ICST Transactions, 3 May 2013.
• Cooke, N. and M. McNeese (2013), preface to special issue on the cognitive science of cyber defense analysis, editorial in EAI
endorsed Transactions on Security and Safety, 13 (2), May 2013
• Tyworth, M., Giacobe, N.A., Mancuso, V.F., McNeese, M.D. and Hall, D.L. (2013). “A Human-in-the-loop Approach to Understanding
Situation Awareness in Cyber Defense Analysis”, research article in EAI Endorsed Transactions on Security and Safety. 13 (2) May
2013
Refereed Conference Proceedings
• Mancuso, V., McNeese, M., “Effects of Integrated and Differentiated Knowledge Structures on Distributed Team Cognition”. (2012),
Proceedings of the 56th annual Meeting of Human Factors and Ergonomics Society Annual Meeting, Boston, 2012
• Giacobe, Nicklaus A.; McNeese, Michael D.; Mancuso, Vincent F.; Minotra, Dev, "Capturing Human Cognition in Cyber-Security
Simulations with NETS," Intelligence and Security Informatics (ISI), 2013 IEEE International Conference on , vol., no., pp.284,288, 4-7
June 2013
• Giacobe, N.A., “A Picture is Worth A Thousand Alerts”, Proceedings of the 57th annual Meeting of Human Factors and Ergonomics
Society Annual Meeting, San Diego, 2013
Presentations
• Mancuso, V., McNeese, M., Effects of Integrated and Differentiated Knowledge Structures on Distributed Team Cognition. (2012),
Proceedings of the 56th Annual Meeting of Human Factors and Ergonomics Society Annual Meeting, Boston, 2012
• Giacobe, Nicklaus A.; McNeese, Michael D.; Mancuso, Vincent F.; Minotra, Dev, "Capturing Human Cognition in Cyber-Security
Simulations with NETS," Intelligence and Security Informatics (ISI), 2013 IEEE International Conference on Intelligence and Security
Informatics, Seattle, WA, pp 284-288, 4-7 June, 2013
• Mancuso, V. and M. McNeese (2013), “TeamNETS: Scaled World Simulation for Distributed Cyber Teams”, poster session at the
International Conference on Human Computing Interaction, HCII 2013, July 21 – 26, 2013
• McNeese, M. , Reset – Alternative Visions for Cyber Worlds, “Cognition and Cyber-Security” panel presentation, 57th annual
Meeting of Human Factors and Ergonomics Society Annual Meeting, San Diego, 2013
4
Publications, Honors and Awards
Year 4
Dissertations and Theses
• N. A. Giacobe (2013), Measuring the Effectiveness of Visual Analytics and Data Fusion Techniques on Situation Awareness in Cybersecurity, Ph.D. dissertation for the Pennsylvania State University, College of Information Sciences and Technology, May, 2013
• V. Mancuso (2012), An Interdisciplinary Evaluation of Transactive Memory in Distributed Cyber Teams, Ph.D. dissertation for the
Pennsylvania State University, College of Information Sciences and Technology, August, 2012
• D. Minotra (2012), The Effect of a Workload-Preview on Task-Prioritization and Task Performance, Ph.D. dissertation for the
Pennsylvania State University, College of Information Sciences and Technology, August, 2012
• E. McMillan (2012), Promoting the Use of Intelligence and Intelligence Analysis as Complementary Components to Enhance Situation
Awareness in Cyber Security: A Qualitative Research Study of the Use of Analytical Techniques and Structured Methodologies by
Cyber Security Experts, M.S. thesis for the Pennsylvania State University, College of Information Sciences and Technology, August,
2012
5
4th Year Focus and
Contributions
• Continued maturation and evolution of
CyberCities/teamNETS/NETS-DART
Development of synthetic test data
• Evaluated a general framework and
metrics for evaluation of SA
• New theoretical frameworks for transactive memory,
dynamic task prioritization & visual analytics
• Completed 3 prototypes for experimentation & conducted
human in the loop experiments
• Conduct of 3 human-in-loop experiments
• Developed new concepts for computer aids for cyber SA and
supporting visual analytics workbench
Test and Evaluation Environment
Two Laboratories in the Penn State College of
Information Sciences and Technology enable
human in the loop experiments with cyber
situational awareness in individual and team
environments
• The Multidisciplinary Initiatives in
Naturalistic Decision Systems (MINDS)
Laboratory
• The Extreme Events Laboratory
7
Promoting the Use of Intelligence and Intelligence Analysis as
Complementary Components to Enhance Situation
Awareness in Cyber Security
E. McMillan, M.S. thesis
Objectives:
• Understand cognitive/contextual elements of
situation awareness in cyber-security domains
• Understand how Intelligence and Intelligence
Analysis methods can be utilized by cyber security
experts in structuring their analysis and assisting in
gaining control of situations they encounter
• Focus on the human element within real context
applications
Scientific/Technical Approach
Living laboratory framework involving;
• Conduct of Ethnographic studies
• Knowledge elicitation of domain experts, using
NVivo tool for content analysis
• Development of cognitive and process
frameworks and theories
Accomplishments
• Conducted extensive literature review in cyber security,
intelligence analysis, situation awareness and cyber
situation awareness
• Enhanced the Data, Information, Knowledge and
Wisdom (DIKW) framework to include Intelligence
analysis and created a taxonomy for cyber security
• Collected interview/survey data from practicing cyber
analysts from a Security Operations Center in a large
healthcare organization in the western U. S. on their
utilization of analytical and structured tools compared
with Intelligence Analysts
An Interdisciplinary Evaluation of Transactive
Memory in Distributed Cyber Teams
V. Mancuso
PhD Dissertation
Objectives:
• To understand how distributed cyber teams form,
maintain and utilize transactive memory systems
• To understand the behavioral, social and
organizational outcomes of transactive memory
systems in distributed collaborations
• To determine how to design collaborative interfaces
to better support transactive memory formulation,
utilization and maintenance in distributed cognition
Scientific/Technical Approach
Living laboratory framework involving;
• Ethnographic studies
• Knowledge elicitation of domain experts
• Development of cognitive and process
frameworks and theories
• Implementation of a scaled world prototype
• Conduct of human in the loop experiments
• Analysis and transition to real-world
environments
Accomplishments
• Conducted literature search related to team-based
cognition and transactive memory in distributed teams
• Designed a scaled-world simulation called teamNETS
for a distributed cyber situation awareness team
• Conducted a human-in-the-loop experiment with 66, 3person teams to evaluate collaboration and transactive
memory formulation and use
• Conducted a quantitative assessment of transactive
memory perceptions, utilization and content, situation
awareness and team perception
• Developed recommendations for transactive memory,
collaboration tools for real-world environments
The Effect of a Workload Preview on
Task-Prioritization and Task Performance
D. Minotra
PhD Dissertation
Objectives:
• To understand the impact of high mental workload
on cyber-security analyst performance
• To determine the utility of task-prioritization
cognitive aids for improved focus of attention and
facilitation of performance under high timepressure
• To evaluate the effectiveness of a workload
preview aid for improved performance
Scientific/Technical Approach
Living laboratory framework involving;
• Ethnographic studies
• Knowledge elicitation of domain experts
• Development of cognitive and process
frameworks and theories
• Implementation of a scaled world prototype
• Conduct of human in the loop experiments
• Analysis and transition to real-world
environments
Accomplishments
• Conducted a literature review of attention-guidance,
task-management, interruptions and workload previews
• Created a scaled-world simulation (NETS-DART) to
emulate cyber-security monitoring and decision-making
• Conducted a human in the loop experiment involving 77
participants under different task-load conditions to
determine the effectiveness of a workload preview aid
on task performance
• Developed recommendations on factors that influence
the effectiveness of cognitive-aids aimed at guiding
attention and improving task-performance
Measuring the Effectiveness of Visual Analytics and Data
Fusion Techniques on Situation Awareness in Cyber Security
N. Giacobe
PhD Dissertation
Objectives:
• To determine effective methods for measuring the
impact of interface design on situation assessment
for the cyber domain
• To create a prototype visualization aid for cyber
situation awareness
•To determine the effectiveness (on inference
accuracy and speed) of an implemented visual aid
prototype
Scientific/Technical Approach
Living laboratory framework involving;
• Ethnographic studies
• Knowledge elicitation of domain experts
• Development of cognitive and process
frameworks and theories
• Implementation of a scaled world prototype
• Conduct of human in the loop experiments
• Analysis and transition to real-world
environments
Accomplishments
• Conducted a literature review of cyber situation
awareness, data fusion and visual analytics for cyber SA
• Conducted an ethnography study and knowledge
elicitation of 60 IT professionals regarding tool utilization
• Developed a visual analytics cyber SA artifact using a
Geo-Visualization toolkit
• Developed a framework for measuring the efficacy of
cognitive aids and collaboration tools based on SAGAT,
SART, HPSM and NASA-TLX
• Conducted a human in the loop experiment with 25
subjects in a 2x2 experiment to evaluate effectiveness of a
visualization tool for cyber situation awareness
Ongoing research
- Extending the modeling concept from passive to active/prediction
- Previous work focused on IDS/Security Analysts
- New work shifts to the Threat Analysis Analysts
- Extending the threat model to include merging of hard and soft data
- How can we use textual data (blogs, tweets, etc.) to inform what kinds of
hard data (IDS, flow, logs, etc.)?
- Extending into the threat domain analysis of cyber security
- How does this kind of cyber-security work differ from the front-line
intrusion detection analyst?
- Exploration of automated processing aids including CEP and
Coherence Network modeling
- What tools can we apply to address the needs of this new domain by
supporting the human’s fusion and awareness?
- Initial concepts for visualization and sonification support
- What visualizations support sense making and decision making in the
threat analysis domain?
Human obs.
Search
engines
Text extraction
& processing
Extending the Threat Model
Analysis Tools (CEP, CNP, etc.)
New Concepts for Cyber
Trans-Action
Developing Extensible Cognitive Artifacts to Support Distributed Cyber Security Work
Objectives: 1) Building Work Support Tools based on Socio-Technical Systems
2) Design Ecological Group-Based Interaction Displays to test in teamNETS
3) Adversarial Team to Team Interaction
• Functional Abstraction Hierarchy (Rasmussen, Sanderson) for Cyber Operations
- Functional work in cyber detection with other cross-function areas (threat assess)
• Decision Ladder to portray decision making flow and adaptations across team members
• Ecological Interface Design for teamNETS to improve SA under changing conditions
- context switching given adaptable emerging priorities
• Models of the Adversary to be embedded within attack framework given above artifacts
- created through hidden knowledge profiles (static implementation)
- extended to intelligent monitors who spoof and deceive the team-force
Complex Event Processing (CEP)
for Cyber Situation Awareness
Complex event processing
(CEP)
• Originally created for
financial and stock trading
applications
• Subsequently applied to
smart energy, RFID
middleware and limited
data fusion applications
• Capabilities for rapid rulebased filtering, aggregation
and event detection
• Allows hierarchy of “levels
of analysis”
Low-level events filtered and
aggregated into a higher-level
event
CEP Processing Infrastructure
16
CEP/CNB Interaction
17
Technology Definitions
StreamBase CEP Engine: a performance-optimized, Java-enabled Complex Event Processing (CEP)
framework.
(see http://www.streambase.com)
JADE: A Java-enabled Multi-Agent System (MAS) framework that enables creation, utilization, and
administration of scalable “communities” of software agents.
(see http://jade.tilab.com)
AMQP/RabbitMQ: The Advanced Message Queuing Protocol (AMQP) is a Message Oriented
Middleware (MOM) for advanced, context-dependent routing of messages between multiple
software tools/nodes. RabbitMQ is an open source client/server implementation of AMQP.
(see http://www.rabbitmq.com)
TML/EML: The Open Geospatial Consortium (OCG) standard Transducer Markup Language (TML)
and Event Pattern Markup Language (EML) protocols encourage structured, standardized
metadata and allow snapshots of complex event patterns (EML only).
(see http://www.opengeospatial.org/standards/dp)
Mulgara: A semantic data store that is optimized for storage and access of “tuples” rather than
strictly relational data.
(see http://www.mulgara.org)
An Emerging Visual Analytics
Workbench
Network
Activity View
Data &
message
view
Social
Network
View
Task: Identify Structure and Methods of Syrian Electronic Army
Timeline
View
Identify individuals, methods used and individual events. Make predictions on whether our organization will be
impacted by the SEA. Check the current network and server status for evidence of similar methods being used
against our organization.
– Website defacement
– DNS Hacks
– XSS Examples
Analysts
Notes
5th Research Plan
• Develop/refine an abstraction hierarchy and decision
ladders to represent team cognition/adversarial model
dynamics
• Refine and evaluate evolving cognitive aids and
visualization tools
• Complex event processing (CEP)
• Coherence Network Builder (CNB)
• Emerging visual analytics workbench
• Conduct human in the loop experiments related to
adversarial dynamics in distributed cyber SA
20
5th Year Analytical Study:
Abstraction Hierarchy /
Decision Ladder Composite Models
Goals: (1) build an abstraction hierarchy to model cyber security
from the socio-technical systems perspective
(2) derive specific decision ladder model to represent
team cognition – adversarial model dynamics
Primary Approach: utilize a composite strategy taking wholistic
knowledge across qualitative/quantitative MURI studies to
integrate and distill an overall model that provides:
° basis for creating ecological interface designs for team cyberSA
° identification of research gaps for future work in cyber SA areas
Outcomes: integrates cyberSA with cognitive systems
engineering approaches, qualitative-quantitative-designmodelinh methods are integrated to form systems perspective
21
5th year Experiment:
Adversarial Dynamics
in Distributed Cyber SA
Objectives:
(1) Do team members use adversarial intelligence in cognitive processing ?
(2) How do specific models influence cyber SA, decision making patterns, and
human performance?
Study: Using hidden profiles to emulate adversarial models
° use of the existing teamNETS simulation with medium level time pressure
° IV#1: Extent of Knowledge of Adversarial Intelligence
- simple model
-- one member has advantageous intel information
- compound model -- two members have advantageous intel information
- complex model -- three members have advantageous intel information
° IV#2: Extent of Visual Analytics Support (present/absent)
° Dependent variables (DV.s)
- When (if) do members use their intelligence?
- When does information congeal together in shared form?
- When does intelligence of adversary influence performance?
22
Personal Computer
Security MOOC
• Developed by IST and
Dr. Gerald Santoro, a
top expert on cybersecurity
• Asynchronous
delivery and
assessment
• 4 modules, each
with 3 topics
• Online quiz for
each module
• Content includes topic
introductions, reading
materials and video
lectures
• Certificate awarded
for successful
completion of quizzes
• Future capability of
instructor interaction
23