Modeling Notations for Security Aspects Models provide an abstraction of the system. They capture an abstraction of certain types of information and descriptions, communicate the information, make implicit information explicit, and act as a repository for knowledge and rationale. Conceptual modeling notations are a major area of study in software engineering toward facilitating requirements extraction, management, analysis, and visualization. Models can be used system and architecture design, analysis, verification, and visualization as well. Modeling notations that are mainly used in software engineering express different concepts since are designed for different purposes. For example, UML provides a set of diagrams for modeling classes and objects, their properties, and relations for objectoriented development. UML provides the means to model the behavior of the system by providing sequence and statechart diagrams. ER diagram is another example of widely used notation for modeling static view of entities in the system and their relationships. Agent and goal-oriented modeling notations capture the “why” aspects in additions to “what” and “how”, to enable analyzing the goals of stakeholders and checking the goal satisfaction. Security issues involve special aspects that traditional software engineering notations do not consider. For example, a general behavior modeling notation expresses interactions of entities in the system without considering the behavior of an external adversary. Therefore, the impacts of the malicious behavior of the adversary on requirements, design, and architecture are not circulated to the next phases of development by the models. To capture specific security aspects such as attackers and their malicious behavior (attacks), vulnerabilities, assets, and countermeasures, various security modeling notations are developed. This section overviews existing security-specific conceptual modeling languages and categorizes them based on their target use and types of concepts they express. We have identified two main categories of notations: One which focuses on the attacks and vulnerabilities, and one that captures countermeasures and security requirements. We call the former black aspects notations and the latter white aspects modeling approaches. 1.1 Black Aspects: Vulnerabilities and Attack Modeling The major effort in developing security-specific conceptual modeling languages is dedicated to modeling attackers, malicious behavior, and attacks. Few of the existing approaches explicitly express the vulnerabilities. To our knowledge, the currently used modeling notations do not provide explicit modeling elements for expressing vulnerabilities in the systems as backdoors used by the attackers to exploit the system. 1.1.1 Attack Tree Schneier was first to associate the term “Attack Tree” with the use of tree style models for expressing different ways in which a system can be attacked [6]. He proposes attack tree as a formal and methodical way for describing the security of the system based on varying attacks. Schneier argues that if designers understand all different ways in which a system can be attacked, understand who the attackers are, and their abilities, motivations, and goals, they can likely design countermeasures for those attacks. The root node of an attack tree is the goal of the attack and different ways to achieve that goal are leaf nodes. Node can be decomposed by AND and OR relations. Schneier suggests assigning values or labels to the nodes, such as cost or probability of achieving a goal or doing a task. Such values assigned to the attack tree, it would help to calculate the probability of success of a given attack and determine where a system is vulnerable. Attack trees can be used for making security decisions. The value of the root node reveals if the system’s goals are vulnerable to the attacks. In addition, properties of attackers need to be added to the attack trees such as level of skill, access, risk aversion, money, etc. The characteristics of the attacker help determining which parts of the attack tree need to be considered. However, this requires that attack trees contain knowledge about attacks. Benefits. Attack tree provides a way for modeling goals of an attack and alternative ways to achieve that goal. This helps us to study the system from the attackers’ point of view, which may lead us to determine possible ways that the system can be compromised. By assigning cost or probability measures to the nods of attack tree, one can analyze if the attackers efforts worth the value that can be achieved or not, and as a result, this helps analyzing if the system is at risk and vulnerable. Finally, Schneier asserts that attack trees help analyzing the what-if by evaluating the impact of applying countermeasures. Limitations and Drawbacks. Attack trees depend heavily on knowledge about attackers’ characteristics such as skill level, risk aversion, resources, and access level. However, attack trees do not provide a way to incorporate this information into the model. Attack tree notation is simple, in which it cannot express resources, access level, or skills required to perform a task or achieve a goal. Various the types of analysis can be done on an attack tree with numerical values distributed among its node about cost or probability of an attack. However, this depends on the availability and accuracy of such measures in real practice. Finally, Schneier does not discuss how attack trees can be linked to other development artifacts, such as design or architecture models, requirements and business models. It is not investigated how attack trees can derive security requirements, or influences the architecture design. 1.1.2 Fault Tree Fault Tree Analysis (FTA) [10] is one of the most commonly-used techniques in reliability engineering. It determines various combinations of hardware and software failures, and human errors that could lead to the occurrence of undesired events/failures (called top events) in the system level. This model starts from identifying system failures. Each failure is analyzed on its own fault tree and by refined it into more detail events and end up with un-develop/external events (called leaf events). FTA visually models logical relationships among infrastructure failures, human errors, and external events that could lead to the system failure. The logical relationship is represented by several types of logical gates such as AND, OR, XOR, PRIORITY, etc. Benefits. Through fault tree models, the analyst can obtain minimal leaf events that could lead to the occurrence of system failure which is called as minimal cut-set. The model can show how to prevent the failures for being occurred by negating the model and minimal cut-set (called minimal path-set). Minimum path-set can be seen as the ways to prevent the occurrence of failures as means to attain the reliable system. The main goal of FTA is to assess likelihood of the system failures/top events based on likelihood of leaf events. Limitations and Drawbacks. Although this method enables modeling faults and tracing them to leaf events or errors, it does not provide means to express vulnerabilities of the system. For modeling security attacks, one may use events, but Fault Tree approach cannot link attacks to vulnerabilities and the system elements that have the vulnerabilities or cause them. Although minimum path-set are proposed for modeling ways to prevent occurrence of a failure, the impact of the countermeasures on other events, vulnerabilities, attacks, and countermeasures cannot be expressed explicitly. 1.1.3 Attack Nets McDermott [9] proposes modeling Attack Nets as a Petri net with a set of places representing interesting states or modes of the security relevant entities of the system. Places are represented by circles and can be compared to nodes in a graph. The Attack Net also has a set of transitions that represent input events, commands, or data that cause one or more security relevant entities to change state. Transitions are represented by rectangles and connected to places by arcs. The attack net has a set of tokens. Tokens move from place to place along the directed arcs to indicate the progress of the attack. The Attack Net model presented by McDermott is a straightforward approach to attack modeling with Petri nets. In this modeling approach, attack steps are represented by places similar to nodes in attack trees. Transitions are used for the explicit modeling of attacker actions which significantly extends the expressiveness of this model compared to attack trees. Benefits. Attack Net approach enables modeling concurrency and attack progress with tokens, intermediate and final objectives as places, and commands or inputs as transitions. Labeled tokens can be used to indicate which attacker is responsible for moving a particular component of the system of interest into a particular state. Limitations and Drawbacks. However, the Attack Nets approach does not support countermeasure modeling and analysis. Similar to attack tree, the attack model is not linked to other development artifacts, such as design or architecture models, requirements and business models. It is not investigated how attack nets can derive security requirements, or influences the architecture design. Although it is possible to express who the attacker is, goals, skills, and capabilities of the attacker cannot be expressed in the attack net models. Finally, this approach does not consider modeling and analyzing the concept of vulnerability, and attacks that exploit vulnerabilities. 1.1.4 Attack Graph Phillips et al. [8] introduced attack graphs to analyze vulnerabilities in computer networks. Attack graphs provide a method for modeling attacks and relating them to the machines in a network and to attackers. The proposal is based on attack templates, attack profiles, and network configurations. Attack templates describe generic steps in known attacks and conditions which must be hold. They are represented as a directed graph in which nodes are user levels, machines, vulnerabilities and capabilities. Edges represents the actions performed by attackers. Network configurations contains information relevant to operating system, network type, router configuration, and a graph of the topology of the network. The attack profile contains information about attackers' capabilities, such as the possession of an automated toolkit or a sniffer, and an assumption about the skill level of the attacker. The skill level is used to determine the probability of success for particular attack methods. The attacker profile represents the initial capabilities of the attacker in the same way that the configuration file represents the initial state of the network. The underlying idea is to match the network configuration, attacker profile, and attack templates to generate the attack graph. The description of attack are added to the nodes of the attack graph based on matching the attack template with the network configuration and attacker profile. An attack graphs is thus an attack template instantiated with particular attackers/users and machines. The edges can be labeled with probability of success or cost. Thereby, one can analyze the attack graph by identifying the attack paths that are most likely to succeed. Benefits. Attack graph provides a method for modeling attacks and relating the attacks to the machines in a network and users which could be the attackers. Using this approach, temporal stages and steps of an attack can be expressed. In addition, it supports expressing multiple ways to achieve malicious goals or attack the system by branches in the attack graph. An interesting suggestion of this work is using a database of common attacks, which contain various information about attacks; such as level of user who imposes the threat, machines which are involved, vulnerabilities which are imposed to the configuration by an attack, capabilities which express the malicious behavior needed to be delivered, and states which are atomic steps of an attack. Limitations and Drawbacks. Although attack graphs are able to model the steps of an attack, its post and pre conditions, required configurations, and capabilities, they do not provide a way to express the impact of the attacks on system functionalities. Finally, the approach mentions the possibility of countermeasure analysis, but it does not provide means to link possible countermeasures to attacks. The description of attack, i.e. user level, machine, vulnerability, capabilities, and state, are added to the nodes of the attack graph based on matching the configuration, attack template, and attacker profile. However, it is not described how the matching is performed. This matching may require defining a taxonomy of attacks, vulnerabilities, and capabilities, and a detailed description of the configuration. In addition, all the information about steps of an attack (user level, machine, vulnerability, capabilities, and state) are presented to the reader or user of the model in the same view. This reduces the use of such models for understanding the behavior of one single user or group of users, or security impact of an action or atomic steps. Although the proposed approach to modeling attacks is able to express the steps of occurring an attack, its post and pre conditions, required configurations, and capabilities, it does not provide a way to express the impact of the attacks on the normal functions of the system. It is not argued why the proposed method is limited to network vulnerability analysis, and why it cannot be generalized to apply to other computer systems such as Internet attacks, applications’ vulnerability, domain specific attacks, wireless communication, etc. The proposed quantitative analysis method which assigns probability or cost to the edges for computing the shortest path, and as a result the most likely attack works well in theory. However, in real-world practices such quantitative values and even approximate measures may not be available. Finally, the approach mentions the possibility of countermeasure analysis, but it does not provide means to link possible countermeasures to the attacks. 1.1.5 Misuse Case Sindre and Opdahl [15] develop an extended notion of use cases to express negative use cases, which are called misuse cases. The major difference is that a use case achieves something of value for the system owner, whereas the misuse cases are harmful. This approach provides the analysts with modeling constructs for modeling and analyzing malicious behavior which is modeled by misuse cases that threaten use cases; and countermeasures for mitigating the threats which are expressed as security use cases that mitigate misuse cases. Therefore, include and extend relationships are extended, and prevent relationship is added to the misuse case models to relate security use cases to misuse cases. The basic concepts of the framework and their relation to the UML metal model are provided in a meta model. The process of eliciting security requirements by misuse cases consists of 5 steps: it starts with identifying critical assets. Then security requirements for each asset are defined. In the third step, threats to each security requirements are defined and expressed as misuse cases. In the fourth step, risks of threats are identified and analyzed, and finally, security requirements for the threats are defined as either security use cases or in the mitigation field of misuse case description. Security requirements are either defined as security use cases or as mitigation in misuse case description. Benefits. Looking at system from a misuser perspective increases the chance of discovering threats that would otherwise have been ignored. The authors assert that the visualization of links between use cases and misuse cases help organize the requirements specification and tracing requirements to threats that motivated them. Limitations and Drawbacks. However, the security requirements elicitation process does not consider why and how security goals are defined without analyzing what may threaten the assets. In addition, the targets of threats are not only security goals, and threats may target an asset or service rather than a security goal. The notion of misuse case cannot express due to what kind of vulnerability a misuse threatens a use case, why a misuser attacks the system, and what is the impact of a security use case on other use cases. Rostad [11] suggests extending misuse case notation for modeling vulnerabilities as weaknesses in the system and insider attackers. The extended notation introduces a new type of actor called insider that has the capabilities and permissions of an authorized actor who misuses the given permissions. The concept of vulnerability is defined as a weakness that may be exploited by the misuse. In the extended modeling notation the concept of vulnerability is expressed a type of use case, with an exploit relationship from the threat to the vulnerability. In the example cases presented in [11], “enter password” or “enter username” use cases are highlighted as the vulnerability that the injection attack or overflow attach can exploit. The author asserts that the threat toward a system may only be realized in an attack if the system contains vulnerabilities that can be exploited, and it is important to consider the vulnerabilities to identify all possible threats and attacks. However, it is not justified why and how vulnerability, as a weakness in the system that can be exploited, is expressed as a type of use case. The examples of vulnerabilities in [11] are ordinary use cases of a system, that bring a vulnerability to the system. In sum, the semantics of the proposed modeling element as vulnerability is not compatible with the definition and concept of vulnerability as a weakness. For example, buffer overflow or password loss are example of vulnerabilities, while in the examples provided in [11] vulnerable use cases were modeled instead of weaknesses. 1.1.6 Abuse Case The other similar notion based on the conventional UML use case modeling is the notion of abuse cases [3]. While use cases are abstract episodes of useful and desired interaction between a system and its environment, abuse cases are specification of types of interactions that their results are harm to system. An abuse case describes the abuse of privilege used to complete the abuse case. If a human that is represented by an actor from a use case might also act maliciously in the corresponding role, then a new actor should be defined in the abuse case. The abuse cases are not distinguished by use cases, although the authors assert they can be labeled. The use case model and abuse case model are basically two separated models. Use case models are accompanied with use case description, mostly written in natural language. Actor description is useful in abuse case model, since the actor’s resources, skills, and objectives for an abuse case can be captured. The authors state that objectives of an actor are not captured in abstract use cases; therefore, objectives are expressed as the long-term goals of the actor that they seek over more than one abuse case. Benefits. The notation is simple and is based on well known and widely-used UML “use case” model, and can be used for communicating the flaws and family of abuse cases of desired use cases to end users and customers. Limitations and Drawbacks. However, the use case model and abuse case model are not semantically related; therefore, the model does not provide means to analyze the impact of a abuse case on other use cases. Actors and malicious actors could be the same entities or different, but the abuse case approach does not differentiate it. It is not clear how abuse cases are used in the requirements elicitation and analysis. 1.2 White Aspects: Security Requirements and Countermeasures Modeling Another major group of contributions to conceptual modeling notations for securityspecific aspects are toward modeling the protection mechanisms against the malicious behavior. Although attacks are considered in many of these conceptual modeling languages, these types of contributions focus on modeling security relevant information, security countermeasures, and security requirements. Since, security requirements modeling and analysis frameworks are extensively researched, we separately describe security requirements frameworks in next section, and this part is limited to related conceptual modeling notations that capture security specific concepts. 1.1.7 UMLsec: Extending UML for Secure System Development UMLsec [12] is an extension to UML that allows expressing security relevant information within UML diagrams. The main uses of such approach are first, to encapsulate knowledge and make it available to developers in form of a widely-used design notation, and secondly, to provide formal evaluation to check if the constraints associated with the UMLsec stereotypes of fulfilled in a given specification. The extensions are suggested in form of UML stereotypes, tags, and constraints that can be used in various UML diagrams such as activity diagrams, statecharts, and sequence diagrams. For a security analysis of the UMLsec subsystem specification, potential adversary behavior needs to be modeled. For this purpose, a function of Threat is defines which takes an adversary type and a stereotype and returns a subset of {delete, read, insert} which changes a state of the subsystem. The stereotypes and tags encapsulate the knowledge of recurring security requirements of distributed object-oriented systems, such as secrecy, fair exchange, and secure communication link. By assigning a stereotype and tag to part of a model, and retrieving the threats, the behavior of the subsystem can be analyzed to check the impact of the threat. Discussion. The general high level idea of adding security stereotypes, tags, and constraints to the UML models is useful for expressing security related information. The formal definition of semantics of the UMLsec extensions is useful for doing security evaluations. The defined security requirements are high level and general. On the one hand, this increases the reusability of the extensions in various contexts. On the other hand, the security requirements and other stereotypes such as Internet and LAN are too high level, and the default threats (and semantics of the requirements or stereotype) may not be valid in every context. It is true that the impact of any security threat is finally deleting, reading, or inserting data in an authorized way. However, security attacks in existing taxonomies, guidelines, and standards are not categorized in this way, and a designer may know more about attacks expressed as SQL injection, viruses, man in the middle, etc. In addition, the notion of {Delete, Read, Insert} is not useful for expressing the impacts of attacks such as Denial of Service. In an example, the author discusses how UMLsec can be used to evaluate a security protocol specified in a statechart diagram. However, it is not clear how the cryptographic operation and message exchanges are related to the <<data security>> stereotype. The author argues that UML is a widely used notation in analysis and design of software systems and developers already know UML modeling notation, and the notation is well defined. However, this does not provide sufficient justification for modeling and analyzing security requirements using extensions to UML. One may argue that security concerns involve aspects that UML diagrams are not able to grasp. Finally, even if we agree with the author that UMLsec is useful for modeling and analyzing security requirements, UML is not a requirements engineering notation. The only diagram that focuses on the system functionalities and expectations from the system from the users’ point of view is the use case diagram, while the formal semantic defined for the use cases is limited. 1.1.8 SecureUML for Model-Driven Security The contribution in [5] presents a methodology for modeling access control policies and their integration into a model-driven software development process. The proposed method uses the SecureUML modeling language for the model-driven development of secure systems, based on the Unified Modeling Language (UML). SecureUML is based on an extended model for role-based access control (RBAC). RBAC is a model for access control where users and their privileges are decoupled by roles. The SecureUML meta model defines authorization constraints as a special kind of UML constraint and uses OCL as the expression language. It defines a vocabulary for annotating UML-based models with information relevant to access control. It is based on the model for RBAC with additional support for specifying authorization constraints. Its meta model is defined as an extension of the UML meta model. The concepts of RBAC are represented directly as meta model types. We introduce the new meta model types User, Role and Permission as well as relations between these types. SecureUML can be used in the context of a model-driven software development process to generate access control infrastructures. However, the meta-model of the SecureUML notation is limited to the access control concepts. The meta model is developed in a way that is not easily expandable to capture other security mechanisms in addition to access control. The usage scope of SecureUML is limited to incorporate the access control roles and conditions in the UML class diagrams to automatically generate required code for access control policy. 1.1.9 Security Extensions to Tropos Mouratidis et al. [1] introduce extensions to Tropos methodology for incorporating security concerns into the agent-oriented development process. For this aim, the concepts of security diagram, security constraint, secure dependency, and security goal, task, and resource is introduced. The security diagrams contain security features of the system-to-be, the protection objectives of the system, the security mechanisms, and also the threats to the system’s security features. To model these concepts Tropos modeling notation is used and extended. The concept of softgoal is used to capture security features. Security feature receives positive contributions from different protection objectives and negative contributions from the threats. Security mechanisms contribute positively or negatively to the protection objectives. Security objectives are expressed by the goal modeling construct, and security mechanisms are modeling using the i* tasks. Threats represent circumstances that have the potential to cause loss or problems that put the security features in danger. To model threats a new modeling constructs is added to the i* modeling notation. All secure entities are tagged with “s” to indicate these tasks, goals, and softgoals are security related. In this work, security requirements are dealt as constraints. Concept of constraint is different from the concept of a goal. By imposing security constraints to different parts of the system, possible conflicts between security and other requirements are identified. The assignment of a security constraint to a goal is indicated using a constraint link that has the “restricts” tag. In this way, the context of the constraint is defined. Using security constraints, a secure dependency introduces security constraint(s) to successfully satisfy the dependency. The proposed diagrams and the modeling notations are fitted into the Tropos methodology. The proposed modeling approach targets early and late requirements engineering phases of development. In another similar work, Mouratidis et al. [2] add secure capabilities of actors to the contribution in [1]. In this work, the Secure Tropos is fitted into all phases of development: early and late requirements, and architecture and detailed design. However, the capabilities or lack of capabilities do not affect the architecture and detailed design. Using constraint on the goals by a contribution from the goal to the constraint, and the “restrict” relation from the constraint is intuitive. By expressing restrictions and introducing security objectives and mechanisms that satisfy the constraints, the rationale for having security objectives and mechanisms are expressed. In this work, all security related entities are tagged with a “s”, while distinguishing the security entities from other systems entities does not affect the result of analysis on the models. The notion of threat introduced in the secure Tropos is limited to a visual representation. The threats are not assigned an attacker and goals and refinements of the attacks are expressed. I still can’t agree or disagree that security constraints are necessary. Are there cases that we need to express the negative goals? Yes, I have seen it in student assignments. They are not anti-goals (malicious), but they are goals with a negative or restricting statement in it, like the constraints that was introduced in this contribution. Functional Requirements Security Requirements Other Quality Requirements Assets Vulnerability Security Mechanisms Vulnerability and Attack Modeling Notations Attack 1.3 Conclusions Modeling notations are divided into two main groups: one captures malicious behavior and one focuses on the protection mechanisms. Studying existing notations for modeling attacks and security mechanisms shows that attacks are modeled in a similar way that security mechanisms are modeled, while the goals and means of the attacker and defender are different. Contributions such as [12] express the need to joint execution of the system with presence of attacks and countermeasures to check if the security mechanisms are successful in protecting the system against the attacks. Attack Tree [6] Fault Tree [10] Attack Nets [9] 1 2 Attack Graph [8] 3 4 Misuse case [15] Abuse Case [3] Usage Modeling and understanding attacks, identifying vulnerable points Assess likelihood of the system failures Modeling steps, concurrency and attack progress Vulnerability and attack modeling and analysis in computer network Modeling attacks in conjunction with functional requirements for security requirements elicitation Communicating the flaws and family of abuse cases of desired use cases to end users and customers. Functional Requirements Security Requirements Other Quality Requirements Functional Requirements Security Requirements Other Quality Requirements Assets Assets Vulnerability 5 Vulnerability Attack Security Mechanisms Security Mechanisms Security Requirements and Countermeasures Modeling Attack Vulnerability and Attack Modeling Notations Extended misuse case [11] UMLsec [12] 6 7 * * SecureUML [5] 8 * * Security Extensions to Tropos [1] 9 Usage Modeling vulnerabilities to identify all possible threats and attacks Usage Expressing security relevant information within UML diagrams Model-driven development of secure systems, based on the UML Incorporating security concerns into the agentoriented development process for security requirements engineering 1. Although in fault trees a failure is refined into a tree, in a same an attack and exploit of the system can be refined to its causes. 2. The model can show how to prevent the failures for being occurred by negating the fault tree, called minimal path-set. 3. Vulnerabilities are expresses in nodes of the attack graph 4. Assets are limited to the network machines. 5. Vulnerabilities, as weaknesses in the system are not expressed. Instead vulnerable use cases are related to the misuse cases by an exploit link. 6. Although Jurjens assert that UMLsec can be used to model threats, UMLsec does not provide a notation to model attacks from the point of view of an attacker. 7. Assets in UMLsec are the components and elements of the system expressed in the UML models, such as classes or messages in sequence diagram. 8. Security mechanisms in SecureUML are limited to the access control mechanisms only. * Generally, the UML-based notation are not intended for eliciting and modeling requirements. However, they express the functionalities at the design phase. 9. The proposed notation can be used for modeling other quality attributes. However, in [1] the focus is on eliciting security requirements as constraints on functional requirements. References 1. Mouratidis, H., Giorgini, P., Manso, G., Philp, I., A Natural Extension of Tropos Methodology for Modelling Security, In Proc. of the Workshop on Agent-oriented methodologies, at OOPSLA (2002) 91-103 2. Mouratidis, H., Giorgini, P., Manso, Modelling Secure Multiagent Systems, In the Proc. of the 2nd Int. Joint Conf. on Autonomous Agents and Multiagent Systems (2003) 859 – 866 3. McDermott, J., Fox, C., Using Abuse Case Models for Security Requirements Analysis, In Proc.15th. IEEE Annual Computer Security Applications Conf. (1999) 55-64 4. Moffett, J. D., Haley, C. B., Nuseibeh, B., Core Security Requirements Artefacts, Department of Computing, The Open University, Milton Keynes UK, Technical Report 2004/23 (2004) 5. Torsten Lodderstedt, David Basin, and Jrgen Doser, SecureUML: A UML-based modeling language for model-driven security, Proceedings of the 5th International Conference on The Unified Modeling Language (2002) 426 - 441 6. Schneier, Bruce, "Attack Trees" 21-29. Dr. Dobb's Journal of Software Tools 24, 12 (December 1999): 21-29 7. Andrew P. Moore, Robert J. Ellison, and Richard C. Linger, Attack Modeling for Information Security and Survivability, Technical Note, CMU/SEI-2001-TN-001 (2001) 8. Phillips, C. and Swiler, L. P. 1998. A graph-based system for networkvulnerability analysis. In Proceedings of the 1998 Workshop on New Security Paradigm, 1998, 71-79. 9. McDermott, J.P. “Attack Net Penetration Testing.” Proc. of the 2000 Workshop on New Security Paradigm, Sept. 2000 10. W. E. Vesely, F. F. Goldberg, N. H. Roberts, and D. F. Haasl, Fault Tree Handbook: U.S Nuclear Regulatory Commission, 1981 11. Lillian Røstad, An extended misuse case notation: Including vulnerabilities and the insider threat, The Twelfth Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ'06) 2006. 12. Jan Jürjens: UMLsec: Extending UML for Secure Systems Development. UML 2002: 412-425 15. Sindre, G., Opdahl, A. L., Eliciting security requirements with misuse cases, RE Journal, Volume 10 (2005) 34-44