Analyzing Goal Models – Different Approaches and How to Choose Among Them Jennifer Horkoff1 Eric Yu2 1Department of Computer Science 2Faculty of Information University of Toronto, Canada SAC’11 RE Track Goal-Oriented Requirements Engineering (GORE) GORE has received much attention in RE research as a means of: Generally, GORE frameworks allow for: Understanding the motivations for system requirements Helping to ensure that the right system is built Representation of stakeholder goals Goals may be assigned to an agent (stakeholder or system) Goals may have relationships to other goals, often describing achievement Several goal modeling frameworks KAOS, GBRAM, AGORA, NFR, i*, Tropos, GRL, … Analyzing Goal Models, Horkoff & Yu 2 Goal-Oriented Requirements Engineering (GORE) Example: Counseling Organization i* Model (Horkoff & Yu, 2009) We can analyze the contents of goal models systematically… Analyzing Goal Models, Horkoff & Yu 3 Example: Qualitative, Interactive Forward Satisfaction Analysis of Goal-Oriented Models What is the effect of using a Cybercafe/ Portal/ Chat Room?” Horkoff, Yu: Evaluating Goal Achievement in Enterprise Modeling Analyzing Goal Models, Horkoff & Yu 4 Models and Analysis become Complex Analyzing Goal Models, Horkoff & Yu 5 Goal Model Analysis Many different analysis techniques for goal models have been introduced: Abundance of approaches is encouraging from a research perspective, but… From a user or practitioner perspective can be confusing Propagate satisfaction values through the model Measure metrics over the model Apply planning techniques Run simulations Perform checks over models What are the differences? When would I use one and not another? Limits adoption Analyzing Goal Models, Horkoff & Yu 6 Motivating Questions Survey of methods Analysis benefits What methods are available? What types of analysis questions can these methods answer? What types of goal modeling constructs do the procedures support? What information is needed in order to use the methods? What are some of the potential benefits of goal model analysis in the requirements process? Mapping and Selection What available methods can be applied to achieve which kinds of usage objectives? How can we use this information to advise on selection? Analyzing Goal Models, Horkoff & Yu 7 Survey of Goal Model Analysis Procedures Approach Paper Additional Notation Supported Satisf Satisf Human Metrics Plan- Simu- Model Qual Quant Binary Depend Soft- Contribut Forwds Backwds Interv ning lation Check -encies goals -ion Links Y Giorgini et al. [21] Y Giorgini et al.[22] Y Giorgini et al. [23] Y Horkoff & Yu [26] Y Maiden et al. [33] Y Amyot et al. [1] Y Asnar & Giorgini [3] Y Letier & vLams. [31] Y Horkoff & Yu [27] Y Wang et al. [35] Y Bryl et al. [6] N Bryl et al. [7] N Asnar et al. [4] Y Gans et al [18] N Wang & Lesper. [34] N Gans et al. [16] [18] N Gans et al. [17] N Fuxman et al. [14] [15] N Giorgini et al.[20] N Bryl et al.[8] N Chung et al. [9] N N Y Y N N N Y Y Y Y N N Y N N N N N N N Y N N N N N N N Y N Y N Procedures N N N N N N Y N N N N Y Y Y Y Y Y N N N Y N N Y M N N N N N Analysis Results N N N N N N N N N N N Y Y Y N N Y N N N Y N N N N N N N N Summary N N N N N N N N N N N N N N N N N Y N N Y Y Y N Y Y Y M N Y N Y N Y Y N Y Y Y N Y N Dimensions Y N N N Y Y Y M N Y Y N N N N Y M Y Y N N N N N N Y N Y N N N N N N Analyzing Goal Models, Horkoff & Yu Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N N N M Y Y Y N M Y N Y Y Y Y N Y Y Y Y Y Y M M Y Y Y Y M N Y M N N M N N Y N Y N N Y Y Y Y Y Y Y Y N Y Y N N Y N N Y N N N N 8 A Survey of GORE Analysis Techniques Summarize results over the following points: Algorithm Approach Format of analysis results Satisfaction Forwards, Backwards, Human Intervention, Metrics, Planning, Simulation, and Model Checking Qualitative ( ), quantitative (0.37), binary (T/F) Goal-oriented concepts supported (beyond AND/OR) Dependencies, softgoals, contribution links Approach Paper Additional Notation Supported Satisf Satisf Human Metrics Plan- Simu- Model Qual Quant Binary Depend Soft- Contribut Forwds Backwds Interv ning lation Check -encies goals -ion Links Smith et al. [1] Y N Y N Analysis Results N N N Y N Y N Y Y Additional information required beyond typical goal model constructs e.g., priority, probabilities, events, delegations, and trust. Analyzing Goal Models, Horkoff & Yu 9 Satisfaction Analysis Example Analysis Questions: What is the effect of this alternative? Can this goal be satisfied? Evaluates the satisfaction or denial of goals given a functional or design alternative Values are propagated forward or backward throughout the model Qualitative or quantitative approaches Techniques take different approaches to resolving multiple values for incoming goals: Adding evidence, combine using probabilistic rules, separate evidence, fixed rules, human judgment Analyzing Goal Models, Horkoff & Yu 10 Satisfaction Analysis Approach Paper Chung et al. [9] Giorgini et al. [21] Giorgini et al.[22] Giorgini et al. [23] Horkoff & Yu [26] Maiden et al. [33] Amyot et al. [1] Asnar & Giorgini [3] Letier & vLams. [31] Horkoff & Yu [27] Wang et al. [35] Analysis Results Additional Notation Supported Satisf Satisf Human Metrics Plan- Simu- Model Qual Quant Binary Depend Soft- Contribut Forwds Backwds Interv ning lation Check -encies goals -ion Links Y N Y N N N N Y N Y N Y Y Y N N N N N N Y Y Y N M Y Y Y N N N N N Y N Y N M Y Y Y N N N N N Y N Y M Y Y Y N Y N N N N Y N Y Y Y Y Y N Y N N N N N N Y Y Y Y Y N N N N N N Y Y Y Y Y Y Y Y N N N N N Y M Y N M Y Y Y N N N N N N Y Y M N N Y Y Y N N N N Y N Y Y Y Y Y Y N N N N N N N Y N M Y Analyzing Goal Models, Horkoff & Yu 11 Metrics Example Analysis Questions: How secure is the system represented by the model? How risky is a particular alternative for a stakeholder? Structural properties of the model and construct classifications are used to calculate metrics Metrics often represent non-functional requirements Examples: predictability, security, privacy, accuracy, etc. They can also represent model properties: Example: counts of dependency classifications (instance, model, duplicate, hidden) in a Strategic Dependency (SD) Examples: completeness, consistency and correctness Metrics can be local or global Analyzing Goal Models, Horkoff & Yu 12 Metrics Approach Paper Franch & Maiden [12] Franch et al. [13] Franch [11] Kaiya et al. [30] Analysis Results Additional Notation Supported Satisf Satisf Human Metrics Plan- Simu- Model Qual Quant Binary Depend Soft- Contribut Forwds Backwds Interv ning lation Check -encies goals -ion Links N N N Y N N N N Y N Y Y N N N N Y N N N M Y N Y Y N N N Y Y N N N Y Y N Y Y Y N N N Y N N N N Y Y N N M Analyzing Goal Models, Horkoff & Yu 13 Planning Example Analysis Questions: What actions must be taken to satisfy goals? What are the best plans according to certain criteria? Work has applied AI-type planning to find satisfactory sequences of actions in models Requires definition of axioms that express possible goal decompositions and delegations Expresses the capabilities of actors in a model A planner finds a delegation of goals to actors which fulfills model goals Plans are evaluated by some criteria Analyzing Goal Models, Horkoff & Yu 14 Planning Approach Paper Additional Notation Supported Satisf Satisf Human Metrics Plan- Simu- Model Qual Quant Binary Depend Soft- Contribut Forwds Backwds Interv ning lation Check -encies goals -ion Links Bryl et al. [6] N Bryl et al. [7] N Asnar et al. [4] Y N N Y N Y Y Y Y Y Analysis Results Y Y Y N N N N Y N N M Y Analyzing Goal Models, Horkoff & Yu Y Y N Y Y Y Y Y Y N N M N N Y 15 Simulation Example Analysis Questions: What happens when an alternative is selected? Are there unexpected properties in a simulation? Adds temporal information including pre- and postconditions to models Translated to ConGolog (situation calculus) programs for simulation Extensions simulate confidence, trust and distrust Analyzing Goal Models, Horkoff & Yu 16 Simulation Approach Paper Gans et al [18] Wang & Lesper. [34] Gans et al. [16] [18] Gans et al. [17] Analysis Results Additional Notation Supported Satisf Satisf Human Metrics Plan- Simu- Model Qual Quant Binary Depend Soft- Contribut Forwds Backwds Interv ning lation Check -encies goals -ion Links N N Y N N Y Y N N Y Y N N N N N N N Y N N N Y N N N N N Y N Y Y Y N Y Y Y Y Y N N N Y N Y M N Y Y Y N N Analyzing Goal Models, Horkoff & Yu 17 Model Checking Example Analysis Questions: Is it possible to achieve a particular goal? Is the model consistent? Models are expanded/converted to a temporal formalism Includes expressions of creation, fulfillment and invariant properties First order temporal logic statements are used to represent desired constraints Model checker is used to validate properties and check for consistency Further work adds in checks for security and trust Analyzing Goal Models, Horkoff & Yu 18 Model Checking Approach Paper Additional Notation Supported Satisf Satisf Human Metrics Plan- Simu- Model Qual Quant Binary Depend Soft- Contribut Forwds Backwds Interv ning lation Check -encies goals -ion Links Fuxman et al. N [14] [15] Giorgini et N al.[20] Bryl et al.[8] N Analysis Results N M N N N Y N N Y Y Y N N N N N N Y N N Y Y N N N N N Y N Y N N Y Y N N Analyzing Goal Models, Horkoff & Yu 19 Tabular Summary Approach Paper Additional Notation Supported Satisf Satisf Human Metrics Plan- Simu- Model Qual Quant Binary Depend Soft- Contribut Forwds Backwds Interv ning lation Check -encies goals -ion Links Y Giorgini et al. [21] Y Giorgini et al.[22] Y Giorgini et al. [23] Y Horkoff & Yu [26] Y Maiden et al. [33] Y Amyot et al. [1] Y Asnar & Giorgini [3] Y Letier & vLams. [31] Y Horkoff & Yu [27] Y Wang et al. [35] Y Bryl et al. [6] N Bryl et al. [7] N Asnar et al. [4] Y Gans et al [18] N Wang & Lesper. [34] N Gans et al. [16] [18] N Gans et al. [17] N Fuxman et al. [14] [15] N Giorgini et al.[20] N Bryl et al.[8] N Chung et al. [9] N N Y Y N N N Y Y Y Y N N Y N N N N N N N Y N N N Y Y N N N Y N N Y Y Y N Y N M N N N N N N N N N N N N N Y Y Y N N N Y N N N Analysis Results N N N N N N N N N N N Y Y Y N N Y N N N Y N N N N N N N N N N N N N N Y Y Y Y N N N N N N N N N N N N N N N Y N Y N Y M Y Y Y Y Y Y Y Y N Y Y N Y N N M Y N N N N N N N Analyzing Goal Models, Horkoff & Yu N Y N N N N Y M Y N N Y Y N N N Y Y N N N Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y N N N M Y Y Y N M Y N Y Y Y Y N Y Y Y Y Y Y M M Y Y Y Y M N Y M N N M N N Y N Y N N Y Y Y Y Y Y Y Y N Y Y N N Y N N Y N N N N 20 Information Required by each Procedure 1 2 3 Additional Information Goal Cost Risk Textual Arguments 4 5 6 Probabilistic Information Events and Treatments Importance/Priority 7 8 Actor Capabilities (Pre/Post) Conditions/ Temporal Information 9 Delegation/Ownership 10 Trust 11 12 13 14 Speech Acts Confidence and Distrust Preferences Cardinalities Required by Satisfaction Analysis: [23][4][22][3], Planning: [6] Satisfaction Analysis: [3], Planning: [4] Satisfaction Analysis:[33], Metrics, Model Checking: [30] Satisfaction Analysis: [23] [31] Satisfaction Analysis: [3] Satisfaction Analysis: [1], Metrics: [13] [1], Simulation: [34] Planning: [6] [7] [4], Model Checking: [8] Simulation: [34] [18] [18] [16] [17], Model Checking: [15] [14] Model Checking: [19] [8] Planning: [4], Simulation: [17], Model Checking: [20][8] Simulation: [17] Simulation: [17] Model Checking: [30] Simulation:[34], Model Checking: [14] Analyzing Goal Models, Horkoff & Yu 21 Goal Model Analysis Objectives Using capabilities of techniques in our survey, as well as our own experience in modeling and analysis, we list categories of objectives for goal model analysis List is likely not complete Objective Categories (goal model analysis can help…): Understand the domain Communicate Improve the model Make scoping decisions Prompt requirements elicitation Improve requirements Design a system Analyzing Goal Models, Horkoff & Yu 22 Mapping Procedures to Objectives We have made suggestions concerning what procedures may map to what objectives Each mapping can be considered as a hypothesis We have included guiding questions with each objective to help motivate the mapping and guide users Example mappings: Category Guidelines Recommended Procedures Domain QU1. Does the domain contain a high Yes. Try: Agent Approaches: i*/GRL Satisfaction Analysis Understanding degree of social interaction, have many ([1][26][27][33]) i* Metrics ([11][12][13]) Tropos Metrics, Planning, stakeholders with differing goals, or or Model Checking ([4][6][7][8][14][15][19]) SNET([16][17][18]) involve many interacting systems? Requirements QR1. Are you working with a system Yes. Try: Analysis over Specific Constructs or Metric Improvement where safety/security/ privacy/risks or Approaches: KAOS([31]) i* Metrics([11][12][13]) AGORA([30]) other specific properties are critical Tropos Risk, Trust, and Security([3][4] [8][19]) SNET Trust([17]) considerations? Downloadable interactive mapping table: www.cs.utoronto.ca/~jenhork/GOREAnalysisSelectionTable.zip Analyzing Goal Models, Horkoff & Yu 23 Mapping of Procedures to Objectives Category Guidelines Recommended Procedures Domain QU1. Does the domain contain a high degree of social interaction, have many stakeholders with Yes. Try: Agent Approaches: i*/GRL Satisfaction Analysis ([1][26][27][33]) i* Metrics Understanding differing goals, or involve many interacting systems? ([11][12][13]) Tropos Metrics, Planning, or Model Checking ([4][6][7][8][14][15][19]) SNET([16][17][18]) QU2. Do you need to understand details of the system at this point? Do you have access to Yes. Try: Quantitative or Detailed Information: Tropos Probabilistic Satisfaction Analysis detailed information such as cost, probabilities, and conditions? Can you express necessary or ([3][21][22][23]) KAOS Satisfaction Analysis ([31]) GRL Quant. Analysis ([1]) i* Quant. Metrics desired domain properties? ([11][12][13]) Tropos Planning ([4][6][7][8]) Tropos Modeling Checking ([8][14][15][19]) SNET([16][17][18][18]) i* Simulation([34]), or Model Checking: Tropos ([8][14][15][19]) SNET([16][18]) Communication QC1. Do you need to communicate with stakeholders? Validate requirements in the model? Yes. Try: Forward Satisfaction Approaches: NFR([9]) Tropos([3][21][22][23]) KAOS([31]) Justify recommendations? i*([26][33]) GRL([1]) Model QM1. Are you confident in the accuracy, structure, and completeness of domain knowledge and No. Try: Interactive Approaches: NFR([9]) i*([26][27][33]) Tropos([4][7]) SNET([16][18]) i* Improvement models? Metrics([11]) QM2. Would you like to verify critical properties over the model? Yes. Try: Model Checking: Tropos([8][14][15][19]) SNET([16][18]) Scoping QS1. Do you need to determine system scope? Yes. Try: Agent Approaches: i*/GRL Satisfaction Analysis ([1][26] [27][33]) i* Metrics ([11][12][13]) Tropos Metrics, Planning, or Model Checking ([4][6][7][8][14][15][19]) SNET ([16][18]) Requirements QE1. Do you need to find more high-level requirements? Are you looking for ways to prompt Yes. Try: Interactive Approaches: NFR([9]) i*([27][27][33]) Tropos([4][7]) SNET([16][18]) i* Elicitation further elicitation? Metrics([11]) QE2. Do you need to find detailed system requirements? Yes. Try: Quantitative or Detailed Information: Tropos Probabalistic Satisfaction Analysis ([3][21][22][23]) KAOS Satisfaction Analysis ([31]) GRL Quant. Analysis ([1]) i* Quant. Metrics ([11][12][13]) Tropos Planning ([4][6][7][8]) Tropos Modeling Checking ([8][14][15][19]) SNET([16][17][18][18]) i* Simulation([34]) Objectives Procedures QE3. Do you need to consider non-functional requirements difficult to quantify? Requirements Improvement Design Yes. Try: Approaches supporting softgoals or contributions: NFR([9]) i* Satisfaction Analysis ([26][27][33]) Tropos Satisfaction Analysis ([3][21][22][23]) Tropos Model Checking([14][15]) GRL([1]) i* Metrics([11][12][13]) SNET([16][17][18]) QE4. Do you need to capture domain assumptions? Yes. Try: Approaches using Satisfaction Arguments: i* Satisfaction Arguments [33] QR1. Are you working with a system where safety/security/ privacy/risks or other specific Yes. Try: Analysis over Specific Constructs or Metric Approaches: KAOS([31]) i* properties are critical considerations? Metrics([11][12][13]) AGORA([30]) Tropos Risk, Trust, and Security([3][4] [8][19]) SNET Trust([17]) QR2. Do you need to find errors and inconsistencies in requirements? Yes. Try: Model Checking: Tropos([8][14][15][19]) SNET([16][18]) QD1. Are you aware of a sufficient number of high-level design alternatives? No. Try: Agent, Planning, Forward and Backward Satisfaction Approaches: NFR([9]) i* Satisfaction Analysis ([26][27][33]) Tropos Planning([4][6][7][8]) KAOS([31]) GRL Forward Satisfaction Analysis([1]) SNET Planning([16][18]) QD2. Are you aware of a sufficient number of detailed design alternatives? No. Try: Quantitative Planning, Forward and Backward Satisfaction Approaches: KAOS Satisfaction Analysis ([31]) GRL Forward Satisfaction Analysis([1]) Tropos Planning([6][7]) SNET Planning([16][18]) QD3. Do you need to evaluate and choose between high-level design alternatives? Yes. Try: Satisfaction Analysis, Metrics and Agent Approaches: KAOS Satisfaction Analysis([31]) i* Forward Satisfaction([26][33]) GRL Satisfaction Analysis([1]) i* Metrics([11][12][13]) Tropos Risk([4]) QD4. Do you need to evaluate and choose between detailed design alternatives? Yes. Try: Quantitative or Detailed Information: Tropos Probabalistic Satisfaction Analysis ([3][21][22][23]) KAOS Satisfaction Analysis ([31]) GRL Quant. Analysis ([1]) i* Quant. Metrics ([11][12][13]) Tropos Planning ([4][6][7][8]) Tropos Modeling Checking ([8][14][15][19]) SNET([16][17][18][18]) i* Simulation([34]) QD5. Do you need to find acceptable processes? QD6. Do you need to test run-time operation before implementation? Yes. Try: Planning Approaches: Tropos Planning([4][6][7][8]) SNET Planning([16][18]) Yes. Try: Simulation Approaches: SNET([16][17][18]) i* Simulation([34]) Analyzing Goal Models, Horkoff & Yu 24 Guideline Usage Examples Example: Online Counseling Domain (Horkoff & Yu, 2009) Online counseling alternatives: text messaging or chat room? Apply guiding questions… High degree of social interaction (QU1) Do no yet understand details, not yet confident in the accuracy and completeness of models (Qu2, QM1) Communication is important, scoping is challenging (QS1, QU2) Etc… Recommendations: Interactive, agent-oriented techniques for forward satisfaction analysis supporting softgoals Analysis for anonymity or privacy with the same techniques or with GRL Satisfaction Analysis, and/or i* Metrics If the required detailed information is available, apply planning and/or simulation techniques Analyzing Goal Models, Horkoff & Yu 25 Conclusions First step towards making goal model analysis techniques more accessible to modelers Enable potential users to user their knowledge of the domain and analysis objectives to select one or more procedures We have attempted to be neutral in our analysis Each procedure has unique abilities Future work is needed to undertake studies to validate or refute the claims made by our guidelines Hope that guidelines will be expanded and refined as more application experiences are available Analyzing Goal Models, Horkoff & Yu 26 Thank you www.cs.utoronto.ca/~jenhork jenhork@cs.utoronto.ca www.cs.utoronto.ca/~eric yu@ischool.utoronto.ca Analyzing Goal Models, Horkoff & Yu 27 Outline Goal Models Goal Model Analysis Motivation: Abundance of Approaches Survey of Goal Model Analysis Approaches Survey Results Summary Objectives of Goal Model Analysis Mapping of Procedures to Objectives Example Selection Conclusions & Future Work Analyzing Goal Models, Horkoff & Yu 28 Survey of GORE Analysis Techniques: Selection Article selection: Started with a set of known relevant papers Alternative selection methods: Linked work through references Stopped with picture of breadth was captured (24 papers) Search for specific key words… … in specific journals, conferences, portals … during specific time periods Challenge: work in goal model analysis appears in a range of venues with a range of keywords Venues: Books, RE, REJ, Agent-related conferences, CAiSE, AI related journal, FSE, PoEM, Journal of Information Systems, Trust-related Conference, ASE, etc… Keywords: agent-oriented software development, goal-oriented requirements analysis, early requirements analysis, multi-agent systems, agent-oriented software engineering, agent-oriented methodologies, risk analysis, countermeasure identification, goal modeling, goal-oriented analysis, quality metrics, etc…. Analyzing Goal Models, Horkoff & Yu 29 Guideline Usage Examples Example: Wireless service from Amyot et al. 2010 New wireless service added to existing network Where should the data and service be located? Apply guiding questions… Domain contains some interacting systems (QU1), no emphasis on communication (QC1) Aware of alternatives, but need to select one (QD1, QD3) Do not yet understand details, detailed alternatives, don’t have access to specific information, don’t want to find processes or perform simulations (QU2, QR2, QD2, QD5, QD6) Domain is well understood, scope is clear, models are sufficiently complete (QE1, QS1, QM1) Must consider non-functional requirements (QE3), data privacy (QR1) Recommendations: Agent-oriented approaches supporting softgoals to consider social and non-functional nature of the problem Satisfaction analysis or metrics to chose between alternatives Analyzing Goal Models, Horkoff & Yu 30 Guideline Usage Examples Example: Online Counseling Domain from Horkoff & Yu, 2009 Online counseling alternatives: text messaging or chat room? Apply guiding questions… High degree of social interaction (QU1) Do no yet understand details, not yet confident in the accuracy and completeness of models (Qu2, QM1) Communication is important, scoping is challenging (QS1, QU2) Consider many non-functional requirements, privacy and security especially, capture assumptions (QE3, QE4, QR1) Need to find and evaluate alternatives (QD1, QD3) Could be useful to find the most successful process (plan) for counseling or simulate throughput (QD4, QD5) Recommendations: Interactive, agent-oriented techniques for forward satisfaction analysis supporting softgoals Analysis for anonymity or privacy with the same techniques or with GRL Satisfaction Analysis, and/or i* Metrics If the required detailed information is available, apply planning and/or simulation techniques Analyzing Goal Models, Horkoff & Yu 31