Focussing Multi-objective Software Architecture Optimization Using Quality of Service Bounds Anne Koziolek, Qais Noorshams, and Ralf Reussner Karlsruhe Institute of Technology, Karlsruhe, Germany {anne.koziolek,noorshams,reussner}@kit.edu http://sdq.ipd.kit.edu/ Abstract. Quantitative prediction of non-functional properties, such as performance, reliability, and costs, of software architectures supports systematic software engineering. Even though there usually is a rough idea on bounds for quality of service, the exact required values may be unclear and subject to trade-offs. Designing architectures that exhibit such good trade-off between multiple quality attributes is hard. Even with a given functional design, many degrees of freedom in the software architecture (e.g. component deployment or server configuration) span a large design space. Automated approaches search the design space with multi-objective metaheuristics such as evolutionary algorithms. However, as quality prediction for a single architecture is computationally expensive, these approaches are time consuming. In this work, we enhance an automated improvement approach to take into account bounds for quality of service in order to focus the search on interesting regions of the objective space, while still allowing trade-offs after the search. We compare two different constraint handling techniques to consider the bounds. To validate our approach, we applied both techniques to an architecture model of a component-based business information system. We compared both techniques to an unbounded search in 4 scenarios. Every scenario was examined with 10 optimization runs, each investigating around 1600 architectural candidates. The results indicate that the integration of quality of service bounds during the optimization process can improve the quality of the solutions found, however, the effect depends on the scenario, i.e. the problem and the quality requirements. The best results were achieved for costs requirements: The approach was able to decrease the time needed to find good solutions in the interesting regions of the objective space by 25% on average. Keywords: Optimization, Performance, Quality Attribute Prediction, Reliability, Software Architecture 1 Introduction The design of software architecture is crucial to exhibit good quality of service (cf. [3]), e.g. performance and reliability. Model-driven, quantitative architecture evaluation approaches help the software architect to reason about the architecture and predict its quality attributes and costs. However, even though there 2 Koziolek, Noorshams, Reussner usually is a rough idea of requirements for the non-functional properties, the exact required values may be unclear and subject to trade-offs. For example, the decision of how much response time of the system is acceptable may depend on the costs to achieve this response time and is subject to negotiation between stakeholders. Still, they may agree on bounds specifying the worst acceptable values of the quality attributes, e.g. the mean response time of the system should not exceed 15 seconds. A system that violates any bounds is declared infeasible, i.e. useless for the stakeholders. Designing architectures that provide optimal trade-offs between multiple quality attributes is difficult. Even with a given functional design, many degrees of freedom in the software architecture (e.g. component deployment or server configuration) still span a large design space. Automated approaches support the software architect to improve their architectural designs and find good trade-offs between quality attributes. They search the design space with multiobjective metaheuristics such as evolutionary algorithms to find many Paretooptimal candidates. However, as quality prediction for a single architecture is computationally expensive, these approaches are time consuming since many possible candidates need to be evaluated. In this work, we present an approach to include bound estimations on quality of service requirements into an automated improvement approach to make the search for optimal trade-offs focus on interesting regions of the objective space. We extend the PerOpteryx approach [15] by two aspects: First, we translate requirements specified with the Quality of service Modeling Language (QML) [12] into constraints in an optimization problem. Second, we use two constraint handling strategies [10, 11] to focus the search on the feasible space. The contribution of this paper is a novel approach that, to the best of our knowledge, is the first to combine multi-criteria architecture optimization and quality of service bounds so that the search can focus on feasible regions of the search space. With this extension, the time needed to find valuable solutions for the software architects can be reduced. We have implemented the approach in the PerOpteryx tool. Using this tool, we demonstrate the benefits of our approach in a case study. This paper extends a previous publication [19] by providing (1) the integration of a second constraint handling technique [11] and (2) more sound evaluation including the second technique, using more optimization runs and four different quality requirement scenarios. The evaluation leads to results with higher statistical significance and a more differentiated interpretation of the approaches’ effects. We found that the constraint handling is beneficial in scenarios with strict quality bounds (i.e., where many candidates are infeasible). In these scenarios, our extension was able to find solutions in the interesting regions of the objective space in average 25.6% faster than the old, unconstrained approach. This paper is structured as follows: Section 2 presents related work to our approach. Section 3 gives background on the architecture evaluation approach Palladio that we use in this work. Section 4 then presents our architecture optimization process, which makes use of the specified bounds to focus the search Focussing Multi-objective Software Architecture Optimization 3 on the feasible architecture candidates. A case study in Section 5 shows the feasibility of our work by applying the process to an example architecture and comparing the effect of the requirements consideration. Finally, Section 6 concludes. 2 Related Work Our approach is based on performance prediction [2], reliability prediction [13], multi-objective metaheuristic optimization [8], and constraint handling in evolutionary algorithms [9, 5]. A survey of constraint handling techniques is omitted here for brevity, but can be found in [18]. In summary, several other approaches to automatically improve software architectures for one or several quality properties have been proposed. Most approaches improve architectures by either applying predefined improvement rules, or by applying metaheuristic search techniques. All approaches except one do not support trade-off between quality attributes after the search. In addition, none of the approaches allows specifying quality requirements for quality attributes that should be optimized, thus, they do not allow to focus on interesting regions of the objective space. Xu et al. [20] present a semi-automated approach to improve performance. Based on a layered queueing network (LQN) model, performance problems (e.g., bottlenecks, long paths) are identified in a first step. Then, mitigation rules are applied. The search stops as soon as specified response time or throughput requirements are met. The approach is limited to performance only. The ArchE framework (McGregor et al. [16]) assists the software architect during the design to create architectures that meet quality requirements. It provides the evaluation tools for modifiability or performance analysis, and stepwise suggests modifiability improvements depending on the yet unsatisfied requirements. The search stops as soon as specified requirements are met. Canfora et al. [7] optimize service composition costs using evolutionary algorithms while satisfying service level agreement (SLA) constraints. They implement constraint handling with dynamic penalty functions. Menascé et al. [17] generate service-oriented architectures that satisfy quality requirements, using service selection and architectural patterns. They model the degree of requirement satisfaction as utility functions. Then, a weighted overall system utility is optimized in a single-objective problem using randomrestart hill-climbing. Thus, preferences for quality attributes and importance of requirements have to be specified in advance. Aleti et al.[1] present a generic framework to optimize architectural models with evolutionary algorithms for multiple arbitrary quality properties, thus enabling trade-off after the search. In addition, the framework allows to specify constraints for the search problem, for example available memory consumption. However, the constraint handling is relatively simple: Infeasible candidates are just discarded. Quality requirements are mentioned, but not included in the optimization. 4 3 Koziolek, Noorshams, Reussner Palladio Component Model Generally, our concepts can be used for different software architecture models. To a certain extent, service-oriented architectures can be regarded as a specialization of component-based software architectures. As a consequence, we focus the scope of our work on component-based software architectures. We apply our approach to the Palladio Component Model (PCM) [4], a modelling language for component-based software architectures with an UMLlike syntax. The PCM enables the explicit definition of the i) components , ii) architecture, iii) allocation, and iv) usage of a system in respective artefacts, which comprise a PCM instance (cf. Figure 1): 1. Component specifications contain an abstract, parametric description of components. Furthermore, the behaviour of the components is specified using a syntax similar to UML activity diagrams. 2. An assembly model defines the software architecture. 3. The resource environment and the allocation of components to resources are specified in an allocation model. 4. The usage model specifies usage scenarios. For each user, one of the scenarios applies defining the frequency and the sequence of interactions with the system, i.e. which system functionalities are used with an entry level system call. Using model transformations, the PCM instance can be analysed or simulated to predict performance (response time and throughput) [4], reliability (probability of failure on demand (POFOD)) [6], and costs [15] of a system. Assembly Model Component Specifications Usage Model Allocation Model PCM Instance Fig. 1. Artefacts of a PCM instance. Figure 2 illustrates an example PCM instance of the so-called business reporting system (BRS) using annotated UML. The BRS provides statistical reports about business processes and is loosely based on a real system. The system consists of 9 components and is allocated to 4 servers. The behaviour description (incl. CPU demands) of one component is illustrated here by an activity diagram. Having only one usage scenario, a user interacts with the system every 5s Focussing Multi-objective Software Architecture Optimization 5 requesting a sequence of reports and views. User requests take different paths through the system based on passed parameters, expressed here as probabilities. graphicalReport onlineReport S1 Webserver graphicalView onlineView user arrival login/logout every 5s S3 Business Reporting System Database S2 S4 Graphical Reporting Scheduler Core Graphic Engine Cache Online Reporting User Management Core Online Engine maintain <<implements>> Demand = 46.5 KInstr prepare P=0.1 Demand = 150 KInstr prepare detailed report P=0.9 Demand = 3.75 KInstr prepare simple report Call DB.getReport Loop 2 times Call Cache .getCachedData generate report Call DB.getReport Demand = 37.5 KInstr Fig. 2. PCM instance of the BRS (more detail in [15]). 4 Finding Satisfactory Architectures The goal of our work is to optimize component-based software architectures. To achieve this, we use metaheuristic techniques, particularly the multi-objective evolutionary algorithm (MOEA) NSGA-II developed by Deb et al. [10]. A disadvantage of a MOEA is that it may spend too much time exploring uninteresting regions of the objective space. Integrating quality requirements into the search aims at improving this algorithm due to the following advantages identified by Branke [5]: 1. Focus – MOEAs are approximate and non-deterministic. Quality requirements can be used to focus the search and identify particularly interesting alternatives. 2. Speed – Focusing the search avoids wasting computational effort on irrelevant regions of the search space. 6 Koziolek, Noorshams, Reussner 3. Gradient – With increasing number of objectives, MOEAs are unable to determine the most promising search direction (gradient). Quality requirements provide additional information ensuring optimization progress. 4.1 Constraint Handling To integrate requirements into this process, we extended the Opt4J framework [14], which implements basic NSGA-II without constraint handling, by the constrained tournament method (a.k.a. constrained NSGA-II) [10] and the goal attainment method [11], both described briefly below. Requirements are transformed into constraints. If a solution violates any constraint, it is infeasible, i.e. useless for the user. Otherwise, it is feasible, thus a possible candidate to solve the problem. In their constrained tournament method (C), Deb et al. [10] handle constraints by modifying the dominance relation during the mating and the environmental selection of NSGA-II. Infeasible solutions are ranked according to their degree of infeasibility and declared inferior to feasible solutions. In their goal attainment method (G), Fonseca et al. [11] define a goal value for each objective and aim at satisfying all goals by prioritizing objectives not fulfilling goals. Figuratively speaking, the Pareto-based comparison of two solutions is modified, such that before applying Pareto-dominance, the solutions are mapped on the goal value in the objectives that already fulfil the goal. Consequently, the objectives not fulfilling the goal have the impact on which solution dominates the other. Objectives for which no requirements exists are assigned a goal value of +∞ when minimizing or −∞ when maximizing. We chose these two methods for constraint handling because of the following advantages: First, they explicitly distinguish between feasible and infeasible solutions and declare all feasible solutions superior to infeasible solutions as opposed to e.g. methods based on penalty functions. Second, no additional parameters are required (an advantage because many other methods are sensitive to parameter changes). Finally, they neither require a specific number of constraints nor assume a relation between objectives and/or constraints. The difference of both methods is how solutions that violate the same constraints are treated: The constrained tournament technique uses a distance measure and favours solutions that are closer to the required values. In contrast, the goal attainment method uses standard Pareto-dominance if two solutions satisfy the same objectives. 4.2 Process Figure 3 illustrates the optimization process as a whole with four main steps: 1. The system to be optimized is modelled with the PCM. Additionally, the degrees of freedom, i.e. the possibilities to influence the non-functional properties of a system without changing its functional properties, are specified. In a component-based context, the degrees of freedom of a system can be Focussing Multi-objective Software Architecture Optimization 7 e.g. component selection, component deployment, and hardware configuration (cf. [15]), but this list is extendable to more and custom degrees of freedom. 2. Quality attributes (e.g. mean response time mrt) and quality requirements of the system (e.g. mrt< 5sec) are modelled using QML as described in [19]. Requirements are attached to a PCM usage scenario using a QML profile. 3. With our tool PerOpteryx, the models are used to optimize the system. The optimization starts with one or more initial candidates, i.e. predefined system configurations, which can also be created randomly. Optimizing quality attributes and minimizing costs is pursued using NSGA-II with consideration of the requirements, using either constrained tournament or goal-attainment. 4. As solving multi-objective optimization problems results in a set of solutions rather than one single solution [9], the set of feasible Pareto-optimal 1 architecture configurations with respect to the quality requirements is presented. Finally, the software architect makes the trade-off decision and chooses one of the solutions. Component-based software architecture 1 PCM Model 3 QML Profile Quality requirements 2 QML Model Optimization (MOEA) Initial candidate(s) Feasible, Pareto-optimal architecture configurations 4 Present Results Degrees of freedom Fig. 3. Process Overview. 5 Case Study This section describes a case study demonstrating the benefit of the consideration of requirements during the optimization process. The goal of this case study is to evaluate the benefits of the two constraint handling methods in different quality requirement scenarios. 5.1 Setup The system under study is the business reporting system (BRS) described in Section 3. The software architect has to choose a candidate that minimizes mean response time, probability of failure on demand (POFOD), and costs. As degrees of freedom, the components can be allocated to up to nine different servers. Three different webserver implementations with different costs, performance and reliability can be chosen. Additionally, each of the nine servers has a continuously variable CPU rate between 0.75 GHz and 3 GHz. The costs of the servers depend on the processing rate and the costs model is derived from Intel’s CPU price list. A power function is fitted to the data resulting in a costs 1 A solution x is Pareto-optimal if no other solution y is better than x w.r.t. all considered attributes (cf. [9]). 8 Koziolek, Noorshams, Reussner P P model of costs = i costs(i) = i 0.7665 p6.2539 [monetary units (MU)] with i the processing rate of each server pi [GHz]. The coefficient of determination is R2 = 0.965. Compared to [19], we used more realistic reliability values: the servers have a mean time to failure (MTTF) of 43800 hours and a mean time to repair (MTTR) of 3 hours. We get 19 (9+1+9) degree of freedom instances that can be independently varied. To study the effects of different quality requirement values on the results, we ran the optimization for four different levels of requirements (weak, i.e., only few candidates are excluded from the Pareto front, to strict, i.e., many candidates are excluded). Table 1 shows the four different scenarios. The requirements are modelled with our metamodel of QML [19]. For each scenario s ∈ {1, 2, 3, 4}, we optimized the system once for each constraint handling technique c ∈ {C, G}, resulting in 8 optimization settings Ssc , 1 ≤ s ≤ 4. As a baseline, we optimized the system without constraint handling (setting S0 ). costs POFOD mean response time Scenario (S1 ) Weak requirements 3000 0.00175 5.0 sec (S2 ) Medium requirements 2000 0.0015 3.0 sec 1500 0.0015 2.5 sec (S3 ) Strict requirements (S4 ) Only costs requirements 1000 ∞ ∞ Table 1. Quality Bound Scenarios For each of the 9 settings, the system is optimized using PerOpteryx. For statistical validity, we ran the optimization 10 times for every setting (runs r, 0 ≤ r ≤ 9), so that in total, 90 runs r Ssc have been performed. To exclude disturbing effects from differently generated random start populations, we randomly generated 10 start populations with 20 candidates each, and used these 10 start populations to run every setting (so that the rth run of any setting A starts with the same start population as the rth run of any setting B). Each optimization was stopped after 200 iterations. 5.2 Evaluation Measures To compare the performance of the different settings, quality indicators have been suggested in the literature. Due to the trade-off nature of multi-objective optimization, there is no single quality indicator that objectively assesses an optimization run’s performance [21]. The coverage metric C(A, B) [22] is a useful measure to compare two optimization runs A and B’s results independent of the scaling of the objectives. However, the metric may be misleading if the Pareto fronts overlap each other with varying distances to the true optimal Pareto front. Additionally, both directions C(A, B) and C(B, A) have to be considered to assess the difference of the fronts. To overcome both problems, we (1) measure size of the dominated space S(A) [22] to assess the quality of each Pareto front A separately and (2) modify the coverage metric C(A, B) to make it symmetric. Additionally, we include the quality bounds in the coverage metric, resulting in the following definition: Let A and B be feasible, non-dominated sets 2 and 2 In a non-dominated set, the elements are pairwise non-dominated (cf. [9]). Focussing Multi-objective Software Architecture Optimization 9 Q ⊆ A ∪ B be the feasible, non-dominated set of A ∪ B. The coverage metric C ∗ ∗ is defined as C ∗ (A, B) := |A∩Q| |Q| (∈ [0, 1]). If C (A, B) > 0.5 then A is considered better than B because A has a higher contribution to Q than B. The size of the dominated space S(A) measures the volume (in the three dimensional case) of the objective space weakly dominated by a Pareto front A. For minimisation problems, this measure requires a reference point to define the upper bounds of this volume. Here, we use the quality of service bounds and thus measure the size of the feasible space covered by A: S ∗ (A). For setting (4), which does not define upper bounds for response time and POFOD, we use the maximum values in all evaluated candidates of all runs as the upper bounds. Because the scale of the objectives are very different (POFOD ranges from 0, ..., 1, costs from 500 to 3500), and different upper bounds are used in the different settings, we normalize the objective values before determining the volume and, as a result, we cannot compare the absolute volumes across different settings. We analyse the coverage C ∗ of optimization runs with constraint handling over the basic optimization S0 and compare the size of the dominated space S ∗ . We study the effect of the constraint handling separately for each scenario 1 ≤ s ≤ 4. To study the development of the optimization runs, we plot the coverage measure over the course of the optimization, i.e. determine it for each iteration 0 ≤ i ≤ 200, written as C ∗ (A(i)) for a run A. Similarly, we compare the size of the dominated feasible space over the course of the optimization runs. The size of the feasible space dominated by the basic approach is determined anew for each scenario 1 ≤ s ≤ 4 with respect to the quality bounds of this scenario. Then, for each scenario s and each method c ∈ {C, G}, we aggregate the measures C ∗ (r Ssc ,r S0 (i)), S ∗ (r Ssc (i)), and S ∗ (r S0 (i)) over all 10 runs r to account for the indeterministic nature of the optimization. 5.3 Results Figure 4 illustrates the result of the optimization run 0 S3C with medium constraints using the constrained tournament method C. 7 Pareto-optimal candidates that satisfy all three bounds were found and are marked with triangles. We present the results in the following by scenario. Figures 5 and 6 show the coverage measure and the size measure for scenario 1. The coverage measure is around 0.5 in average over most of the iterations for both constraint handling methods C and G. With both measures, thus, no improvement towards the basic approach is visible. The size of the dominated feasible space grows similarly for all approaches, too. Figures 7 and 8 show the coverage measure and the size measure for scenario 2. For both the coverage measure and the size measure, the runs with constraint handling start well (coverage > 0.5 and size larger than size of basic approach). However, the basic approach catches up: At iteration 200, all approaches perform equally well (G has a slightly better coverage, C a slightly larger dominated space, so none performs better than the other). Figures 9 and 10 show the coverage measure and the size measure for scenario 3 with strict quality requirements. Here, we see an improvement of the search: Koziolek, Noorshams, Reussner Mean response time (mrt) in sec 10 POFOD ~0.002 POFOD req. = 0.0015 POFOD ~0.0014 Feasible, Paretooptimal solutions mrt requirement costs requirement Costs Fig. 4. Result of an optimization run 0 S3C with medium requirements s = 3 and the constrained tournament method c = C. The coverage measure of method C is higher that 0.5 during all iterations, and the size measure is significantly larger than for the basic approach, too. Method G does not perform as well, even has a coverage < 0.5 at the beginning while still having a better size measure than the basic approach. Finally, figures 11 and 12 show the results for the common case of a budgetonly limitation. While both constraint handling method do not perform well in the first 75 iterations, they catch up and provide better results in the last iterations, both regarding coverage and size measure. To summarise, we observe that the quality bounds have almost no effect in lowly constrained scenarios 1 and 2. In scenario 3, the constrained tournament method C performs well in both coverage and even more so regarding the size of the dominated feasible space. The goal attainment method is less successful. In scenario 4, both constraint handling methods perform well. We conclude that using quality bounds to focus the search is only effective if a large portion of the search space are excluded by the quality bounds, such as given in scenarios 3 and 4. In the two first scenarios, fewer solutions on the Pareto-front are infeasible, so that the constraint handling is seldom used and thus cannot steer the search well. Because it is not necessarily known in advance whether given requirements are strict or lax, the constraint handling methods should always be used, as they do not worsen the performance of the search. Furthermore, we examined after how many iterations type runs with constraint handling find solutions equivalent to the final result of basic approach runs based on both quality indicators C ∗ and S ∗ . For both indicators, we first find the smallest j for C ∗ (r Ssc (j),r S0 (200)) = 0.5 or S ∗ (r Ssc (j)) > S ∗ (r S0 (200)), then we find the smallest i for C ∗ (S0 (i), S0 (200)) = 0.5 or S ∗ (r S0 (i)) > S ∗ (r S0 (200)). In other words, we compare the runs with constraint handling with the earliest Coverage C*(Sc1(i),S0(i)) Focussing Multi-objective Software Architecture Optimization 11 1 Mean for c = C Min for c = C Max for c = C Std dev for c = C Mean for c = G Min for c = G Max for c = G Std dev for c = G 0.75 0.5 0.25 0 0 25 50 75 100 125 150 175 200 Iteration i Hypervolume S*(Ssc) Fig. 5. Coverage measure C ∗ (r S1c ,r S0 (i)) in scenario 1, aggregated over runs r 0.09 0.08 0.07 0.06 0.05 0.04 0.03 0.02 Mean s = 1 c = C Mean s = 1 c = G Mean s = 0 Iteration i 0 25 50 Min s = 1 c = C Min s = 1 c = G Min s = 0 75 100 125 Max s = 1 c = C Max s = 1 c = G Max s = 0 150 175 200 Coverage C*(Sc2(i),S0(i)) Fig. 6. Size of the dominated space S ∗ (r S1c (i)) in scenario 1, compared to the basic scenario S(r S0 (i)), aggregated over runs r 1 Mean for c = C Min for c = C Max for c = C Std dev for c = C Mean for c = G Min for c = G Max for c = G Std dev for c = G 0.75 0.5 0.25 0 0 25 50 75 100 125 150 175 200 Iteration i Hypervolume S*(Ssc) Fig. 7. Coverage measure C ∗ (r S2c ,r S0 (i)) in scenario 2, aggregated over runs r 0.014 0.012 0.01 0.008 0.006 0.004 0.002 0 Mean s = 2 c = C Mean s = 2 c = G Mean s = 0 Min s = 2 c = C Min s = 2 c = G Min s = 0 Max s = 2 c = C Max s = 2 c = G Max s = 0 Iteration i 0 25 50 75 100 125 150 175 200 Fig. 8. Size of the dominated space S ∗ (r S2c (i)) in scenario 2, compared to the basic scenario S(r S0 (i)), aggregated over runs r Koziolek, Noorshams, Reussner Coverage C*(Sc3(i),S0(i)) 12 1 Mean for c = G Min for c = G Max for c = G Std dev for c = G Mean for c = C Min for c = C Max for c = C Std dev for c = C 0.75 0.5 0.25 0 Iteration i 0 25 50 75 100 125 150 175 200 Hypervolume S*(Ssc) Fig. 9. Coverage measure C ∗ (r S3c ,r S0 (i)) n scenario 3, aggregated over runs r 1.00E-03 Mean s = 3 c = C Mean s = 3 c = G Mean s = 0 8.00E-04 6.00E-04 Min s = 3 c = C Min s = 3 c = G Min s = 0 Max s = 3 c = C Max s = 3 c = G Max s = 0 4.00E-04 2.00E-04 0.00E+00 Iteration i --> 0 25 50 75 100 125 150 175 200 Coverage C*(Sc4(i),S0(i)) Fig. 10. Size of the dominated space S ∗ (r S3c (i)) in scenario 3, compared to the basic scenario S(r S0 (i)), aggregated over runs r 1 Mean for c = C Min for c = C Max for c = C Std dev for c = C Mean for c = G Min for c = G Max for c = G Std dev for c = G 0.75 0.5 0.25 0 0 25 50 75 100 125 150 175 200 Iteration i Hypervolume S*(Ssc) Fig. 11. Coverage measure C ∗ (r S4c ,r S0 (i)) in scenario 4, aggregated over runs r 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 Mean s = 4 c = C Mean s = 4 c = G Mean s = 0 Min s = 4 c = C Min s = 4 c = G Min s = 0 Max s = 4 c = C Max s = 4 c = G Max s = 0 Iteration i 0 25 50 75 100 125 150 175 200 Fig. 12. Size of the dominated space S ∗ (r S4c (i)) in scenario 4, compared to the basic scenario S(r S0 (i)), aggregated over runs r Focussing Multi-objective Software Architecture Optimization 13 0.4 0.3 0.2 Coverage 0.1 Size 0 s=1 s=1 s=2 s=2 s=4 s=4 c=C c=G c=C c=C c=C c=G Fig. 13. Time Savings iteration of basic approach runs where there is no change in solutions w.r.t. the i−j final iteration. We measure the relative time saving t = max(j,i) . As an exam2 c 2 ∗ 2 c ple, we compare Ss and S0 regarding the coverage C . Ss has an equivalent solution set than 2 Ssc (200) after 171 iterations. 2 S0 has the last changes in the solution set in iteration 192. Thus, the run with constraint handling found equiv= 10.9% faster. alent results 192−171 192 Figure 13 shows the relative time savings for scenarios 1, 2, and 4. In scenario 3, too few solutions were feasible and Pareto-optimal at the end, so that a sensible assessment of the time saving is not possible. We observe that for all scenarios, the constraint handling methods can find an equivalent front faster than the basic approach. The average time saving is 11.1% with respect to C ∗ and 11.8% with respect to S ∗ , and with the most time saving in scenario 4 with the constrained tournament method (30.3% for C ∗ and 21.0% for S ∗ , average 25.6%). In further experiments [18], we have also studied to add lower bounds indicating that a quality values is good enough so that further improvement does not bring additional benefit, i.e. that other quality properties should not be traded off for more improvement of this value. However, we found that including such lower bounds does not significantly improve the optimization performance, neither in isolation nor in combination with upper bounds as presented in this work. Note that while we assume minimisation problems in this work, maximization problems can be inverted and handled as well, so that minimal allowed values are translated to upper bounds in our approach. 6 Conclusion This paper presents a novel extension of multi-criteria architecture optimization to consider bounds for quality requirements so that the search can focus on feasible regions of the search space. We translate the quality requirements modelled in QML to constraints in an optimization problem. Then, we use existing constraint handling strategies to make the search focus on the feasible space. We compared the performance of two constraint handling strategies, namely constrained tournament methods and 14 Koziolek, Noorshams, Reussner the goal attainment method, in several scenarios in a case study. We found that constraint handling, especially the constrained tournament method, improves the efficiency of the search if strict requirements are used, i.e. if a significant portion of the objective space is defined to be infeasible. Additionally, we found that the constrained tournament method was superior to the goal attainment method in our setting. With this extension, software architects can reduce the time needed to find valuable solutions. Our extension found solutions in the interesting regions of the objective space in average between 15% and 30% faster than the old, unconstrained approach for scenario 4 with strict requirements. The application of this approach can be interesting in different phases of the software architecture design process. First, the approach can be applied after a first phase of creating an architecture with focus on functional requirements (definition of components and interfaces). This architecture can be used as an input for the optimization to improve the non-functional properties. Second, the optimization could already be used to support decisions during the architectural design: When making a more high level decision, the optimization can be used to assess the potential of the different alternatives. Finally, by modelling more high level decisions as transformations, these decisions could be included in the optimization process as degrees of freedom, thus letting the optimization explore different combinations of decisions. As future work, we could investigate the effect of constraint handling if other metaheuristic optimization approaches than NSGA-II are used. Additionally, we plan to integrate quality attribute tactics into the search, to allow the search to improve a given candidate using domain knowledge, e.g. by balancing the load on the used servers to improve response time. In combination with bounds, tactics could be used to more directedly steer the search towards feasible regions, which could be especially beneficial in highly constrained problems. Acknowledgments. The authors would like to thank the anonymous reviewers for their valuable feedback. References 1. Aleti, A., Björnander, S., Grunske, L., Meedeniya, I.: Archeopterix: An extendable tool for architecture optimization of AADL models. In: Proc. of ICSE 2009 Workshop on Model-Based Methodologies for Pervasive and Embedded Software (MOMPES). pp. 61–71. IEEE Computer Society (2009) 2. Balsamo, S., Di Marco, A., Inverardi, P., Simeoni, M.: Model-Based Performance Prediction in Software Development: A Survey. IEEE Transactions on Software Engineering 30(5), 295–310 (May 2004) 3. Bass, L., Clements, P., Kazman, R.: Software Architecture in Practice, Second Edition. Addison-Wesley, Reading, MA, USA (2003) 4. Becker, S., Koziolek, H., Reussner, R.: The Palladio component model for modeldriven performance prediction. Journal of Systems and Software 82, 3–22 (2009) 5. Branke, J.: Consideration of partial user preferences in evolutionary multiobjective optimization. In: Multiobjective Optimization: Interactive and Evolutionary Approaches. pp. 157–178. Springer-Verlag, Berlin, Heidelberg (2008) Focussing Multi-objective Software Architecture Optimization 15 6. Brosch, F., Koziolek, H., Buhnova, B., Reussner, R.: Parameterized Reliability Prediction for Component-based Software Architectures. In: Proc. of QoSA. LNCS, vol. 6093, pp. 36–51. Springer (2010) 7. Canfora, G., Penta, M.D., Esposito, R., Villani, M.L.: An approach for QoS-aware service composition based on genetic algorithms. In: Proc. of Genetic and Evolutionary Computation Conference (GECCO). pp. 1069–1075. ACM (2005) 8. Coello Coello, C.A., Dhaenens, C., Jourdan, L.: Multi-objective combinatorial optimization: Problematic and context. In: Advances in Multi-Objective Nature Inspired Computing, Studies in Computational Intelligence, vol. 272, pp. 1–21. Springer, Heidelberg (2010) 9. Deb, K.: Multi-Objective Optimization using Evolutionary Algorithms. John Wiley & Sons, Chichester, UK (2001) 10. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm : NSGA-II. Evolutionary Computation, IEEE Transactions on 6(2), 182–197 (August 2002) 11. Fonseca, C.M., Fleming, P.J.: Genetic algorithms for multiobjective optimization: Formulation, discussion and generalization. In: ICGA. pp. 416–423. Morgan Kaufmann (Jun 1993) 12. Frølund, S., Koistinen, J.: QML: A Language for Quality of Service Specification. Tech. Report HPL-98-10, Hewlett-Packard Laboratories (1998) 13. Gokhale, S.S.: Architecture-based software reliability analysis: Overview and limitations. IEEE Trans. on Dependable and Secure Computing 4(1), 32–40 (2007) 14. Lukasiewycz, M., Glaß, M., Reimann, F., Helwig, S.: Opt4J - The Optimization Framework for Java. http://www.opt4j.org (2010) 15. Martens, A., Koziolek, H., Becker, S., Reussner, R.: Automatically improve software architecture models for performance, reliability, and cost using evolutionary algorithms. In: Proc. of Proceedings of the first joint WOSP/SIPEW International Conference on Performance Engineering (WOSP/SIPEW). pp. 105–116. ACM, New York, NY, USA (2010) 16. McGregor, J.D., Bachmann, F., Bass, L., Bianco, P., Klein, M.: Using arche in the classroom: One experience. Tech. Rep. CMU/SEI-2007-TN-001, Software Engineering Institute, Carnegie Mellon University (2007) 17. Menascé, D.A., Ewing, J.M., Gomaa, H., Malex, S., Sousa, J.a.P.: A framework for utility-based service oriented design in SASSY. In: Proc. of Proceedings of the first joint WOSP/SIPEW International Conference on Performance Engineering (WOSP/SIPEW). pp. 27–36. ACM (2010) 18. Noorshams, Q.: Focusing the Optimization of Software Architecture Models Using Non-Functional Requirements. Master’s thesis, Karlsruhe Institute of Technology, Germany (2010) 19. Noorshams, Q., Martens, A., Reussner, R.: Using quality of service bounds for effective multi-objective software architecture optimization. In: QUASOSS ’10: Proceedings of the 2nd International Workshop on the Quality of Service-Oriented Software Systems. pp. 1:1–1:6. ACM, New York, NY, USA (2010) 20. Xu, J.: Rule-based automatic software performance diagnosis and improvement. Performance Evaluation 67(8), 585–611 (2010), special Issue on Software and Performance 21. Zitzler, E., Knowles, J., Thiele, L.: Quality Assessment of Pareto Set Approximations, LNCS, vol. 5252, pp. 373–404. Springer-Verlag, Berlin (2008) 22. Zitzler, E., Thiele, L.: Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Trans. Evolutionary Computation 3(4), 257–271 (1999)