advertisement

Draft Draft Draft Proposal for Data-Driven Hierarchical Decision Making I. Introduction The selection of alternative restoration plans and operational strategies for the highly complex system that is the Florida Everglades constitutes a multiobjective decision making problem of unprecedented dimension. From the perspective of the scientist, the system is characterized by uncertainty; of measurements, model representation and interconnectivity of the restoration objectives. For stakeholders, scientist or not, the evaluation of competing plans is impeded by myriad, often competing and likely overlapping criteria known as the Performance Measures (PMs). Decision making is further complicated by the lack of guidelines for weighting of the PMs, a subjective process subject to concerns regarding consistency, transparency and repeatability of results. Finally, the spatial extent of the system and spatial distribution of PMs, reliance on model results versus monitoring and uncertain sensitivity of ecological assets to perturbations of the PMs must be considering in restoration plan evaluations. This proposal will describe a two part process for evaluating restoration plan alternatives addressing the two major uncertainties that that impact selection. The first uncertainty relates to the information content of the PMs and data used to evaluate them. The second addresses the uncertainty associated with the weighting of the many PMs for use in scoring alternative plans. In accordance with this characterization of the decision problem, the proposal components are: Optimization of PM data information content in support of decision making. Methods to reduce uncertainty and characterize relationships, if they exist, between performance measures based on pattern recognition and noise reduction techniques, among others. Such techniques may be used to optimize the information content of the available data such that the subjective process of criteria weighting is based on true trade-offs. Analytical Hierarchical Process (AHP) incorporating the Delphi Method of decision making based on expert opinion elicitation. AHP systematizes the multi-objective decision making to enforce consistency while the Delphi Method uses an iterative process that engenders consensus and reduces uncertainty (spread) in judgments produced by diverse decision makers. The result is a multi-objective decision process specifically tailored for the complexities of the Florida Everglades restoration. -1- Draft Draft Draft II. Optimization of Information Content in Data Performance Measures and model output are used to evaluate the expected outcomes of different operational alternatives. Each has associated uncertainty and unrevealed connectivity that jeopardizes the meaningfulness of straightforward combinations and evaluations of resulting values. Furthermore, overlap and redundancy in measures, measures that are difficult to model, and obscuring of key causative processes due to a focus on endpoints may lead to decision making based on false tradeoffs and evaluation of inferior alternatives in a Pareto sense. For example, Performance Measures may appear nominatively independent as defined by stakeholders but in fact be highly correlated with PMs independently derived by separate stakeholders. Some PMs are distributed over space and time and it is not transparent that all locations or timeframes should be treated equally. Yet each PM is an indication of a particular impact that is important to at least one group of stakeholders and these evaluation points should not be discarded. For this reason a method is required that preserves the information content of each PM while accounting for these various sources of uncertainty. Tools are available that define, in a statistical sense, relationships between correlated and causally connected data, such as the PMs. One example of such a method is Principal Component Analysis (PCA). PCA is a mapping of a data set of n measures and m realizations from their original n x m dimensional space to a new n x m dimensional space that maximizes the modes of variability within the data set and is defined by orthogonal axes. Using PCA on the set of PMs, including spatially distributed PMs, will assign a weight to each PM objectively based on its contribution to each mode of variability. In addition, the use of PCA with a dataset with significant cross-correlations often yields the result that there are p modes that explain most of the variability of the measures and that p << n. This is a welcome result as it allows the subjective process of prioritizing PMs to be performed on a much smaller set of measures. The benefits of PCA as applied to the PMs extend beyond the reduction in dimension of performance measures. The nebulous interconnections between PMs defies identification of true tradeoffs. In that sense, it is not clear that any alternative is located on the Pareto frontier of non-inferior, feasible alternatives. At the Pareto frontier for a set of PMs, the PMs would be maximized to the extent that any further improvement in one via a decision variable necessitates a decrease in another. Identification of the Pareto frontier requires an orthogonal, or uncorrelated relationship between PMs. Figure 1 displays the hypothetical case of two orthogonal PMs and a hypothetical set of PM values for competing alternatives. In this case a lower value of the PM is better for each and thus the -2- Draft Draft Draft Pareto frontier is the subset of values that have no values located below and left of them. Realignment of the PMs along the orthogonal modes of maximum variability allows one to move from within the alternative space to the Pareto frontier, as depicted in the figure, although in actuality it is the axes that move. The PCA of PMs insures orthogonality of the resultant measures, which are linear combinations of the individual PM values. Any point along this frontier is optimal in the Pareto sense, although each point assigns a different priority or weight to each PM. Thus within this subset the optimal tradeoffs can be decided. The final benefit of the PCA of the PMs is the possible identification of key causal processes. Once the major modes of variability within the set of PMs are defined, one can analyze investigate these modes are linked to a key causal process, one that influences a large subset of PMs but may not have been identified as an important objective in and of itself. It is further possible that these key processes may be physically monitored and provide important feedback regarding the performance of selected alternatives and adaptively managed operation that would not be possible for a large set of spatially distributed PMs. A Bayesian network analysis will conducted to assess the causes of variability in the PM set and for identification of the key causal processes for each mode of variability. III. Criteria (Performance Measure) Weighting Once the available data has been processed to yield the transparent information content the second stage of the proposal can begin implemented. Here, again, there are two challenges: achieving consensus opinion for the ranking of the criteria, or PM, and quantification of that ranking. Returning the example of two orthogonal PM, one can envision this step as moving along the Pareto frontier to -3- Draft Draft Draft select the weightings, wi and wj that in turn reveal the optimal alternative. The Analytic Hierarchy Process will be used to quantify the subjective rankings. Within this process, the Delphi Method will be used for eliciting expert opinion. Each is described below. Analytic Hierarchy Process Analytic Hierarchy Process (AHP) is a quantitative technique of subjective judgments (Saaty, 1980; Saaty and Alexander, 1981). AHP has been successful in cases where multi-objectives (i.e., multiple criteria or multi-attribute decision making) have to be satisfied for decision making when there are several alternatives and a hierarchy of objectives does not exist. Let n again be the number of items (i.e., performance measures) being considered with the goal of providing and quantifying judgments on the relative weight, importance, and priority of each item with respect to the rest. In AHP, the problem is formulated in several layers, the first being the satisfaction of the objectives represents the top, followed by the criteria and finally the alternatives. The challenge that AHP solves is subjectively assigning ranks to rival criteria, i.e., the performance measures. While this subjective element can not be eliminated from multi-objective decision making, AHP systematizes the process to insure consistency and transparency in the rankings The weighting process is conducted as follows. Pair-wise comparisons are carried out by stakeholders between all measures with respect to their contribution toward the overarching objectives of the restoration. Returning to our previous example, stakeholders or decision makers need to decide which of the two PMs in figure 1 is more important. The preference strength is expressed on a ratio scale of 1-9 (Saaty, 1980). The resulting matrix of the pairwise comparison A R nn is necessarily reciprocal with aii 1 and aij 1 a ji . If judgments are completely consistent then by providing the first row of A one could deduce the rest of the elements due to the transitivity of the relative importance of the items. In practice, consistency is not assumed and the process of comparison for each column of the matrix is carried out. Suppose that at the end of the comparisons, we have filled the matrix A with the exact relative weights; if we multiply the matrix with the vector of preferences w ( w1 , w2 ,..., wn ) we obtain: (1). Aw nw or A nIw 0 To recover the overall score from the matrix of ratios the homogeneous linear equations from (1) must be solved. Since n is an eigenvalue of A thus we would have a nontrivial solution, unique to within a multiplicative constant, with all positive entries. Small perturbation in the coefficient implies perturbations in the resulting eigenvalues. From the nature of the matrix the maximum eigenvalue has to equal n and its corresponding eigenvector w gives a unique estimate of the underlying ratio scale between the elements of the problem. The consistency -4- Draft Draft Draft index could be easily measured to evaluate the degree of uniqueness in the solution and it measure the error due to inconsistency. The consistency ratio (CR) reflects the consistency of the pair-wise judgments and shows the degree to which various sets of importance relativities can be reconciled into a single set of weights a to verify the goodness of the judgments (Satty, 1981; Ahti and Raimo, 2001). In literature there are threshold CR’s that could be estimated and they provide guidelines for recommendations regarding the pair-wise comparison matrix. This process is described for only one level in the hierarchy. Hierarchical composition should be used to combine all the levels. The sum is taken over all weighted eigenvector entries corresponding to those in the lower level, and so on, resulting in a global priority vector for the lowest level of the hierarchy. Implementation of AHP is shown schematically in figure 2. Delphi Method Participatory decision-making is frequently advocated for resolving issues involving common pool resources that have been traditionally decided by governments or their agencies. Participatory approaches are still subject to social asymmetries in power and access and may fall prey to the usual inequities common to centralized approaches, where certain economic or social classes, ethnic groups or genders dominate. In addition, it is not clear that the results of participatory decision-making are better than the results that would be derived from a single informed decision maker. Results may, in fact, be worse. Still, involving stakeholders in the decision process for resources that affect their livelihoods is deemed beneficial it its own right. The decrease in the accuracy or quality of decision is less than the benefit gained by including stakeholders in the decision. We make the assumption that stakeholder decision-making is desirable and will be a factor in most future common pool resource decisions. The questions we would like to address are: what elements of group decisionmaking degrade the quality of the decision; and with these elements identified, can the process be designed to improve the quality of the decision. We are especially interested in decision-making under uncertainty regarding natural resources, and we examine the specific case of water resources. While the adage “two heads are better than one” is generally accepted as true and could be extended to the “n heads” case, the way in which the heads interact may detract from the benefit of the increased number of participants. The Delphi Method evolved from this observation. It was developed by the RAND Corporation as a method to maximize the benefit of opinions from groups of experts on complex and uncertain questions. The Delphi Method was designed to minimize detrimental elements of group decision-making, namely the biasing effects of dominant individuals, irrelevant communications (noise) and group pressure toward conformity. This was accomplished through soliciting opinions -5- Draft Draft Draft on questionnaires, providing selective feedback to the group based on the first round answers, then repeating the solicitation of opinions with a modified questionnaire. Their trials found that the resulting answers were often more accurate than those produced by face to face meetings of experts in controlled studies. Dinar et al. (2004) used this approach to estimate the probability of success for irrigation reforms in Pakistan. -6- Draft Draft Draft 2 1 3 Ecological benefit quantification Analyze rank reversal using ANOVA and F-test (Problem recognition) Perform pairwise comparison Run PCA for evaluation of the relationships between PMs and their importance Define the probability distribution and probability matrix Estimate the impact of variation Replicate the eigenvectors Select a set of PM for further analyses (Monte Carlo Simulation or Bayesian networks) Perform "what if" scenario using sensitivity charts Select a group of subject matter experts for preference assignments Yes Is the consistency index < threshold? No 4 Perform "confirmatory AHP Refine the preference assignment Use sensitivity analysis Define scope and boundaries of AHP Hierarchy decomposition and strategy analysis Benefits quantification and alternative selection Eliminate insensitive simulations Analysis of confidence intervals Figure 2: Integrated framework for ecological benefit quantification. -7- Draft Draft Draft References: Saaty, T.L.,1980. The Analytic Hierarchy Process. McGraw Hill, New York. Saaty, T.L., and J. M. Alexander, 1981. Thinking with Models. Pergamon Press. Banuelasy, R. and J. Antony, 2004. Modified analytic hierarchy process to incorporate uncertainty and managerial aspects. International Journal of Production Research, 42 (18), pp: 3851–3872. Ahti A. S. and P. H. Raimo, 2001. Preference Ratios in Multiattribute Evaluation (PRIME)—Elicitation and Decision Procedures Under Incomplete Information. IEEE Transactions on Systems, Man, and Cybernetics, 31 (6). Notes ulall: I have it as follows --> All PMs derive from water variables and are typically monotonic functions of them...hence we know that they will be correlated, so it makes sense to come up with a weighted measure that reduces them to an index. However, there may be a hierarchy of relations between them and some may be negatively correlated (e.g., flood reduction and ecological releases), so important to filter out how they are related as well and present it --- this corresponds to your data analysis step. Underlying is the thought that the importance variable or index needs to be connected back to simple observable and measurable hydrologic and ecologic state variables, not just model computed stuff. So, if we can compare model hydro variable (time series) with observed and characterize ulall: reliability/uncertainty in the primary variable, then we can propagate that into the PM and hence into the weighted PM based index, using Bayes nets, thus recognizing explicitly how we connected the many pieces. Even though they have 10 or so PMs now, these are defined at many spatial locations and over time, so aggregation into an index that can be logically disaggregated and maintaining the hierarchy of relations -- ie recovering where the index is coming from and how each measurable piece simultaneously impacts many PMs is important to reconstruct ulall: We can do the reduction via PCA on model time series of PMs. In this case we'll have +ve and -ve coeffs in each eigenvector and we focus on identifying simultaneous +ve and -ve impacts -- for obviously +ve and -vely correlated variables this makes sense, but the eigencoeffs can be + and - for even +vely correlated variables which would make subsequent interpretation difficult. ulall: So, at this stage we can use also Archetype Analysis which is like PCa except that it uses only +ve weights and then we can say that a particular PC is composed of x1, x2, xi % of each each of the PMs. Of course if we know that the PMs fall in 2 groups -- mutually +vely and mutually -vely correlated then we can just multiply one group by a -ve and then have archetype PCs that are meaningful. ulall: So, in the first stage, we use a dimension reduction step -- likely PCA or archetype but also explore BNs -8- Draft Draft Draft caseyobrown: got it so far. am thinking delphi method used to come to consensus for interpretation of pcs after our initial cut prior to AHP, make sure raters are using same info ulall: yes --!! however, we still need to deal with space and time to some extent -ulall: lets take time -- they will have one PM number from 1 run I think which will be the total time average for that PM over the model run for that spatial box or region caseyobrown: ok ulall: so, ideally if PM=PMbase or value for base conditions, it would be good to know the number of years in which PMalt >PMbase caseyobrown: ok ulall: so ideally I would want to compute PM as a time series for all alts including base and then use the diff from base to develop my aggregated PM score -- and then I also have its PDF instead of just mean value ulall: so the idea is that we could then convert the PC of the PMs (defined as this diff PM) into a pdf as well caseyobrown: ok i see this, but problem is pc < or> not clear which is better possibly ulall: Then we are still stuck with as many such pdfs for each PM as there are spatial locations caseyobrown: but can ignore this as detail for later ulall: well -- the simplest case is that the mean of the PC (of PM) time series is their equivalent score or index ulall: and we now have a pdf for that score -- I expect that the pdf is more for our benefit to design an internal algorithm than for overt discussion with stakeholders caseyobrown: ok, yep, remembering we're in time dimension only. ulall: yes space is messy ulall: so now space -- first thought is that one should average over space for each time step -- but how should such an average be computed caseyobrown: within single PM? ulall: 1) arith mean of the diff based PM at each location ? -- this gives me a time series for each PM averaged over space ulall: yes within single PM -- else hard to make any sense caseyobrown: PCA provides weight according to effect on variance-seems good ulall: however, we expect that there may be strong correlations across space within a PM -some areas are all +vely impacted while others are -vely impacted -9- Draft Draft Draft caseyobrown: and some pms are deadends, others key points that influence downstream ulall: yes u r right so 2) is to take PCA across space using time correlation for a given PM and that gives us our PM time series aggregated over space ulall: then we do pca across these spatial PCs for PMs -- fear is that if the leading PC or 2 does not pick up most of the variance we are starting to lose spatial detail and then pm detail ulall: but these are details to worry about when we work and not so much for proposal perhaps ulall: Now, if we can get stakeholders to see the degree of overlap between 2 alts in terms of leading pcpms, if one clearly dominates the other they may "vote" for that alt -- if it does not, then they will want to understand more, and perhaps revise even their weights on original PMs -where they can now be quickly shown how changing those weights impacts the PCPM structure. -- iek maintianing correspondence between aggregate and disaggregate representations while the hydro variable that is measured remains the same ulall: so this still sounds more like delphi than AHP in that we are trying to facilitate the process of their voting for alternatives rather than expressing prefs for PM weights or scores ulall: I think this is an important difference --since the AHp tryies to get weights for PMs which then are used to choose an optikmal solution and the stakeholder is asked to accept the blackbox caseyobrown: ok, i see. iterative with explanations of why choosing one over the other as effective of ranking pms caseyobrown: transparency and ability to iterate probably extremely important for acceptance ulall: whereas in Delphi with votes as to alternatives and discussion re intermediates and impacts of weights etc you proceed more directly towards participatory selection of the alternative rather than participatory slection of weights and then black box for solution caseyobrown: exactly ulall: so you agree -- good -- then we need to work this into the strategy in a coherent way ulall: and mention AHP and Delphi as possibly complementary but likely competing ways to address the problem -- complementary if weights pre-PCA are selected using AHP -- but not sure if that makes sense caseyobrown: maybe abed can answer that best. seems that still presents the problem of no consensus/negotiation on weighting if done prior to pca because meaning will not be clear -10-