Does choice of multicriteria method matter? An experiment in water resources planning (original) (raw)

1992, Water Resources Research

Many multiple criteria decision making methods have been proposed and applied to water planning. Their purpose is to provide information on tradeoffs among objectives and to help users articulate value judgments in a systematic, coherent, and documentable manner. The wide variety of available techniques confuses potential users• causing inappropriate matching of methods with problems. Experiments in which water planners apply more than one multicriteria procedure to realistic problems can help dispel this confusion by testing method appropriateness, ease of use, and validity. We summarize one such experiment where U.S. Army Corps of Engineers personnel used several methods to screen urban water supply plans. The methods evaluated include goal programming, ELECTRE I, additive value functions, multiplicative utility functions, and three techniques for choosing weights (direct rating, indifference tradeoff, and the analytical hierarchy process). Among the conclusions we reach are the following. First, experienced planners generally prefer simpler, more transparent methods. Additive value functions are favored. Yet none of the methods are endorsed by a majority of the participants; many preferred to use no formal method at all. Second, there is strong evidence that rating, the most commonly applied weight selection method, is likely to lead to weights that fail to represent the trade-offs that users are willing to make among criteria. Finally, we show that decisions can be as or more sensitive to the method used as to which person applies it. Therefore, if who chooses is important, then so too is how a choice is made. The first is to provide information on trade-offs by displaying how options perform on the various criteria. Trade-off displays can help users to better understand the nature of the choices they face. The information can also be used to eliminate alternatives that are dominated in every criterion by another alternative. This general role is especially important in public sector problems. This is because the political process needs information, not a single "answer" based on assumptions and value judgments more appropriately left to negotiation among the interests involved [Cohon, 1978]. The second is to help users articulate and apply their values to the problem rationally and consistently and to document the process. The object is to inspire confidence in the soundness of the decision without being unnecessarily difficult. An example is the use of additive value functions to rank alternatives. The National Environmental Policy Act of 1970 gave strong impetus to this use of these methods in the United States because of court decisions that ruled that recommendations made in environmental impact statements must be based on a systematic balancing of the effects. A related application of multicriteria methods is as an aid to negotiation: they can quantify and communicate the values Copyright 1992 by the American Geophysical Union. Paper number 92WR00712. 0043-1397/92/92WR-00712505.00 held by different people or interest groups [Brown, 1984, 1990]. Although multiobjective planning is no longer enshrined in U.S. federal guidelines for water planning [U.S. Water Resources Council, I983], water planners still use multicriteria methods for both of these purposes [Hobbs et al., 1989]. In this paper we focus on multicriteria methods for eliciting and applying value judgments. In selecting a multicriteria method for this purpose the user should be concerned with whether the method yields the information desired, its appropriateness to how the organization makes decisions, how easy the method is to use, and its validity. By "valid," we mean that the method is likely to yield choices that accurately reflect the values of users. A related concern is whether the results of different methods significantly differ. If the answer is "yes," then validity is crucial. Users then need to think about which method is likely to yield the most valid results. Unfortunately, answers to these questions are disputed. Each method has its champions, and their conflicting claims confuse potential users. To help with this problem, researchers have compiled methods and compared their theoretical properties [e.g., Chankong and Haimes, 1983; Cohon, 1978; Goicoechea et al., 1982; Hobbs, 1979]. But theoretical arguments cannot resolve all disputes concerning method performance. Experiments are often better than reasoning in determining (1) user perceptions of the appropriateness of techniques, (2) difficulty of use, (3) the relative validity of methods as actually applied, and (4) whether choice of method makes a practical difference. Ideal experiments in multicriteria decision making, whether in-1767