A Method for Selecting the Best Performance Systems (original) (raw)
Related papers
Selecting the Best System Using the Three Stage and the Four-Stage Selection Approaches 1
2012
Statistical selection approaches are used to select the best stochastic system from a flnite set of alternatives. The best system will be the system with minimum or maximum performance measure. We consider the problem of selecting the best system when the number of alternative systems is huge. Three-Stage and Four-Stage selection approaches are proposed to solve this problem. The main strategy in these two selection approaches involves a combination method of cardinal and ordinal optimization. Ordinal optimization procedure is used to reduce the number of systems in the search space such that to be appropriate for cardinal optimization procedures. Three-Stage selection approach consists three procedures; Ordinal Optimization, Subset Selection and Indifierence-Zone. While, Four-Stage selection approach consists four procedures; Ordinal Optimization, Optimal Computing Budget Allocation, Subset Selection and Indifierence-Zone. In this paper, we compare the performance between the two s...
Selecting the best stochastic systems for large scale engineering problems
International Journal of Electrical and Computer Engineering (IJECE), 2021
Selecting a subset of the best solutions among large-scale problems is an important area of research. When the alternative solutions are stochastic in nature, then it puts more burden on the problem. The objective of this paper is to select a set that is likely to contain the actual best solutions with high probability. If the selected set contains all the best solutions, then the selection is denoted as correct selection. We are interested in maximizing the probability of this selection; P(CS). In many cases, the available computation budget for simulating the solution set in order to maximize P(CS) is limited. Therefore, instead of distributing these computational efforts equally likely among the alternatives, the optimal computing budget allocation (OCBA) procedure came to put more effort on the solutions that have more impact on the selected set. In this paper, we derive formulas of how to distribute the available budget asymptotically to find the approximation of P(CS). We then present a procedure that uses OCBA with the ordinal optimization (OO) in order to select the set of best solutions. The properties and performance of the proposed procedure are illustrated through a numerical example. Overall results indicate that the procedure is able to select a subset of the best systems with high probability of correct selection using small number of simulation samples under different parameter settings. This is an open access article under the CC BY-SA license.
Two sequential algorithms for selecting one of the best simulated systems
We consider the problem of selecting one of the best simulated systems when the number of alternatives is large. We propose two sequential algorithms for selecting a good enough simulated system based on the idea of ordinal optimization that focuses on ordinal rather the cardinal of the competent systems. In the first algorithm, we use the idea of ordinal optimization together with the Ranking and Selection procedure. In the second algorithm, we use the ordinal optimization with the optimal computing budget allocation algorithm. Numerical experiments for comparing these algorithms are presented.
Comparison of selection rules for ordinal optimization
Mathematical and Computer Modelling, 2006
The evaluation of performance of a design for complex discrete event systems through simulation is usually very time consuming. Optimizing the system performance becomes even more computationally infeasible. Ordinal optimization (OO) is a technique introduced to attack this difficulty in system design by looking at "order" in performances among designs instead of "value" and providing a probability guarantee for a good enough solution instead of the best for sure. The selection rule, known as the rule to decide which subset of designs to select as the OO solution, is a key step in applying the OO method. Pairwise elimination and round robin comparison are two selection rule examples. Many other selection rules are also frequently used in the ordinal optimization literature. To compare selection rules, we first identify some general facts about selection rules. Then we use regression functions to quantify the efficiency of a group of selection rules, including some frequently used rules. A procedure to predict good selection rules is proposed and verified by simulation and by examples. Selection rules that work well most of the time are recommended.
Chapter 17 Selecting the Best System
Simulation, 2006
We describe the basic principles of ranking and selection, a collection of experimentdesign techniques for comparing "populations" with the goal of finding the best among them. We then describe the challenges and opportunities encountered in adapting ranking-and-selection techniques to stochastic simulation problems, along with key theorems, results and analysis tools that have proven useful in extending them to this setting. Some specific procedures are presented along with a numerical illustration.
A quantile-based approach to system selection
European Journal of Operational Research, 2010
We propose a quantile-based ranking and selection (R&S) procedure for comparing a finite set of stochastic systems via simulation. Our R&S procedure uses a quantile set of the simulated probability distribution of a performance characteristic of interest that best represents the most appropriate selection criterion as the basis for comparison. Since this quantile set may represent either the downside risk, upside risk, or central tendency of the performance characteristic, the proposed approach is more flexible than the traditional mean-based approach to R&S. We first present a procedure that selects the best system from among K systems, and then we modified that procedure for the case where K − 1 systems are compared against a standard system. We present a set of experiments to highlight the flexibility of the proposed procedures.
Subset selection of best simulated systems
Journal of The Franklin Institute-engineering and Applied Mathematics, 2007
In this paper, we consider the problem of selecting a subset of k systems that is contained in the set of the best s simulated systems when the number of alternative systems is huge. We propose a sequential method that uses the ordinal optimization to select a subset G randomly from the search space that contains the best simulated systems with high probability. To guarantee that this subset contains the best systems it needs to be relatively large. Then methods of ranking and selections will be applied to select a subset of k best systems of the subset G with high probability. The remaining systems of G will be replaced by newly selected alternatives from the search space. This procedure is repeated until the probability of correct selection (a subset of the best k simulated systems is selected) becomes very high. The optimal computing budget allocation is also used to allocate the available computing budget in a way that maximizes the probability of correct selection. Numerical experiments for comparing these algorithms are presented.
Ordinal optimization: A nonparametric framework
Proceedings of the 2011 Winter Simulation Conference (WSC), 2011
Simulation-based ordinal optimization has frequently relied on large deviations analysis as a theoretical device for arguing that it is computationally easier to identify the best system out of d alternatives than to estimate the actual performance of a given design. In this paper, we argue that practical implementation of these large deviations-based methods need to estimate the underlying large deviations rate functions of the competing designs from the samples generated. Because such rate functions are difficult to estimate accurately (due to the heavy tails that naturally arise in this setting), the probability of mis-estimation will generally dominate the underlying large deviations probability, making it difficult to build reliable algorithms that are supported theoretically through large deviations analysis. However, when we justify ordinal optimization algorithms on the basis of guaranteed finite sample bounds (as can be done when the associated random variables are bounded), we show that satisfactory and practically implementable algorithms can be designed.
2012
Consider the optimization of a stochastic system in which the objective is to find a set of controllable system parameters that minimize some performance measure of the system. This situation is representative of many real-world optimization problems in which random noise is present in the evaluation of the objective function. In many cases, the system is of sufficient complexity so that the objective function, representing the performance measure of interest, cannot be formulated analytically and must be evaluated via a representative model of the system. In particular, the use of simulation is emphasized as a means of characterizing and analyzing system performance. The term simulation is used in a generic sense to indicate a numerical procedure that takes as input a set of controllable system parameters (design variables) and generates as output a response for the measure of interest. It is assumed that the variance of this measure can be reduced at the expense of additional computational effort, e.g., repeated sampling from the simulation. Applications involve the optimization of system designs where the systems under analysis are represented as simulation models, such as those used to model manufacturing systems, production-inventory situations, communication or other infrastructure networks, logistics support systems, or airline operations. In these situations, a search methodology is used to drive the search for the combination of values of the design variables that optimize a system measure of performance. A model of such a stochastic optimization methodology via simulation is depicted in Figure 1.1. 16. SECURITY CLASSIFICATION OF: 19a. NAME OF RESPONSIBLE PERSON James W. Chrissis, AFIT/ENS a. REPORT
Stochastic dominance based comparison for system selection
European Journal of Operational Research, 2012
We present two complementing selection procedures for comparing simulated systems based on the stochastic dominance relationship of a performance metric of interest. The decision maker specifies an output quantile set representing a section of the distribution of the metric, e.g., downside or upside risks or central tendencies, as the basis for comparison. The first procedure compares systems over the quantile set of interest by a first-order stochastic dominance criterion. The systems that are deemed nondominant in the first procedure could be compared by a weaker almost first-order stochastic dominance criterion in the second procedure. Numerical examples illustrate the capabilities of the proposed procedures.