Ranking and Selection Procedures for Bernoulli and Multinomial Data

Abstract

Ranking and Selection procedures have been designed to select the best system from a number of alternatives, where the best system is defined by the given problem. The primary focus of this thesis is on experiments where the data are from simulated systems. In simulation ranking and selection procedures, four classes of comparison problems are typically encountered. We focus on two of them: Bernoulli and multinomial selection. Therefore, we wish to select the best system from a number of simulated alternatives where the best system is defined as either the one with the largest probability of success (Bernoulli selection) or the one with the greatest probability of being the best performer (multinomial selection). We focus on procedures that are sequential and use an indifference-zone formulation wherein the user specifies the smallest practical difference he wishes to detect between the best system and other contenders.

We apply fully sequential procedures due to Kim and Nelson (2004) to Bernoulli data for terminating simulations, employing common random numbers. We find that significant savings in total observations can be realized for two to five systems when we wish to detect small differences between competing systems. We also study the multinomial selection problem. We offer a Monte Carlo simulation of the Bechhofer and Kulkarni (1984) MBK multinomial procedure and provide extended tables of results. In addition, we introduce a multi-factor extension of the MBK procedure. This procedure allows for multiple independent factors of interest to be tested simultaneously from one data source (e.g., one person will answer multiple independent surveys) with significant savings in total observations compared to the factors being tested in independent experiments (each survey is run with separate focus groups and results are combined after the experiment). Another multi-factor multinomial procedure is also introduced, which is an extension to the MBG procedure due to Bechhofer and Goldsman (1985, 1986). This procedure performs better that any other procedure to date for the multi-factor multinomial selection problem and should always be used whenever table values for the truncation point are available.