Methods · Sensitivity
Bootstrap Resampling
It redraws your set of alternatives, with replacement, many times over, rebuilds the ranking from scratch on every draw, and shows how much an alternative's rank depends on that randomness.
Base method's data type: Classical
What Is the Method?
Bootstrap Resampling is not a ranking method; it does not produce a ranking on its own. If you already have a decision table, criterion weights, and a ranking method run on the two of them (TOPSIS, SAW, VIKOR, whatever it may be), this method interrogates that ranking. Its question is simple: if your set of alternatives were slightly different, say some alternatives appeared twice in the sample and others not at all, how much would the ranking change? Its output is not a new ranking; it is a distribution of the rank each alternative received, that distribution's mean, and a ninety-five per cent interval.
The method's statistical root is the bootstrap principle Efron defined in 1979: the sample you hold is a single draw from a larger, unknown population; by drawing repeatedly from that sample itself, with replacement, you can estimate that population's uncertainty. Its adaptation to multi-criteria decision analysis is relatively recent; Rook, Hoos and Trautmann (2024) used bootstrap resampling directly to measure ranking robustness.
The Philosophy Behind It
The idea behind this method is that a decision analysis is only as trustworthy as its input. The alternatives in a decision table are usually not the full set of possible candidates but a subset judged worth examining; in another round of a call for tenders, another year, or another list of experts, this set could have come out slightly different. Bootstrap Resampling mimics this uncertainty: it redraws the real alternatives repeatedly with replacement, leaving some in the sample more than once and others out entirely, and reruns the same ranking method on every draw.
The consequence of this philosophy is that the method does not answer "which alternative is best"; it answers "how robust is that answer to small chance variation in the set of alternatives." If an alternative stays near the top in almost every draw, its superiority is solid. If an alternative's rank varies substantially from draw to draw, its rank in the original analysis was partly a matter of chance, and the report must say so.
How It Works
The method proceeds through five steps, all resting on running the chosen base ranking method repeatedly.
First, the base run. The base method is run once with the full decision table and weights you hold. The rank that emerges from this run is stored as the benchmark.
Second, drawing with replacement. From the table's m alternatives, m are drawn with replacement (an alternative may be drawn more than once, or not at all). This produces a sample table the same size as the original but different in content.
Third, rerunning. The base method is run again on this sample table, and that draw's rank is recorded. This is repeated B times (hundreds or thousands of times).
Fourth, summary statistics. All the draws are pooled. Each alternative's mean rank, the ninety-five per cent interval of its ranks, and the distribution of how often it landed at each rank are computed.
Fifth, overall agreement. The agreement among all the draws' rankings is reduced to a single number using Kendall's W coefficient; close to one indicates strong agreement among the draws, close to zero indicates substantial scatter.
The formulas behind each step and the intermediate tables are given on the DecisionMind method page; this card carries no formulas.
How to Read the Output
An alternative's mean rank and ninety-five per cent interval show which rank it can be expected to occupy; a narrow interval close to a single rank means the alternative's position is solid, while a wide interval spanning several ranks means its position is sensitive to chance. This figure is not a probability; instead of saying "this alternative is second with ninety-five per cent probability," the correct statement is "in ninety-five per cent of the samples drawn with replacement, this alternative's rank fell between these two values."
The single rank the base run produces does not by itself show how robust any alternative is; only the bootstrap distribution shows that. If two alternatives' mean ranks are close and their intervals overlap, the gap between them could be reversed by a small chance variation.
Thus instead of writing:
"The analysis proved A2 is the best alternative"
the report should read:
"With the table as given, A2 is the best alternative; in resampling with replacement, A2 stayed within the top two ranks in the large majority of draws, and that is what makes this position solid"
Data Type and Inputs
Bootstrap Resampling works with crisp data and needs no decision table of its own; what it needs is an already completed ranking analysis: a decision table, criterion directions, weights, and a base method run on them. In addition, the number of draws (B) must be set; a few hundred to a few thousand is usually sufficient. DecisionMind holds no separate extension of this method; it is used alongside Weight Sensitivity Analysis, which asks the same question through the weights, and Kendall's τ and Kendall's W, which reduce the agreement between two ranks to a single number.
You need at least three alternatives and a base ranking method; in two-alternative tables, drawing with replacement produces very little variation and yields no information. As the number of alternatives grows, the number of possible draws grows combinatorially, and the distribution becomes more reliable.
When to Use It, When Not To
Bootstrap Resampling is exactly right when you already have a completed ranking analysis and want to know whether the top-ranked alternative reached that position through genuine, solid superiority or through chance in the set of alternatives. It is especially meaningful when the list of alternatives comes from a pool that could change over time, such as a call for tenders, a candidate list, or an annual set of indicators.
There are also situations where it should not be used. If the number of alternatives is very small (two or three), the number of distinct tables that drawing with replacement can produce is already limited, and the result offers little information. If the question of interest is "what happens if the weights change," Weight Sensitivity Analysis should be used instead of this method; Bootstrap Resampling varies the set of alternatives, not the weights. If no base ranking analysis has yet been carried out, this method has nothing to test.
How much the ranking depends on the set of alternatives is the question → Bootstrap Resampling
How much the ranking depends on the weights is the question → Weight Sensitivity Analysis
Two different ranks should be summarised into a single agreement figure → Kendall's τ
The consensus among several rankers (experts, methods) should be measured → Kendall's W
Very few alternatives (two or three) → Bootstrap Resampling's informational value is limited
Strengths
The method's greatest strength is that it makes no assumption about the underlying distribution; it does not assume the data is normally distributed or fits any other pattern, it simply uses the table you hold. It can be attached to any ranking method; it does not alter the base method, it merely reruns it repeatedly. The result can be conveyed to a decision-maker concretely and intuitively, as "this alternative stayed near the top in this proportion of draws."
Weaknesses
Its limitations follow from this same design. First, as the number of alternatives falls, the number of distinct possible samples falls too, and with only three alternatives just twenty-seven distinct draws are possible; in that case the distribution can remain coarse. Second, if the same alternative is drawn more than once in a given draw, or if a criterion column becomes constant in this small sample (all rows copies of the same alternative), the base method's normalisation step can become undefined; these edge cases need separate handling. Third, the method adds computational load; if the base method is slow (a large ELECTRE or PROMETHEE calculation), thousands of repetitions can take time (Triantaphyllou and Sánchez, 1997, raise a similar computational-load warning for sensitivity analysis). Fourth, the method does not decide which alternative is the "true" winner; it only shows how robust the current winner is to chance variation in the set of alternatives.
Common Mistakes
The most common mistake is reading the bootstrap rank interval as a probability; the sentence "A2 is first with ninety-five per cent probability" is wrong, the correct statement is "in ninety-five per cent of the samples drawn with replacement, A2's rank fell within this interval."
A second mistake is applying bootstrap to a table with very few alternatives (two) and reporting broad confidence from it; at that scale the sampling space is already narrow. A third mistake is overlooking the edge case where a criterion column becomes constant in a given draw (the same alternative drawn several times) and failing to notice the resulting normalisation error. A fourth mistake is reading a bootstrap result as "the base ranking was wrong"; bootstrap does not invalidate the base ranking, it only shows how robust it is.
The governing principle is this:
Bootstrap Resampling does not validate a ranking; it shows how robust that ranking is against chance variation in the set of alternatives, and the report must state this robustness as a probabilistic interval, not a certainty.
Cases
Each case opens with a decision table, describes in words what the method does to it, and shows how to read the result. The first case is DecisionMind's validation example; the figures were recomputed and verified in Python by trying, one by one, all twenty-seven draws possible in a three-alternative table (exhaustive enumeration, not random sampling). The remaining cases are illustrative constructions.
1. Illustrative example: Three alternatives, weighted sum, with exhaustive enumeration (DecisionMind validation example)
This example is not a case from the literature; it is a small table built to make the method's logic traceable by hand. Three alternatives are assessed on three criteria; the first two criteria are "higher is better," the third is a cost-type criterion, "lower is better." The simplest ranking method is chosen as the base method: weighted sum (each alternative's scores, scaled to between 0 and 1, are multiplied by the weights and summed).
| Alternative | C1 | C2 | C3 (cost) |
|---|---|---|---|
| A1 | 3 | 5 | 4 |
| A2 | 5 | 3 | 2 |
| A3 | 4 | 4 | 3 |
| Direction | higher is better | higher is better | lower is better |
| Weight | 0.40 | 0.35 | 0.25 |
The base run gives the three alternatives' weighted sum scores: 0.35 for A1, 0.65 for A2, 0.50 for A3. The rank is A2, A3, A1. Drawing three from three alternatives with replacement yields exactly twenty-seven distinct draws; three of these are the extreme case where the same alternative is drawn three times (in these three draws the columns become constant and the scaling is fixed at the midpoint, 0.5). The weighted sum was recomputed for every draw and each alternative's rank was recorded.
| Alternative | Mean rank | 95% interval | Share of draws finishing in rank 1 |
|---|---|---|---|
| A1 | 2.67 | 2-3 | 0% |
| A2 | 1.33 | 1-2 | 44% (outright), 89% (within top two) |
| A3 | 2.00 | 1-3 | 11% (outright), 33% (within top two) |
The result reads as follows. A1 never reached first place in any of the twenty-seven draws, its best rank was second; this shows that A1's last-place position is solid. A2 stayed within the top two ranks in eighty-nine per cent of draws; this strongly supports its first place in the base run. A3's interval is the widest, ranging from first to third; its second place in the base run is not as solid as A2's first.
The analyst's hesitation is this: the base ranking came out A2 > A3 > A1, and this exact full ranking was repeated identically in only six of the twenty-seven draws (twenty-two per cent). A3's second place is not as reliable as A2's first place; the report must state this difference separately.
In the report: "The base weighted-sum rank is A2 > A3 > A1. In the exhaustive enumeration with replacement (27 draws), A2 stayed within the top two ranks in eighty-nine per cent of draws, and its first place is solid; A3's second place is less solid, because its rank varies across a wide interval reaching down to third."
Source: DecisionMind BOOTSTRAP-RESAMPLING manifest, validation example. The manifest's numerical J.expected_primary field carries an old placeholder value that cannot be run in this repository, so it was not used; the figures in this card were reproduced from scratch by exhaustive enumeration. The method's statistical foundation is Efron (1979).
2. Healthcare: A robustness check on a hospital unit's supplier selection
A hospital procurement unit has ranked six medical-consumables suppliers on four measures (price, delivery time, quality-certificate score, past failure rate) using TOPSIS. Suppose supplier D comes out first in the base ranking. Considering that these six suppliers are in fact a sample drawn from a larger candidate pool, and that the pool could change next year, the unit decides to apply Bootstrap Resampling.
The method drew six of the six suppliers with replacement, one thousand times, and reran TOPSIS on every draw. Suppose supplier D stayed first or second in the large majority of draws, but supplier E's rank was scattered widely, from first to sixth.
The unit's hesitation: supplier D's first place can be reported with confidence, but the ranks of the suppliers in the middle of the list depend heavily on which suppliers happened to be redrawn in the sample. The unit decides to include in the report not just the top rank but how volatile the middle ranks are too.
In the report: "In the base ranking produced by TOPSIS, supplier D is first. Across one thousand draws with replacement, D stayed within the top two ranks in the large majority of draws; the position of the mid-ranked suppliers is more sensitive to chance variation in the set of alternatives."
3. Education: Robustness of a university's scholarship-application assessment
A university's scholarship committee has ranked eight finalists, selected from forty applications, on four measures (academic average, project score, interview score, need score) using a weighted-sum method. Suppose applicant F comes out first in the base ranking. Considering that the finalist list is itself a selection, and that a different panel might have chosen a different eight finalists, the committee requests a robustness check.
The method drew eight of the eight finalists with replacement, five hundred times, and rebuilt the ranking on every draw. Suppose applicant F stayed first or second in most draws, but the ranks between third and sixth shuffled substantially from draw to draw.
The committee's hesitation: if the number of scholarships is limited and only the top two receive one, the result can be applied with confidence. But if the limit is four, the gap between the candidates in third and fourth place overlaps heavily in the bootstrap distribution, and the committee must discuss how to explain this uncertainty to applicants.
In the report: "In the base ranking, applicant F is first, and this position held solid across five hundred draws. The gap between the candidates in third and fourth place, however, is sensitive to chance variation in the set of alternatives; if the scholarship limit falls between these two ranks, the result should be communicated with caution."
4. What Not to Do
In the illustrative example, interpreting A2 finishing outright first in forty-four per cent of draws as "A2 is the best alternative with a forty-four per cent probability" is wrong; the correct statement is that A2 finished outright first in forty-four per cent of the samples drawn with replacement, which is not a probability claim. A second error is keeping the number of draws (B) as low as thirty in a three-alternative table while claiming "an exhaustive count was performed"; exhaustive enumeration with three alternatives requires exactly twenty-seven draws, and fewer than that leaves it incomplete. A third error is ignoring A3's wide rank interval and reporting only its mean rank (2.00), stating flatly "A3 is second"; the mean alone hides the fact that A3's position is far less solid than A2's.
Sources
For the formulas behind each step, the intermediate tables and citation formats, see the DecisionMind method page: decisionmind.app/library/bootstrap-resampling
Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1-26. DOI: 10.1214/aos/1176344552
Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC. DOI: 10.1007/978-1-4899-4541-9
Rook, J., Hoos, H. H., & Trautmann, H. (2024). Multi-objective ranking using bootstrap resampling. Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO '24 Companion). DOI: 10.1145/3638530.3654436
Triantaphyllou, E., & Sánchez, A. (1997). A sensitivity analysis approach for some deterministic multi-criteria decision-making methods. Decision Sciences, 28(1), 151-194. DOI: 10.1111/j.1540-5915.1997.tb01306.x