Cross-Validation

Cross-Validation adapts a classical idea from statistics for testing how well a model generalises (Stone, 1974; Geisser, 1975) to the decision table. You already hold a baseline ranking; in DecisionMind this base is classical TOPSIS. The method works on the logic of "hold one alternative back in a fold and rebuild the ranking with the rest": each time, one or a few alternatives are removed from the table, the ranking is recomputed with what remains, and the question asked is how well this reduced ranking preserves the order among the remaining alternatives. The output is a stability score for every alternative.

Important note: this method is currently disabled in the DecisionMind calculator until the base method and the fold convention are settled; the engine refuses to run with a "canonical implementation not yet complete" warning (fail-closed). This card has been written to introduce the method and to prepare it for future use; the case figures below were not obtained from a live DecisionMind run but from independently rerunning the steps defined for the method in the manifest.

The idea is the same as cross-validation in statistics: to find out whether a model (here, a ranking) is a coincidence specific only to the data at hand, or a structure that survives even when the data changes a little, a portion of the data is held back, the model is rebuilt with the rest, and the held-back portion's prediction is compared with reality. Applied to a decision table, this becomes the question "if one alternative were removed from the list, would the order of the rest stay the same".

This view carries a consequence: Cross-Validation does not declare a single ranking right or wrong. It shows how resilient the ranking is to small changes in the data set. If removing one alternative leaves the order of the rest undisturbed, the ranking rests on a broad structure; if it disturbs the order, the ranking is sensitive to that alternative's presence.

The method proceeds through four steps.

First, the baseline ranking and the number of folds. DecisionMind passes the decision table to classical TOPSIS and builds a baseline ranking with the full alternative set. A number of folds k is chosen; when k equals the number of alternatives (k=m), only one alternative is held back in each fold, which is called "leave-one-out". In larger data sets, a coarser split such as k=5 or k=10 can also be used.

Second, forming the folds. The m alternatives are split into k folds; each fold is a group of alternatives of roughly equal size.

Third, testing each fold. For every fold, that fold's alternatives are removed from the table and the ranking is recomputed with those remaining. The relative order of the remaining alternatives in this reduced ranking is compared with their relative order in the full table through Spearman correlation.

Fourth, aggregation. Every alternative is assigned, as its stability score, the correlation value computed when it was the one held back. The average across all alternatives gives the ranking's overall cross-validation stability.

The formulas behind each step, the intermediate tables and the citation formats are given on the DecisionMind method page; this card carries no formulas.

An alternative's stability score tells you how well the order of the rest was preserved when that alternative was absent; it does not tell you how "correct" the alternative's own rank is. A score near 1 means the order of the rest, without that alternative, is identical to that in the full table; near -1 means the order of the rest, without that alternative, has reversed. With a small number of alternatives (three or four), each fold leaves only one or two alternatives behind, which means correlation can only take a few discrete values (1, 0, -1); in this case, the stability score is not a fine measurement but a coarse yes-or-no signal.

Thus instead of writing:

"This ranking passed cross-validation"

the report should read:

"The ranking is preserved against the removal of these alternatives one at a time; removing this alternative changes the order of the rest"

Crisp data. In DecisionMind, this method currently rests on classical TOPSIS as its base and carries no extension. You need a complete decision table and weights; a minimum of two alternatives is required, but at least three are recommended for the fold logic to be meaningful. The number of folds k defaults to the number of alternatives (leave-one-out); k can be lowered for larger data sets. The method does not produce weights, it takes them from outside.

If you hold a baseline ranking and want to see how dependent it is on the presence of a particular alternative, especially an unusual one with an extreme value, Cross-Validation is a suitable diagnostic tool. It is useful in decisions where you want to understand how sensitive the current ranking is before a new alternative is added.

The situations where it should not be used are as follows. If the number of alternatives is very small (two or three), correlation can only take a few discrete values and the result stays coarse. If a strong dependency exists between alternatives (one determining another's price, for instance), removing one alternative indirectly affects the others' values too, and this method does not account for that. In addition, since this method cannot currently be computed in DecisionMind until the base-method contract is settled, it cannot yet be used for a live report.

A baseline ranking is ready, the question is "which alternative does it depend on" → Cross-Validation

The question targets criteria, not alternatives → Criteria Removal

The question targets weights → weight sensitivity analysis

Very few alternatives (two or three) → the result stays coarse; a direct sensitivity discussion may be preferable

Cross-Validation's core strength is that it carries an idea tested for decades in statistics (Stone, 1974; Geisser, 1975) into decision analysis. The method answers directly the question "is this ranking specific only to this alternative set", and because every fold amounts to rerunning the same base method, the computational load is small. The result, collapsing into a single stability figure, is easy to show in a report.

Its limitations become pronounced with small data sets. First, with few alternatives, each fold leaves behind only one or two, so correlation can take only a few discrete values; this is a coarse signal, not a fine measure of robustness. Second, it depends on the base method (TOPSIS here); TOPSIS's own limitations carry across as well. Third, if a dependency exists between alternatives (one alternative's presence affecting the assessment of another), the method does not see this, because the remaining alternatives are re-evaluated unchanged in every fold. Azzini and Munda (2025) stress that in social multi-criteria evaluation, such sensitivity and robustness analyses are not sufficient on their own, and the decision context itself must also be questioned.

The most common mistake is reading the stability score as "how good this alternative is"; the score does not show the alternative's quality, it shows how much its presence affects the order of the others. A second mistake is presenting the coarse ±1 values obtained from very small alternative sets (two or three) as if they were a fine sensitivity measurement. A third mistake is assuming that the same stability profile remains valid when the base method changes (a different ranking method used instead of TOPSIS). A fourth mistake is presenting this method, in its current state, as a step that can be run in a live DecisionMind report; the method is documented for now only for introduction and future use.

The governing principle is this:

Cross-Validation shows how dependent a ranking is on the presence of a particular alternative; this indicator strengthens as the number of alternatives grows and coarsens as it shrinks.