Sobol Variance-Based Sensitivity Analysis

A decision result (for example, an alternative's score) is affected by several uncertain factors (criterion weights, expert scores). Sobol variance-based sensitivity analysis shares out the total variability (variance) in the result among these factors: it calculates how much each factor is responsible for on its own (the first-order index) and how much it is responsible for in total together with its interactions (the total effect index). Its difference from Morris screening is that it gives not a rough ranking but exact percentages that sum to one (where there is no interaction). It was introduced by Sobol in 1993 and developed into practically usable estimators by Saltelli and colleagues.

The idea behind Sobol is to think of uncertainty as a pie: the total variability in the result is a fixed quantity that can be shared out among the factors. If a factor's share is large, it is the one whose measurement or determination would change the result most if refined. Unlike Morris, this is a full accounting rather than an elimination: each factor's share on its own and the extra share it forms together with other factors are calculated separately. The philosophical consequence is this: the method does not merely say which factor is "important," it also separates how much of that importance comes from the factor itself and how much from its interaction with other factors.

The method proceeds through three steps, all resting on running the decision model repeatedly with a large number of random inputs.

First, sampling. A probability distribution is set for every factor to be examined (for example, a Dirichlet distribution for criterion weights, which produces random weight sets summing to one). Two large, independent sample sets (A and B) are drawn.

Second, mixed samples. For every factor, a new mixed sample set is built by taking that one factor from set A and the rest from set B. This produces, for every factor, a version "perturbed on its own while the others stay fixed." The decision model is run at every point in A, B and all the mixed sets.

Third, two indices. By comparing the outputs of A, B and the mixed sets, two numbers are calculated for every factor: the first-order index (the share of variance that factor explains on its own, without interacting with the others) and the total effect index (the total share that factor explains on its own together with all its interactions). The difference between the two shows the share coming from that factor's interaction with others.

The formulas behind each step (the Jansen estimators) are given on the DecisionMind method page; this card carries no formulas.

The first-order index shows how much knowing one factor precisely on its own would reduce the uncertainty in the result; close to zero, clarifying that factor alone is of no use. The total effect index shows the total share of variance that would disappear if that factor were never changed, that is, held fixed; close to zero, that factor can safely be fixed. If the gap between the two indices is large, that factor is interacting strongly with other factors and cannot be interpreted on its own. If the first-order indices sum to close to one, the factors are acting independently of one another; if the total falls markedly short of one, the difference comes from interactions. No index says "this factor moves the result in this direction"; it only shows the share in variability. Thus:

"The Sobol analysis showed that the C2 weight determines the result"

should read instead:

"The uncertainty in the C2 weight is responsible for approximately this much of the total variability in the result; the remaining share comes from the other factors and their interactions"

Sobol sensitivity indices work with crisp data and need no decision table of their own; they need a host decision model. Required inputs are a probability distribution for every factor (usually Dirichlet for criterion weights), a sample size (a few hundred or more is recommended), and the assumption that the factors are independent of one another. DecisionMind holds no extension of this method; where many factors are present, unimportant ones are first eliminated with Morris screening and the remainder examined in detail with Sobol.

Where exact contribution percentages for the factors are needed, particularly where interactions also need to be separated out, Sobol is exactly the right tool. It is also used to decide which factor would benefit most from being measured or clarified more precisely.

The cases where it should not be used are as follows. If the number of factors is large (dozens) and the computational budget is limited, Morris screening should be done first to reduce the number of factors before turning to Sobol; Sobol's sample requirement grows rapidly with the number of factors. If the factors are not independent of one another, that is, one is forced to change when another does, standard Sobol estimates can be misleading, and special methods are needed for dependent factors. If only whether the ranking is disrupted, that is, whether the winner changes, is being asked, weight sensitivity analysis is a simpler and sufficient tool rather than a full variance decomposition.

Exact contribution percentages and interaction shares are needed → Sobol sensitivity indices

Many factors are present, a rough elimination will be done first → Morris screening

Only whether the winner is overturned is being asked → Weight Sensitivity Analysis

The similarity of two rankings to each other is being measured → Spearman rank correlation

Sobol's greatest strength is that it shares out the uncertainty in the result with a complete accounting; the first-order and total effect indices together show both the importance of a factor on its own and its interaction share. It requires no strong assumptions about the factors' distributions and can work with any probability distribution. It does not look inside the decision model, it only samples the input-output relationship, so it can be applied to any host method (weighted sum, TOPSIS, VIKOR).

Its limitations arise from computational load and from its assumptions. The sample size needed for reliable estimates grows rapidly with the number of factors; below a few hundred samples, the estimates are unstable (Saltelli, Annoni, Azzini, Campolongo, Ratto and Tarantola, 2010). The factors are assumed independent of one another; for factors such as criterion weights whose total is fixed (summing to one), this independence cannot be fully satisfied and must be handled indirectly, for instance by rescaling within the model. If strong interaction is present, the sum of the first-order indices comes out markedly below one, and this gap can only be interpreted together with the total effect indices. Finally, the method does not apply directly to discrete (ordinal) outputs such as a ranking; it must be run on a score or some other continuous output.

The most common mistake is keeping the sample size very small (below a few hundred) and reporting the indices as though they were definitive; at small sample sizes, the indices vary markedly from run to run. A second mistake is expecting the first-order indices to sum exactly to one and treating anything else as a computational error; a total below one shows the presence of interactions, it is not an error. A third mistake is applying Sobol directly to a ranking (who wins); Sobol needs a continuous output (a score, an index), and for a ranking, an expected rank or a similarly transformed output should be used instead. A fourth mistake is eliminating as "unimportant" a factor whose total effect index is large but whose first-order index is small; this gap shows the factor is important not on its own but through interaction.

The governing principle is this:

Sobol indices show the source of the uncertainty in the result as percentages; a small first-order share does not mean that factor is unimportant, it can mean its effect is intertwined with other factors.