About the benchmark

22 estimators · 24 datasets · 2,048 games. We compare contribution estimates for predictions, training data and feature selection.

Games, Shapley values and estimation error

• Game: for players \(N = \{1, \ldots, d\}\), a function \(f: 2^N \to \mathbb{R}\) assigns a value \(f(S)\) to each coalition \(S \subseteq N\). The applications below define the players and \(f\).
• Shapley value: player \(i\)'s average marginal contribution across all player orderings:

• Estimator: an algorithm queries \(f\) at most \(B\) times and returns an estimated vector \(\hat{\boldsymbol{\phi}}\), to approximate the exact contribution vector \(\boldsymbol{\phi}\).
• Normalized mean squared error (nMSE):

Lower is better: zero is exact; one is the error of estimating every contribution as zero. Zero or negligible reference vectors are excluded. For interaction targets, the vectors contain the requested interaction coefficients; the empty-coalition baseline is excluded. Main effects and pairs can also be viewed separately.

• Scope: 512 recipes and 2,048 prepared instances; four seeds per recipe.
• Coverage: all 20 required estimators have 100% successful supported coverage. ShaplEIG and SPEX retain partial coverage. Unsupported or ineligible cells do not count.
• Selection: 477 core recipes and 35 KNN/TNN recipes. A recipe is excluded if any seed fails the model-quality check. Replacements preserve all 24 datasets without selecting for estimator accuracy; see the selection manifest.

Explaining an individual prediction

Players are input features. Fit a predictor \(h\) once and explain one held-out point \(x\). For the small games, keep \(x\)'s selected features and replace the others with fixed training-background means \(b\):

The output is a fixed class probability or a regression prediction. Exact reference: evaluate all \(2^d\) coalitions.

Tree games extend this to larger player sets. Interventional games average predictions with missing features filled from fixed background rows. Path-dependent games average missing branches using training-path weights. Matching exact tree solvers supply the references. Both tree constructions use random forests only.

Models: linear/logistic regression and random forest for baseline games, with nested feature sets of at most 12 players. RF tree games use 8, 16 and 32 features where available. A dataset's full width below 12 is also included in the baseline grid. Tables show selected player counts; “—” means no selected recipe.

Valuing training data

Players are groups of training rows. For a coalition \(S\), fit \(h_S\) on the union of its groups and evaluate on a fixed held-out set \(T\):

Exact reference: refit and score every coalition of 4, 8 or 12 groups. Both linear/logistic regression and random forest use exactly \(\min(12, \text{dataset width})\) input features. There is no input-width sweep; input width is separate from the number of players.

Neighbor games use individual training rows as players for a fixed test example. KNN utility counts matching labels among the coalition's nearest \(\min(k, |S|)\) rows, divided by fixed \(k\). TNN utility is the matching-label fraction within a fixed radius, or \(1 / \text{number of classes}\) when no rows are present. Selected classification games use 32 or 64 players, with exact native Shapley references. All input selections are fixed before estimator evaluation.

Selecting useful features

Players are input features. For every coalition \(S\), fit a fresh predictor \(h_S\) using only those columns, keeping the training and held-out rows fixed:

Exact reference: fit and score all \(2^d\) coalitions. Unlike an individual-prediction game, this measures how well a model can learn from the chosen features.

Models: linear/logistic regression and random forest, with the same nested feature grids of at most 12 players used by the baseline prediction games.

Comparing and reproducing results

• Budgets: 0.5, 1, 2, 4, 8, 16 and 32 × \(d\) coalition queries; three estimator seeds per game. Actual query use can be smaller.
• Targets: Shapley values and order-two k-SII, SII, STII, FSII and FBII interactions, where supported. KNN/TNN games provide SV only.
• Scores: weighted means and medians. Applications receive equal weight, followed by subtype and recipe; datasets with more recipes can have more weight.
• Completion: 100% successful supported coverage for every estimator except ShaplEIG and SPEX. ProxySPEX is included. Failed and missing runs for the two exempt estimators remain visible and unscored.
• Coverage: exact-reference coverage and successful estimator coverage are separate. Unsupported targets are not failures. Zero-signal and quality exclusions are reported without changing frozen games to improve coverage.
• Ground truth: exhaustive tables for generic games of at most 12 players; matching exact native solvers for random-forest tree and neighbor games. References are subject to floating-point precision; small-game solver checks do not establish independent full-dimensional validation.
• Data notes: Wine classification differs from Wine Quality regression. NHANES I uses its supplied survival label as a regression surrogate.
• Timing: measured estimator time on cached games; estimated oracle costs are shown separately. These are diagnostic timings, not isolated end-to-end timings.
• Reproduction: download the frozen games and results from the dashboard. Historical broader results are archived separately and excluded from this benchmark.

Estimator combinations

For each target and budget, we fit signed ridge regression with no intercept:

• Fit: weighted mean nMSE plus \(\lambda\lVert\boldsymbol{\alpha}\rVert_2^2\). Large weights are penalized; negative weights remain allowed.
• Validation: nested dataset-grouped cross-validation chooses \(\lambda\) from \(10^{-6}, 10^{-5}, \ldots, 1\). Test datasets enter neither fitting nor tuning. Panels match games, budgets and seeds across estimators.
• Display: coefficients use all datasets; R² and nMSE use held-out predictions. PCA shows shared errors. Coefficients describe combinations, not estimator importance.