compml.bayesian

compml.bayesian.tests

Bayesian tests.

class compml.bayesian.tests.BayesianCorrelatedTTest(rope=0.01)

Bases: ROPEBayesianTest[CVScores]

Bayesian correlated t-test for comparing two algorithms on one dataset.

This implementation follows the correlated t-test setup for cross-validation where the effective variance accounts for the correlation induced by folds. It returns posterior probabilities that the mean difference lies left of the ROPE, inside the ROPE, or right of the ROPE.

Parameters:

rope (float)

compute_posterior(scores)

Compute the posterior distribution for the test.

Parameters:

scores (CVScores) – Cross-validation scores for one dataset and two algorithms.

Returns:

Posterior distribution.

Return type:

AnalyticalPosterior

class compml.bayesian.tests.BayesianFriedmanTest(prior_strength=1.0, alpha=0.05, imprecise=False)

Bases: object

Bayesian Friedman test for comparing multiple algorithms across multiple datasets.

This implementation follows the Bayesian Friedman test in Benavoli et al. (2015). It uses a Dirichlet prior over the ranks of the algorithms and computes the posterior distribution over the ranks. It can be used to test the null hypothesis of no difference between the algorithms.

Parameters:

• prior_strength (float)

• alpha (float)

• imprecise (bool)

run(scores)

Run the Bayesian Friedman test on the given evaluation scores.

Parameters:

scores (EvaluationScores) – The evaluation scores object containing algorithm performance over datasets.

Returns:

The result of the Bayesian Friedman test.

Return type:

BayesianFriedmanTestResult

class compml.bayesian.tests.BayesianHierarchicalTest(rope=0.01, n_posterior_samples=10000, n_chains=4, upper_sigma=1000.0, lower_alpha=1.0, upper_alpha=2.0, lower_beta=0.01, upper_beta=0.1)

Bases: ROPEBayesianTest[CVScores]

Bayesian hierarchical test for comparing two algorithms on multiple datasets.

This implementation follows the hierarchical Bayesian test of Corani et al. It uses the Stan model in bayesian/stan_models/bayesian_hierarchical.stan to infer the posterior distribution parameters, and then estimates ROPE probabilities from the posterior distribution.

Parameters:

• rope (float)

• n_posterior_samples (int)

• n_chains (int)

• upper_sigma (float)

• lower_alpha (float)

• upper_alpha (float)

• lower_beta (float)

• upper_beta (float)

compute_posterior(scores)

Compute the posterior distribution for the test.

Parameters:

scores (CVScores) – Cross-validation evaluation scores for one or more datasets and two algorithms.

Returns:

Monte Carlo posterior distribution.

Return type:

MCPosterior

class compml.bayesian.tests.BayesianSignTest(rope=0.01, pseudo_observation=0.0, prior_strength=1.0, n_posterior_samples=10000, random_state=None)

Bases: ROPEBayesianTest[EvaluationScores]

Bayesian sign test for comparing two algorithms across multiple datasets.

Uses a Dirichlet posterior over the counts of (left, rope, right) or (left, right) when the ROPE is a point. Posterior probabilities are estimated by drawing samples from the Dirichlet and averaging.

Parameters:

• rope (float)

• pseudo_observation (float)

• prior_strength (float)

• n_posterior_samples (int)

• random_state (int | None)

compute_posterior(scores)

Compute the posterior distribution for the test.

Parameters:

scores (EvaluationScores) – Evaluation scores for multiple datasets and two algorithms.

Returns:

Posterior distribution.

Return type:

MCPosterior

class compml.bayesian.tests.BayesianSignedRankTest(rope=0.01, pseudo_observation=0.0, prior_strength=1.0, n_posterior_samples=10000, random_state=None)

Bases: ROPEBayesianTest[EvaluationScores]

Bayesian signed rank test for comparing two algorithms across multiple datasets.

Uses a Dirichlet posterior over the counts of (left, rope, right) or (left, right) when the ROPE is a point. Posterior probabilities are estimated by drawing samples from the Dirichlet and averaging.

Parameters:

• rope (float)

• pseudo_observation (float)

• prior_strength (float)

• n_posterior_samples (int)

• random_state (int | None)

compute_posterior(scores)

Compute the posterior distribution for the test.

Parameters:

scores (EvaluationScores) – Evaluation scores for multiple datasets and two algorithms.

Returns:

Posterior distribution.

Return type:

MCPosterior

compml.bayesian.post_hoc

Bayesian post-hoc methods.

class compml.bayesian.post_hoc.BayesianPostHoc(prior_strength=1.0, rope=0.0, alpha=0.05, n_posterior_samples=15000, random_state=None)

Bases: object

Bayesian post-hoc method for comparing multiple algorithms across multiple datasets.

Follows the procedure in Benavoli et al. (2015): pairwise Bayesian sign tests with direction selection, then Gelman-Tuerlinckx multiple comparisons via joint posterior Monte Carlo. Use after a global test (e.g. Bayesian Friedman) indicates that not all algorithms perform equally.

Parameters:

• prior_strength (float)

• rope (float)

• alpha (float)

• n_posterior_samples (int)

• random_state (int | None)

run(scores)

Run the Bayesian post-hoc method on the given evaluation scores.

Parameters:

scores (EvaluationScores) – The evaluation scores object containing algorithm performance over datasets.

Returns:

Result with accepted statements and joint posterior probability.

Return type:

BayesianPostHocResult

compml.bayesian.posteriors

Posterior distributions for Bayesian tests.

class compml.bayesian.posteriors.AnalyticalPosterior(rope, dist, params)

Bases: Posterior

Analytical posterior distribution.

Parameters:

• rope (float)

• dist (rv_continuous)

• params (dict[str, Any])

plot(color_pdf='#7FB3D5', color_left='#82C9A1', color_rope='#F7C873', color_right='#F2A7B8', color_rope_line='#F4B183')

Plot the posterior density distribution along with the ROPE.

Parameters:

• color_pdf (str) – Color of the PDF. Defaults to "#7FB3D5" (pastel blue).

• color_left (str) – Color of the left region. Defaults to "#82C9A1" (pastel green).

• color_rope (str) – Color of the ROPE region. Defaults to "#F7C873" (pastel amber).

• color_right (str) – Color of the right region. Defaults to "#F2A7B8" (pastel pink/red).

• color_rope_line (str) – Color of the ROPE line. Defaults to "#F4B183" (darker amber).

Returns:

Figure object.

Return type:

matplotlib.figure.Figure

class compml.bayesian.posteriors.MCPosterior(rope, samples)

Bases: Posterior

Monte Carlo posterior distribution.

Parameters:

• rope (float)

• samples (ndarray)

plot()

Plot the posterior distribution.

If a ROPE is used and there are samples with significant probability mass in the ROPE region, the distribution is shown in a simplex. Otherwise, it is shown as a histogram.

Returns:

Figure object.

Return type:

matplotlib.figure.Figure

plot_histogram(color_hist='#A7D8F0', color_hist_line='#F4B183')

Plot the posterior distribution as a histogram.

Parameters:

• color_hist (str) – Color of the histogram. Defaults to "#A7D8F0" (pastel blue).

• color_hist_line (str) – Color of the histogram line. Defaults to "#F4B183" (pastel amber).

Returns:

Figure object.

Return type:

matplotlib.figure.Figure

plot_simplex(color_simplex_edge='orange', color_simplex_grid='#FFCC80', color_simplex_hexbin='turbo', hexbin_gridsize=72, hexbin_bins=None, show_hexbin_colorbar=True)

Plot the posterior distribution in a simplex.

Parameters:

• color_simplex_edge (str) – Color of the simplex edge. Defaults to "orange" (orange).

• color_simplex_grid (str) – Color of the simplex grid. Defaults to "#FFCC80" (smooth orange).

• color_simplex_hexbin (str) – Matplotlib colormap name for hexbin density (low to high count). Defaults to "turbo" (blue to red through yellow/green).

• hexbin_gridsize (int) – Number of hexagons along the x-axis direction (larger means finer bins). Defaults to 72.

• hexbin_bins (str | int | None) – Passed to matplotlib.axes.Axes.hexbin as bins. Use None for linear count coloring. Use "log" for log-scaled count coloring when density is concentrated in a small region. Defaults to None.

• show_hexbin_colorbar (bool) – If True, draw a colorbar for the hexbin counts. Defaults to True.

Returns:

Figure object.

Return type:

matplotlib.figure.Figure

Raises:

ValueError – If samples do not contain ROPE probabilities (shape must be (n_samples, 3)).

class compml.bayesian.posteriors.Posterior(rope)

Bases: ABC

Abstract base class for posterior distributions.

Parameters:

rope (float)

compute_probs()

Compute the posterior probabilities for the ROPE.

Returns:

Posterior probabilities with the keys "p_left", "p_right" and "p_rope" if a ROPE is being used.

Return type:

dict[str, float]

abstractmethod plot()

Plot the posterior distribution.