compml.results#

compml.results.base#

Base class for statistical methods results.

class compml.results.base.BaseResult(metric, procedure_name, dataset_names, algorithm_names, metadata)[source]#

Bases: ABC

Abstract base class for results produced by comparison procedures.

Parameters:

  • metric (str) – Name of the evaluation metric used to produce the underlying scores (e.g. "accuracy").

  • procedure_name (str) – Name of the statistical procedure that generated the result.

  • dataset_names (tuple[str, ...]) – Names of the datasets used for the comparison.

  • algorithm_names (tuple[str, ...]) – Names of the algorithms being compared.

  • metadata (dict[str, Any]) – Additional procedure-specific information.

abstractmethod summary()[source]#

Return the method result as a formatted string.

Returns:

Formatted summary of the method result.

Return type:

str

compml.results.frequentist#

Frequentist results.

class compml.results.frequentist.FrequentistPostHocResult(metric, procedure_name, dataset_names, algorithm_names, metadata, alpha, multiple_test, control_algorithm, adjusted_p_values, reject_null)[source]#

Bases: BaseResult

Result of the frequentist post-hoc method.

Parameters:

  • metric (str) – Name of the evaluation metric.

  • procedure_name (str) – Name of the frequentist post-hoc method that generated the result.

  • dataset_names (tuple[str, ...]) – Names of the datasets used for the comparison.

  • algorithm_names (tuple[str, ...]) – Names of the algorithms being compared.

  • metadata (dict[str, Any]) – Procedure-specific metadata.

  • alpha (float) – Size of the test.

  • multiple_test (str) – Name of the multiple test used for the post-hoc method.

  • control_algorithm (str) – Name of the control algorithm.

  • adjusted_p_values (list[float]) – Adjusted p-values for the pairwise comparisons.

  • reject_null (list[bool]) – Whether the null hypothesis was rejected (True) or not (False).

plot()[source]#

Plot adjusted p-values against the significance level.

Pairwise comparisons are ordered by ascending adjusted p-value. Bars show adjusted p-values. The dashed horizontal line marks the significance level (reject the null hypothesis when the bar is below this line).

Returns:

Figure object.

Return type:

Figure

summary()[source]#

Return the test result as a formatted string.

Returns:

Formatted summary of the test result.

Return type:

str

class compml.results.frequentist.FrequentistTestResult(metric, procedure_name, dataset_names, algorithm_names, metadata, alpha, statistic, p_value, c_value, reject_null)[source]#

Bases: BaseResult

Result of the frequentist tests.

Parameters:

  • metric (str) – Name of the evaluation metric.

  • procedure_name (str) – Name of the frequentist procedure that generated the result.

  • dataset_names (tuple[str, ...]) – Names of the datasets used for the comparison.

  • algorithm_names (tuple[str, ...]) – Names of the algorithms being compared.

  • metadata (dict[str, Any]) – Procedure-specific metadata.

  • alpha (float) – Size of the test.

  • statistic (float) – Statistic of the test.

  • p_value (float) – P-value of the test.

  • c_value (float) – Critical value of the test.

  • reject_null (bool) – Whether the null hypothesis was rejected (True) or not (False).

summary()[source]#

Return the test result as a formatted string.

Returns:

Formatted summary of the test result.

Return type:

str

class compml.results.frequentist.NemenyiPostHocResult(metric, procedure_name, dataset_names, algorithm_names, metadata, alpha, average_ranks, critical_distance, pairwise_significant)[source]#

Bases: BaseResult

Result of the Nemenyi post-hoc method.

The Nemenyi post-hoc method performs all-pairs comparisons on algorithms average ranks. It stores the critical distance and the resulting significant/not-significant decisions for each algorithm pair.

Parameters:

  • metric (str) – Name of the evaluation metric.

  • procedure_name (str) – Name of the Nemenyi post-hoc method that generated the result.

  • dataset_names (tuple[str, ...]) – Names of the datasets used for the comparison.

  • algorithm_names (tuple[str, ...]) – Names of the algorithms being compared.

  • metadata (dict[str, Any]) – Procedure-specific metadata.

  • alpha (float) – Size of the test.

  • average_ranks (tuple[float, ...]) – Average ranks of the algorithms.

  • critical_distance (float) – Critical distance of the Nemenyi post-hoc method.

  • pairwise_significant (tuple[tuple[str, str, bool], ...]) – Pairwise significant decisions for each algorithm pair.

plot()[source]#

Plot a CD diagram for the post-hoc Nemenyi method.

Returns:

Figure object.

Return type:

Figure

summary()[source]#

Return the test result as a formatted string.

Returns:

Formatted summary of the test result.

Return type:

str

compml.results.bayesian#

Bayesian results.

class compml.results.bayesian.BayesianFriedmanTestDecision(*values)[source]#

Bases: StrEnum

Decision of the Bayesian Friedman test.

DO_NOT_REJECT_NULL = 'do_not_reject_null'#
INDETERMINATE = 'indeterminate'#
REJECT_NULL = 'reject_null'#
class compml.results.bayesian.BayesianFriedmanTestResult(metric, procedure_name, dataset_names, algorithm_names, metadata, decision, posterior_mean_ranks, posterior_covariance, imprecise_decisions)[source]#

Bases: BaseResult

Result of the Bayesian Friedman test across multiple algorithms.

This result stores the global decision of the test, posterior mean ranks for each algorithm, and posterior covariance matrix for the mean ranks. It also stores the per-permutation decisions for the imprecise variant.

Parameters:

  • metric (str) – Name of the evaluation metric.

  • procedure_name (str) – Name of the Bayesian procedure that generated the result.

  • dataset_names (tuple[str, ...]) – Names of the datasets used for the comparison.

  • algorithm_names (tuple[str, ...]) – Names of the algorithms being compared.

  • metadata (dict[str, Any]) – Procedure-specific metadata.

  • decision (BayesianFriedmanTestDecision) – Global decision: DO_NOT_REJECT_NULL if the null is not rejected, REJECT_NULL otherwise, INDETERMINATE if the decision depends on the prior.

  • posterior_mean_ranks (ndarray) – Posterior mean ranks for each algorithm, shape (n_algorithms,).

  • posterior_covariance (ndarray | None) – Posterior covariance matrix for mean ranks, shape (n_algorithms, n_algorithms), or None when not computed (imprecise variant).

  • imprecise_decisions (list[BayesianFriedmanTestDecision] | None) – Per-permutation decisions for the imprecise variant, or None for the precise variant.

summary()[source]#

Return the test result as a formatted string.

Returns:

Formatted summary of the test result.

Return type:

str

class compml.results.bayesian.BayesianPostHocResult(metric, procedure_name, dataset_names, algorithm_names, metadata, accepted_statements, joint_posterior_prob)[source]#

Bases: BaseResult

Result of the Bayesian post-hoc multiple comparisons procedure.

Stores the list of accepted statements (which algorithm is better than which, or practical equivalence) and the joint posterior probability of the accepted set.

Parameters:

  • metric (str) – Name of the evaluation metric.

  • procedure_name (str) – Name of the Bayesian procedure that generated the result.

  • dataset_names (tuple[str, ...]) – Names of the datasets used for the comparison.

  • algorithm_names (tuple[str, ...]) – Names of the algorithms being compared.

  • metadata (dict[str, Any]) – Procedure-specific metadata.

  • accepted_statements (list[tuple[int, int, bool, float]]) – Each element is (winner_idx, loser_idx, is_directional, probability). For a directional statement, algorithm winner_idx is better than loser_idx. For Rope (is_directional=False), they are practically equivalent.

  • joint_posterior_probability (float) – Joint posterior probability that all accepted statements hold.

  • joint_posterior_prob (float | None)

summary()[source]#

Return the method result as a formatted string.

Returns:

Formatted summary of the method result.

Return type:

str

class compml.results.bayesian.ROPEBayesianTestResult(metric, procedure_name, dataset_names, algorithm_names, metadata, rope, probs, posterior)[source]#

Bases: BaseResult

Result of a ROPE-based Bayesian comparison between two algorithms.

This result stores the ROPE width, posterior probabilities for the three regions (left/rope/right), the posterior distribution and a derived decision string.

Parameters:

  • metric (str) – Name of the evaluation metric.

  • procedure_name (str) – Name of the Bayesian procedure that generated the result.

  • dataset_names (tuple[str, ...]) – Names of the datasets used for the comparison.

  • algorithm_names (tuple[str, ...]) – Names of the two algorithms being compared (order influence decision).

  • metadata (dict[str, Any]) – Procedure-specific metadata.

  • rope (float) – Width of the ROPE.

  • probs (dict[str, float]) – Posterior probabilities with the keys "p_left", "p_rope", and "p_right".

  • posterior (Posterior) – Posterior distribution.

plot()[source]#

Plot the corresponding posterior visualization.

Returns:

Figure object.

Return type:

matplotlib.figure.Figure

summary()[source]#

Return the test result as a formatted string.

Returns:

Formatted summary of the test result.

Return type:

str