compml.frequentist

compml.frequentist.tests

Frequentist tests.

class compml.frequentist.tests.BartlettTest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Bartlett homocedasticity test.

With one dataset, each group’s observations are scores obtained by the algorithms for that dataset. With multiple datasets, each group is the vector of the mean over samples within each dataset for that algorithm.

Bartlett’s test is sensitive to departures from normality. The Levene test is often preferred when the data may not be normally distributed.

Parameters:

alpha (float)

compute_c_value(scores)

Compute critical value for the Bartlett test at 1 - alpha (upper-tail).

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute p-value from the Bartlett statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Bartlett statistic value computed from the scores.

Returns:

Upper-tail p-value under the chi-squared distribution.

Return type:

float

compute_statistic(scores)

Compute Bartlett’s chi-squared statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Bartlett statistic.

Return type:

float

class compml.frequentist.tests.CorrelatedTTest(alpha=0.05)

Bases: BaseFrequentistTest[CVScores]

Correlated t-test on CV differences between two algorithms on one dataset.

Uses the Nadeau-Bengio variance correction for the mean of paired fold differences. The null hypothesis is that the mean difference is zero. Inference is two-sided.

Parameters:

alpha (float)

compute_c_value(scores)

Compute two-sided critical value for the correlated t statistic.

Parameters:

scores (CVScores) – Cross-validation scores container.

Returns:

Critical value at 1 - alpha / 2 (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute two-sided p-value from the correlated t statistic.

Parameters:

• scores (CVScores) – Cross-validation scores container.

• statistic (float) – Correlated t statistic.

Returns:

Two-sided p-value computed from the correlated t statistic.

Return type:

float

compute_statistic(scores)

Compute the correlated t statistic.

Parameters:

scores (CVScores) – Cross-validation scores container.

Returns:

Correlated t statistic (mean difference divided by its corrected standard error).

Return type:

float

class compml.frequentist.tests.DAgostinoPearsonTest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Paired-samples D’Agostino-Pearson normality test.

Parameters:

alpha (float)

compute_c_value(scores)

Compute critical value for the D’Agostino-Pearson statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute p-value from the D’Agostino-Pearson statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – D’Agostino-Pearson statistic value computed from the scores.

Returns:

p-value of the paired-samples D’Agostino-Pearson statistic.

Return type:

float

compute_statistic(scores)

Compute the D’Agostino-Pearson statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

D’Agostino-Pearson statistic value computed from the scores.

Return type:

float

class compml.frequentist.tests.FriedmanAlignedRanksTest(alpha=0.05, minimize=False)

Bases: BaseFrequentistTest[EvaluationScores]

Friedman aligned ranks test for repeated measurements.

The null hypothesis is that the mean scores are equal across all algorithms.

Parameters:

• alpha (float)

• minimize (bool)

compute_c_value(scores)

Compute critical value for the Friedman aligned ranks test statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute p-value from the Friedman aligned ranks test statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Friedman aligned ranks test statistic.

Returns:

p-value computed from the Friedman aligned ranks test statistic.

Return type:

float

compute_statistic(scores)

Compute the Friedman aligned ranks test statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Friedman aligned ranks test statistic.

Return type:

float

class compml.frequentist.tests.FriedmanTest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Friedman test for repeated measurements.

The null hypothesis is that the mean scores are equal across all algorithms.

Parameters:

alpha (float)

compute_c_value(scores)

Compute critical value for the Friedman test statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute p-value from the Friedman test statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Friedman test statistic.

Returns:

p-value computed from the Friedman test statistic.

Return type:

float

compute_statistic(scores)

Compute the Friedman test statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Friedman test statistic.

Return type:

float

class compml.frequentist.tests.KolmogorovSmirnovTest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Paired-samples Kolmogorov-Smirnov test for normality.

Parameters:

alpha (float)

compute_c_value(scores)

Compute critical value for the Kolmogorov-Smirnov statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute p-value from the Kolmogorov-Smirnov statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Kolmogorov-Smirnov statistic value computed from the scores.

Returns:

p-value of the paired-samples Kolmogorov-Smirnov test.

Return type:

float

compute_statistic(scores)

Compute the Kolmogorov-Smirnov statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Kolmogorov-Smirnov statistic value computed from the scores.

Return type:

float

class compml.frequentist.tests.KruskalWallisTest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Kruskal-Wallis test for independent samples.

With one dataset, each group’s observations are scores obtained by the algorithms for that dataset. With multiple datasets, each group is the vector of the mean over samples within each dataset for that algorithm.

Parameters:

alpha (float)

compute_c_value(scores)

Compute critical value for the Kruskal-Wallis statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute p-value from the Kruskal-Wallis statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Kruskal-Wallis statistic.

Returns:

p-value computed from the Kruskal-Wallis statistic.

Return type:

float

compute_statistic(scores)

Compute the Kruskal-Wallis statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Kruskal-Wallis statistic.

Return type:

float

class compml.frequentist.tests.LeveneTest(alpha=0.05, center='mean')

Bases: BaseFrequentistTest[EvaluationScores]

Levene homocedasticity test.

With one dataset, each group’s observations are scores obtained by the algorithms for that dataset. With multiple datasets, each group is the vector of the mean over samples within each dataset for that algorithm.

Parameters:

• alpha (float)

• center (Literal['mean', 'median', 'trimmed'])

compute_c_value(scores)

Critical value of the F distribution at 1 - alpha (upper-taiL).

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute p-value from the Levene F statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Levene F statistic value computed from the scores.

Returns:

Upper-tail p-value under the F distribution.

Return type:

float

compute_statistic(scores)

Compute the Levene F statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Levene F statistic.

Return type:

float

class compml.frequentist.tests.MannWhitneyUTest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Two-sided Mann-Whitney U test for independent samples (two algorithms).

Parameters:

alpha (float)

compute_c_value(scores)

Compute two-sided critical value for the Mann-Whitney U statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha / 2 (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute two-sided p-value using the normal approximation.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Mann-Whitney U statistic.

Returns:

Two-sided p-value using the normal approximation.

Return type:

float

compute_statistic(scores)

Compute the Mann-Whitney U statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Mann-Whitney U statistic.

Return type:

float

class compml.frequentist.tests.PairedTTest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Paired t-test on differences between two algorithms on one dataset.

The null hypothesis is that the mean difference is zero. Inference is two-sided.

Parameters:

alpha (float)

compute_c_value(scores)

Compute two-sided critical value for the paired t statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha / 2 (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute two-sided p-value from the paired t statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Paired t statistic.

Returns:

Two-sided p-value computed from the paired t statistic.

Return type:

float

compute_statistic(scores)

Compute the paired t statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Paired t statistic (mean difference divided by its standard error).

Return type:

float

class compml.frequentist.tests.QuadeTest(alpha=0.05, minimize=False)

Bases: BaseFrequentistTest[EvaluationScores]

Quade test for repeated measurements.

The null hypothesis is that the mean scores are equal across all algorithms.

Parameters:

• alpha (float)

• minimize (bool)

compute_c_value(scores)

Compute critical value for the Quade test statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute p-value from the Quade test statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Quade test statistic.

Returns:

p-value computed from the Quade test statistic.

Return type:

float

compute_statistic(scores)

Compute the Quade test statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Quade test statistic.

Return type:

float

class compml.frequentist.tests.RepeatedMeasuresANOVATest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Repeated-measures ANOVA over multiple algorithms and datasets.

The same datasets (subjects) are evaluated under all algorithms (conditions). The null hypothesis is that the mean scores are equal across all algorithms.

Parameters:

alpha (float)

compute_c_value(scores)

Compute critical value for the F statistic at 1 - alpha (upper-tail).

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value for the F statistic at 1 - alpha (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute p-value from the F statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – F statistic.

Returns:

p-value computed from the F statistic.

Return type:

float

compute_statistic(scores)

Compute the repeated-measures ANOVA F statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

F statistic computed from the repeated-measures ANOVA.

Return type:

float

class compml.frequentist.tests.ShapiroWilkTest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Shapiro-Wilk normality test.

Parameters:

alpha (float)

compute_c_value(scores)

Compute critical value for the Shapiro-Wilk statistic.

Note: The critical value is not computed for the Shapiro-Wilk test.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value is not computed for the Shapiro-Wilk test.

Return type:

float

compute_p_value(scores, statistic)

Compute p-value from the Shapiro-Wilk statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Shapiro-Wilk statistic value computed from the scores.

Returns:

p-value of the Shapiro-Wilk test.

Return type:

float

compute_statistic(scores)

Compute the Shapiro-Wilk statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Shapiro-Wilk statistic value computed from the scores.

Return type:

float

class compml.frequentist.tests.SignTest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Two-sided sign test for paired comparisons between two algorithms.

Parameters:

alpha (float)

compute_c_value(scores)

Compute two-sided critical value for the sign-test statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha / 2 (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute two-sided exact p-value from the sign-test statistic.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Sign-test statistic.

Returns:

Two-sided exact p-value computed from the sign-test statistic.

Return type:

float

compute_statistic(scores)

Compute the sign-test statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Sign-test statistic.

Return type:

float

class compml.frequentist.tests.SignedRankTest(alpha=0.05)

Bases: BaseFrequentistTest[EvaluationScores]

Two-sided Wilcoxon signed-rank test for paired comparisons between two algorithms.

Parameters:

alpha (float)

compute_c_value(scores)

Compute two-sided critical value for the signed-rank statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Critical value at 1 - alpha / 2 (upper-tail).

Return type:

float

compute_p_value(scores, statistic)

Compute two-sided p-value using the normal approximation.

Parameters:

• scores (EvaluationScores) – Evaluation scores container.

• statistic (float) – Signed-rank statistic.

Returns:

Two-sided p-value using the normal approximation.

Return type:

float

compute_statistic(scores)

Compute the signed-rank statistic.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Signed-rank statistic.

Return type:

float

compml.frequentist.post_hoc

Post-hoc procedures for frequentist tests.

class compml.frequentist.post_hoc.BonferroniPostHoc(alpha=0.05, multiple_test='Friedman', minimize=False)

Bases: BaseFrequentistPostHoc

Bonferroni post-hoc method.

Parameters:

• alpha (float)

• multiple_test (Literal['Friedman', 'FriedmanAlignedRanks', 'Quade'])

• minimize (bool)

compute_adjusted_p_values(unadjusted_p_values)

Compute the adjusted p-values using the Bonferroni correction.

Parameters:

unadjusted_p_values (list[float]) – Unadjusted p-values.

Returns:

Adjusted p-values using the Bonferroni correction.

Return type:

list[float]

class compml.frequentist.post_hoc.FinnerPostHoc(alpha=0.05, multiple_test='Friedman', minimize=False)

Bases: BaseFrequentistPostHoc

Finner post-hoc method.

Parameters:

• alpha (float)

• multiple_test (Literal['Friedman', 'FriedmanAlignedRanks', 'Quade'])

• minimize (bool)

compute_adjusted_p_values(unadjusted_p_values)

Compute the adjusted p-values using the Finner correction.

Parameters:

unadjusted_p_values (list[float]) – Unadjusted p-values.

Returns:

Adjusted p-values using the Finner correction.

Return type:

list[float]

class compml.frequentist.post_hoc.HochbergPostHoc(alpha=0.05, multiple_test='Friedman', minimize=False)

Bases: BaseFrequentistPostHoc

Hochberg post-hoc method.

Parameters:

• alpha (float)

• multiple_test (Literal['Friedman', 'FriedmanAlignedRanks', 'Quade'])

• minimize (bool)

compute_adjusted_p_values(unadjusted_p_values)

Compute the adjusted p-values using the Hochberg correction.

Parameters:

unadjusted_p_values (list[float]) – Unadjusted p-values.

Returns:

Adjusted p-values using the Hochberg correction.

Return type:

list[float]

class compml.frequentist.post_hoc.HollandPostHoc(alpha=0.05, multiple_test='Friedman', minimize=False)

Bases: BaseFrequentistPostHoc

Holland post-hoc method.

Parameters:

• alpha (float)

• multiple_test (Literal['Friedman', 'FriedmanAlignedRanks', 'Quade'])

• minimize (bool)

compute_adjusted_p_values(unadjusted_p_values)

Compute the adjusted p-values using the Holland correction.

Parameters:

unadjusted_p_values (list[float]) – Unadjusted p-values.

Returns:

Adjusted p-values using the Holland correction.

Return type:

list[float]

class compml.frequentist.post_hoc.HolmPostHoc(alpha=0.05, multiple_test='Friedman', minimize=False)

Bases: BaseFrequentistPostHoc

Holm post-hoc method.

Parameters:

• alpha (float)

• multiple_test (Literal['Friedman', 'FriedmanAlignedRanks', 'Quade'])

• minimize (bool)

compute_adjusted_p_values(unadjusted_p_values)

Compute the adjusted p-values using the Holm correction.

Parameters:

unadjusted_p_values (list[float]) – Unadjusted p-values.

Returns:

Adjusted p-values using the Holm correction.

Return type:

list[float]

class compml.frequentist.post_hoc.HommelPostHoc(alpha=0.05, multiple_test='Friedman', minimize=False)

Bases: BaseFrequentistPostHoc

Hommel post-hoc method.

Parameters:

• alpha (float)

• multiple_test (Literal['Friedman', 'FriedmanAlignedRanks', 'Quade'])

• minimize (bool)

compute_adjusted_p_values(unadjusted_p_values)

Compute the adjusted p-values using the Hommel correction.

Parameters:

unadjusted_p_values (list[float]) – Unadjusted p-values.

Returns:

Adjusted p-values using the Hommel correction.

Return type:

list[float]

class compml.frequentist.post_hoc.LiPostHoc(alpha=0.05, multiple_test='Friedman', minimize=False)

Bases: BaseFrequentistPostHoc

Li post-hoc method.

Parameters:

• alpha (float)

• multiple_test (Literal['Friedman', 'FriedmanAlignedRanks', 'Quade'])

• minimize (bool)

compute_adjusted_p_values(unadjusted_p_values)

Compute the adjusted p-values using the Li correction.

Parameters:

unadjusted_p_values (list[float]) – Unadjusted p-values.

Returns:

Adjusted p-values using the Li correction.

Return type:

list[float]

class compml.frequentist.post_hoc.NemenyiPostHoc(alpha=0.05, minimize=False)

Bases: object

Nemenyi post-hoc method based on critical distance.

This procedure performs all-pairs comparisons on algorithms average ranks computed across datasets.

Parameters:

• alpha (float)

• minimize (bool)

run(scores)

Run the Nemenyi post-hoc method on the given evaluation scores.

Parameters:

scores (EvaluationScores) – Evaluation scores container.

Returns:

Result object containing the Nemenyi post-hoc result.

Return type:

NemenyiPostHocResult

class compml.frequentist.post_hoc.RomPostHoc(alpha=0.05, multiple_test='Friedman', minimize=False)

Bases: BaseFrequentistPostHoc

Rom post-hoc method.

Parameters:

• alpha (float)

• multiple_test (Literal['Friedman', 'FriedmanAlignedRanks', 'Quade'])

• minimize (bool)

compute_adjusted_p_values(unadjusted_p_values)

Compute the adjusted p-values using the Rom correction.

Parameters:

unadjusted_p_values (list[float]) – Unadjusted p-values.

Returns:

Adjusted p-values using the Rom correction.

Return type:

list[float]

class compml.frequentist.post_hoc.ShafferPostHoc(alpha=0.05, multiple_test='Friedman', minimize=False)

Bases: BaseFrequentistPostHoc

Shaffer post-hoc method.

Parameters:

• alpha (float)

• multiple_test (Literal['Friedman', 'FriedmanAlignedRanks', 'Quade'])

• minimize (bool)

compute_adjusted_p_values(unadjusted_p_values)

Compute the adjusted p-values using the Shaffer correction.

Parameters:

unadjusted_p_values (list[float]) – Unadjusted p-values.

Returns:

Adjusted p-values using the Shaffer correction.

Return type:

list[float]