compml.evaluation_scores#

Provides classes to manage evaluation scores of ML algorithms across one or more datasets.

class compml.evaluation_scores.CVScores(scores, metric, dataset_names, algorithm_names, n_runs, n_folds)[source]#

Bases: EvaluationScores

Evaluation scores from cross-validation (runs * folds = samples).

Extends EvaluationScores with n_runs and n_folds so that n_samples == n_runs * n_folds.

Parameters:

  • scores (ndarray)

  • metric (str)

  • dataset_names (tuple[str, ...])

  • algorithm_names (tuple[str, ...])

  • n_runs (int)

  • n_folds (int)

class compml.evaluation_scores.EvaluationScores(scores, metric, dataset_names, algorithm_names)[source]#

Bases: object

Container for ML algorithms evaluation scores with shape (datasets, algorithms, samples).

Holds a 3-D array of scores for multiple datasets, algorithms, and samples, along with metadata (metric name, dataset/algorithm names). Supports loading from and saving to compressed NumPy .npz files.

Parameters:

  • scores (ndarray)

  • metric (str)

  • dataset_names (tuple[str, ...])

  • algorithm_names (tuple[str, ...])

classmethod load(file_path)[source]#

Load evaluation scores from a compressed .npz file.

Parameters:

file_path (str | Path) – Path to an existing .npz file containing keys: scores, metric, dataset_names, algorithm_names.

Returns:

Instance populated from the file.

Return type:

EvaluationScores

Raises:

ValueError – If path does not exist, is not a file, or does not have .npz extension.

compute_means()[source]#

Compute mean score per (dataset, algorithm) over samples.

Returns:

Array of shape (n_datasets, n_algorithms) with sample means.

Return type:

ndarray

compute_variances(ddof=0)[source]#

Compute variance of scores per (dataset, algorithm) over samples.

Parameters:

ddof (int) – Delta degrees of freedom for variance (default 0, population variance, use 1 for sample variance)

Returns:

Array of shape (n_datasets, n_algorithms) with sample variances.

Return type:

ndarray

store(file_path)[source]#

Save evaluation scores to a compressed .npz file.

Creates parent directories if they do not exist.

Parameters:

file_path (str | Path) – Destination path. It must have .npz extension.

Raises:

ValueError – If file_path does not have .npz extension.

ALGORITHM_AXIS = 1#
DATASET_AXIS = 0#
SAMPLE_AXIS = 2#