compml.sklearn_utils#

Utilities that integrate scikit-learn with compml.

compml.sklearn_utils.run_cross_val(estimators, algorithm_names, datasets, dataset_names, scoring, cv, *, n_jobs=None, fit_params=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, **kwargs)[source]#

Run cross-validation per (dataset, algorithm) and build CVScores.

For each dataset and algorithm, sklearn.model_selection.cross_val_score is called with a clone of the estimator. The resulting score vector fills scores[d, a, :] where d is the dataset index and a the algorithm index. The sample axis (third dimension) is ordered by runs, so all folds of run 0 appear first, then all folds of run 1, and so on.

Parameters:

  • estimators (Sequence[Any]) – Sklearn-compatible estimators. Length must match algorithm_names.

  • algorithm_names (tuple[str, ...]) – Name for each algorithm.

  • datasets (Sequence[tuple[Any, Any]]) – One pair per dataset. Length must match dataset_names.

  • dataset_names (tuple[str, ...]) – Name for each dataset.

  • scoring (str) – Sklearn scoring name. Stored as metric on the result.

  • cv (Any) – Cross-validation strategy: can be integer fold count, LeaveOneOut, RepeatedKFold, RepeatedStratifiedKFold, or a splitter with n_splits (e.g. KFold, StratifiedKFold).

  • n_jobs (int | None) – Parallel jobs for cross_val_score.

  • fit_params (dict[str, Any] | None) – Parameters to pass to the fit method of the estimator.

  • verbose (int) – Verbosity for cross_val_score.

  • pre_dispatch (str | int) – Job batching for cross_val_score.

  • error_score (float | str) – Value assigned to failed fits in cross_val_score.

  • **kwargs (Any) – Additional keyword arguments forwarded to cross_val_score.

Returns:

Cross-validated scores with shape (n_datasets, n_algorithms, n_samples).

Return type:

CVScores

Raises:

  • ImportError – If scikit-learn is not installed.

  • ValueError – On invalid inputs or when inferred (n_runs, n_folds) does not match cross_val_score output length.

compml.sklearn_utils.run_hold_out(estimators, algorithm_names, datasets, dataset_names, scoring, *, test_size=0.25, train_size=None, random_state=None, shuffle=True, stratify=False, **kwargs)[source]#

Run hold-out per (dataset, algorithm) and build EvaluationScores.

For each dataset and algorithm, sklearn.model_selection.train_test_split is called with a clone of the estimator. The resulting score vector fills scores[d, a, :] where d is the dataset index and a the algorithm index.

Parameters:

  • estimators (Sequence[Any]) – Sklearn-compatible estimators. Length must match algorithm_names.

  • algorithm_names (tuple[str, ...]) – Name for each algorithm.

  • datasets (Sequence[tuple[Any, Any]]) – One pair per dataset. Length must match dataset_names.

  • dataset_names (tuple[str, ...]) – Name for each dataset.

  • scoring (str) – Sklearn scoring name. Stored as metric on the result.

  • test_size (float | int) – Size of the test set (used in train_test_split).

  • train_size (float | int | None) – Size of the train set (used in train_test_split).

  • random_state (int | RandomState | None) – Random seed (used in train_test_split).

  • shuffle (bool) – Whether to shuffle the data (used in train_test_split).

  • stratify (bool) – Whether to stratify the data (used in train_test_split).

  • **kwargs (Any) – Additional keyword arguments forwarded to train_test_split.

Returns:

Evaluation scores with shape (n_datasets, n_algorithms, 1).

Return type:

EvaluationScores

Raises:

  • ImportError – If scikit-learn is not installed.

  • ValueError – On invalid inputs.