mlxtend version: 0.15.0dev

BootstrapOutOfBag

BootstrapOutOfBag(n_splits=200, random_seed=None)

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/BootstrapOutOfBag/

Methods


get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations in the cross-validator

Parameters

Returns


split(X, y=None, groups=None)

y : array-like or None (default: None) Argument is not used and only included as parameter for compatibility, similar to KFold in scikit-learn.

PredefinedHoldoutSplit

PredefinedHoldoutSplit(valid_indices)

Train/Validation set splitter for sklearn's GridSearchCV etc.

Uses user-specified train/validation set indices to split a dataset into train/validation sets using user-defined or random indices.

Parameters

Methods


get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations in the cross-validator

Parameters

Returns


split(X, y, groups=None)

Generate indices to split data into training and test set.

Parameters

Yields

RandomHoldoutSplit

RandomHoldoutSplit(valid_size=0.5, random_seed=None, stratify=False)

Train/Validation set splitter for sklearn's GridSearchCV etc.

Provides train/validation set indices to split a dataset into train/validation sets using random indices.

Parameters

Methods


get_n_splits(X=None, y=None, groups=None)

Returns the number of splitting iterations in the cross-validator

Parameters

Returns


split(X, y, groups=None)

Generate indices to split data into training and test set.

Parameters

Yields

bias_variance_decomp

bias_variance_decomp(estimator, X_train, y_train, X_test, y_test, loss='0-1_loss', num_rounds=200, random_seed=None)

estimator : object A classifier or regressor object or class implementing a fit predict method similar to the scikit-learn API.

Returns

bootstrap

bootstrap(x, func, num_rounds=1000, ci=0.95, ddof=1, seed=None)

Implements the ordinary nonparametric bootstrap

Parameters

Returns

Examples

>>> from mlxtend.evaluate import bootstrap
>>> rng = np.random.RandomState(123)
>>> x = rng.normal(loc=5., size=100)
>>> original, std_err, ci_bounds = bootstrap(x,
...                                          num_rounds=1000,
...                                          func=np.mean,
...                                          ci=0.95,
...                                          seed=123)
>>> print('Mean: %.2f, SE: +/- %.2f, CI95: [%.2f, %.2f]' % (original,
...                                                         std_err,
...                                                         ci_bounds[0],
...                                                         ci_bounds[1]))
Mean: 5.03, SE: +/- 0.11, CI95: [4.80, 5.26]
>>>

For more usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/bootstrap/

bootstrap_point632_score

bootstrap_point632_score(estimator, X, y, n_splits=200, method='.632', scoring_func=None, random_seed=None, clone_estimator=True)

Implementation of the .632 [1] and .632+ [2] bootstrap for supervised learning

References:

Parameters

estimator is a classifier and mean squared error if the estimator is a regressor.

Returns

Examples

>>> from sklearn import datasets, linear_model
>>> from mlxtend.evaluate import bootstrap_point632_score
>>> iris = datasets.load_iris()
>>> X = iris.data
>>> y = iris.target
>>> lr = linear_model.LogisticRegression()
>>> scores = bootstrap_point632_score(lr, X, y)
>>> acc = np.mean(scores)
>>> print('Accuracy:', acc)
0.953023146884
>>> lower = np.percentile(scores, 2.5)
>>> upper = np.percentile(scores, 97.5)
>>> print('95%% Confidence interval: [%.2f, %.2f]' % (lower, upper))
95% Confidence interval: [0.90, 0.98]

For more usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/bootstrap_point632_score/

cochrans_q

cochrans_q(y_target, y_model_predictions)*

Cochran's Q test to compare 2 or more models.

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/cochrans_q/

combined_ftest_5x2cv

combined_ftest_5x2cv(estimator1, estimator2, X, y, scoring=None, random_seed=None)

Implements the 5x2cv combined F test proposed by Alpaydin 1999, to compare the performance of two models.

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/combined_ftest_5x2cv/

confusion_matrix

confusion_matrix(y_target, y_predicted, binary=False, positive_label=1)

Compute a confusion matrix/contingency table.

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/confusion_matrix/

feature_importance_permutation

feature_importance_permutation(X, y, predict_method, metric, num_rounds=1, seed=None)

Feature importance imputation via permutation importance

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/feature_importance_permutation/

ftest

ftest(y_target, y_model_predictions)*

F-Test test to compare 2 or more models.

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/ftest/

lift_score

lift_score(y_target, y_predicted, binary=True, positive_label=1)

Lift measures the degree to which the predictions of a classification model are better than randomly-generated predictions.

The in terms of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN), the lift score is computed as: [ TP / (TP+FP) ] / [ (TP+FN) / (TP+TN+FP+FN) ]

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/lift_score/

mcnemar

mcnemar(ary, corrected=True, exact=False)

McNemar test for paired nominal data

Parameters

Returns

Examples

For usage examples, please see
[http://rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar/](http://rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar/)

mcnemar_table

mcnemar_table(y_target, y_model1, y_model2)

Compute a 2x2 contigency table for McNemar's test.

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar_table/

mcnemar_tables

mcnemar_tables(y_target, y_model_predictions)*

Compute multiple 2x2 contigency tables for McNemar's test or Cochran's Q test.

Parameters

Returns

Examples

For usage examples, please see
[http://rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar_tables/](http://rasbt.github.io/mlxtend/user_guide/evaluate/mcnemar_tables/)

paired_ttest_5x2cv

paired_ttest_5x2cv(estimator1, estimator2, X, y, scoring=None, random_seed=None)

Implements the 5x2cv paired t test proposed by Dieterrich (1998) to compare the performance of two models.

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/paired_ttest_5x2cv/

paired_ttest_kfold_cv

paired_ttest_kfold_cv(estimator1, estimator2, X, y, cv=10, scoring=None, shuffle=False, random_seed=None)

Implements the k-fold paired t test procedure to compare the performance of two models.

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/paired_ttest_kfold_cv/

paired_ttest_resampled

paired_ttest_resampled(estimator1, estimator2, X, y, num_rounds=30, test_size=0.3, scoring=None, random_seed=None)

Implements the resampled paired t test procedure to compare the performance of two models (also called k-hold-out paired t test).

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/paired_ttest_resampled/

permutation_test

permutation_test(x, y, func='x_mean != y_mean', method='exact', num_rounds=1000, seed=None)

Nonparametric permutation test

Parameters

Returns

p-value under the null hypothesis

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/permutation_test/

proportion_difference

proportion_difference(proportion_1, proportion_2, n_1, n_2=None)

Computes the test statistic and p-value for a difference of proportions test.

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/proportion_difference/

scoring

scoring(y_target, y_predicted, metric='error', positive_label=1, unique_labels='auto')

Compute a scoring metric for supervised learning.

Parameters

Returns

Examples

For usage examples, please see http://rasbt.github.io/mlxtend/user_guide/evaluate/scoring/