Categories

## Sklearn, Classification and Regression metrics

in the post will reviewed a number of metrics for evaluating classification and regression models. For that we use the functions we use of the sklearn library. We’ll learn how to generate model data and how to train linear models and evaluate their quality.

## sklearn.metrics

```import warnings
warnings.filterwarnings("ignore")
```
```from sklearn import model_selection, datasets, linear_model, metrics
from matplotlib.colors import ListedColormap
%pylab inline
```

### Datasets generation

Since we are to solve 2 problems: Classification and Regression, we’ll need 2 sets of data.

For that we use make_classification and make_regression functions.

For each problem we generate datasets of 2 features and we visualize them. In the Classification problem both features are informative, while in Regression problem, only 1 of 2 features is an informative feature.

```clf_data, clf_target = datasets.make_classification(n_features = 2, n_informative = 2, n_classes = 2,
n_redundant = 0, n_clusters_per_class = 1,
random_state = 7)
```
```reg_data, reg_target = datasets.make_regression(n_features = 2, n_informative = 1, n_targets = 1,
noise = 5., random_state = 7)
```
Classification dataset display
```colors = ListedColormap(["red", "blue"])
pylab.scatter(clf_data[:,0], clf_data[:,1], c = clf_target, cmap = colors)
pylab.rcParams["figure.figsize"] = [10, 7]
``` Regression dataset display
```pylab.scatter(reg_data[:,1], reg_target, color = "r")
pylab.scatter(reg_data[:,0], reg_target, color = "b")
pylab.rcParams["figure.figsize"] = [10, 7]
``` #### We split data into train and test sets.

```clf_train_data, clf_test_data, clf_train_labels, clf_test_labels = model_selection.train_test_split(clf_data, clf_target,
test_size = 0.3, random_state = 1)
```
```reg_train_data, reg_test_data, reg_train_labels, reg_test_labels = model_selection.train_test_split(reg_data, reg_target,
test_size = 0.3, random_state = 1)
```

### Quality metrics in classification tasks

#### Classification model training

We’ll use SGDClassifier. It is a Linear classifier based on Stochastic gradient decent .

• Probabilistic classifier (Loss funciton: loss = ‘log’)
```classifier = linear_model.SGDClassifier(loss = "log", random_state = 1, max_iter=1000)
classifier.fit(clf_train_data, clf_train_labels)
```
```SGDClassifier(loss="log", random_state=1)
```

Generate classifier predicted labels:

```predictions = classifier.predict(clf_test_data)
```

Generate a prediction in the form of the probability that the object belongs to the zero class or the first class.

```probability_predictions = classifier.predict_proba(clf_test_data)
```
```# original dataset labels
print(clf_test_labels)
```
```[1 0 0 1 0 1 1 0 1 0 0 0 1 1 0 0 1 0 0 1 0 0 0 0 0 0 1 1 1 0]
```
```# predicted labels
print(predictions)
```
```[1 0 0 1 0 1 1 0 1 0 0 1 1 1 0 0 1 0 0 1 0 0 0 0 0 0 0 1 1 0]
```

Probabilities that the object belongs to the zero class or the first class.

The first probability is that object belongs to the the zero class and the second probability is that object belongs to the first class

```print(probability_predictions[:10])
```
```[[0.00000000e+00 1.00000000e+00]
[9.99999997e-01 2.90779994e-09]
[9.99990982e-01 9.01818055e-06]
[0.00000000e+00 1.00000000e+00]
[1.00000000e+00 7.01333183e-14]
[5.16838702e-07 9.99999483e-01]
[6.66133815e-16 1.00000000e+00]
[1.00000000e+00 6.21822808e-13]
[0.00000000e+00 1.00000000e+00]
[9.99999998e-01 2.30155106e-09]]
```

We’ve done the preparational work. Now we come to the calculating metrics.

#### Accuracy

Accuracy is metric that shows closeness of the measurements to a specific value , designating a portion of correctly classified objects.

• pair – correct label
• pair – predicted label
• len(clf_test_labels) – data volume
```# calculating thru Python means
acc1 = sum([1. if pair == pair else 0. for pair in zip(clf_test_labels, predictions)])/len(clf_test_labels)
# inbuilt accuracy score
acc2 = metrics.accuracy_score(clf_test_labels, predictions)
print (acc1, acc2)
```
```0.9333333333333333 0.9333333333333333
```

#### Confusion matrix

Confusion matrix is the NxN matrix where N – number of classes (N=2 in our case). Confusion matrix provides to calculate many statistics metrics.

It shows the main characteristic of a built model:
1. True positive (TP) eqv. with hit
2. True negative (TN) eqv. with correct rejection
3. False positive (FP) eqv. with false alarm, type I error or underestimation
4. False negative (FN) eqv. with miss, type II error or overestimation ```matrix = metrics.confusion_matrix(clf_test_labels, predictions)
print("Confusion matrix\n",  matrix)
```
```Confusion matrix
[[17  1]
[ 1 11]]
```
```# manual calculations of True positives and True negatives
sum([1 if pair == pair else 0 for pair in zip(clf_test_labels, predictions)])
```
```28
```
```matrix.diagonal().sum()
```
```28
```

#### Precision

Precision describes the random errors, a measure of statistical variability.

First we estimate the accuracy of the classification to the zero class. We call the precision_score() function, pass it the correct class labels, the predicted class labels. And since by default our label is 1, we need to explicitly say that in this case we estimate the classification accuracy to the zero class. To do this, we use the pos_label argument and say that it is equal to 0 .

```metrics.precision_score(clf_test_labels, predictions, pos_label = 0)
```
```0.9444444444444444
```

We estimate the accuracy of the objects classification to the first class. Default pos_label = 1.

```metrics.precision_score(clf_test_labels, predictions)
```
```0.9230769230769231
```

#### Recall

See the Presision and Recall at a vivid picture: ```metrics.recall_score(clf_test_labels, predictions, pos_label = 0)
```
```0.9444444444444444
```
```metrics.recall_score(clf_test_labels, predictions)
```
```0.9166666666666666
```

#### F-score

```metrics.f1_score(clf_test_labels, predictions, pos_label = 0)
```
```0.9444444444444444
```
```metrics.f1_score(clf_test_labels, predictions)
```
```0.9166666666666666
```

#### Classification report

We use the classification_report function to get the summary table for each class.

```print(metrics.classification_report(clf_test_labels, predictions))
```
```              precision    recall  f1-score   support
0       0.94      0.94      0.94        18
1       0.92      0.92      0.92        12
accuracy                           0.93        30
macro avg       0.93      0.93      0.93        30
weighted avg       0.93      0.93      0.93        30
```

#### ROC curve – Receiver Operating Characteristic curve

An ROC curve plots TPR vs FPR at different classification thresholds. We use probability_predictions in our case.

• probability_predictions[:,1] is the probability that object is of the first class.
```fpr, tpr, _ = metrics.roc_curve(clf_test_labels, probability_predictions[:,1])
# _ contains thresholds, we not using them
```
```pylab.plot(fpr, tpr, label = "Linear model classifier")
pylab.plot([0, 1], [0, 1], "--", color = "grey", label = "Random classifier")
pylab.xlim([-0.05, 1.05])
pylab.ylim([-0.05, 1.05])
pylab.xlabel("False Positive Rate")
pylab.ylabel("True Positive Rate")
pylab.title("ROC curve")
pylab.legend(loc = "lower right")
pylab.rcParams["figure.figsize"] = [10, 7]
``` #### ROC AUC

ROC AUC shows a square area under the ROC function.

```metrics.roc_auc_score(clf_test_labels, predictions)
```
```0.9305555555555554
```
```metrics.roc_auc_score(clf_test_labels, probability_predictions[:,1])
```
```0.9907407407407407
```

#### PR AUC – Precision AUC

```metrics.average_precision_score(clf_test_labels, predictions)
```
```0.873611111111111
```

#### log_loss – logistical losses of Logistic regression

```metrics.log_loss(clf_test_labels, probability_predictions[:,1])
```
```0.21767621111290084
```

### Metrics in the Regression problem

#### Training the regression model

```regressor = linear_model.SGDRegressor(random_state = 1, max_iter = 20)
regressor.fit(reg_train_data, reg_train_labels)
```
```SGDRegressor(max_iter=20, random_state=1)
```
```reg_predictions = regressor.predict(reg_test_data)
```
```print(reg_test_labels)
```
```[   2.67799047    7.06525927  -56.43389936   10.08001896  -22.46817716
-19.27471232   59.44372825  -21.60494574   32.54682713  -41.89798772
-18.16390935   32.75688783   31.04095773    2.39589626   -5.04783924
-70.20925097   86.69034305   18.50402992   32.31573461 -101.81138022
15.14628858   29.49813932   97.282674     25.88034991  -41.63332253
-92.11198201   86.7177122     2.13250832  -20.24967575  -27.32511755]
```
```print(reg_predictions)
```
```[ -1.46503565   5.75776789 -50.13234306   5.05646094 -24.09370893
-8.34831546  61.77254998 -21.98350565  30.65112022 -39.25972497
-17.19337022  30.94178225  26.98820076  -6.08321732  -3.46551
-78.9843398   84.80190097  14.80638314  22.91302375 -89.63572717
14.5954632   31.64431951  95.81031534  21.5037679  -43.1101736
-95.06972123  86.70086546   0.47837761 -16.44594704 -22.72581879]
```

#### MAE – Mean Absolute Error

```metrics.mean_absolute_error(reg_test_labels, reg_predictions)
```
```3.748761311885298
```

#### MSE – Mean Squared Error

```metrics.mean_squared_error(reg_test_labels, reg_predictions)
```
```24.114925597460914
```

#### Mathematical Square Root of MSE

```sqrt(metrics.mean_squared_error(reg_test_labels, reg_predictions))
```
```4.91069502183356
```

#### Coefficient of determination – R2 score

The close R2 score to 1 the better/preciser model we have.

```metrics.r2_score(reg_test_labels, reg_predictions)
```
```0.989317615054695
```

This site uses Akismet to reduce spam. Learn how your comment data is processed.