scikit-learn classification

OpenIntro - Spam

We will start by looking at a data set on spam emails from the OpenIntro project. A full data dictionary can be found here. To keep things simple this week we will restrict our exploration to including only the following columns: spam, exclaim_mess, format, num_char, line_breaks, and number.

spam - Indicator for whether the email was spam.
exclaim_mess - The number of exclamation points in the email message.
format - Indicates whether the email was written using HTML (e.g. may have included bolding or active links).
num_char - The number of characters in the email, in thousands.
line_breaks - The number of line breaks in the email (does not count text wrapping).
number - Factor variable saying whether there was no number, a small number (under 1 million), or a big number.

email = pd.read_csv('data/email.csv')[ 
  ['spam', 'exclaim_mess', 'format', 'num_char', 'line_breaks', 'number'] 
]
email

      spam  exclaim_mess  format  num_char  line_breaks number
0        0             0       1    11.370          202    big
1        0             1       1    10.504          202  small
2        0             6       1     7.773          192  small
3        0            48       1    13.256          255  small
4        0             1       0     1.231           29   none
...    ...           ...     ...       ...          ...    ...
3916     1             0       0     0.332           12  small
3917     1             0       0     0.323           15  small
3918     0             5       1     8.656          208  small
3919     0             0       0    10.185          132  small
3920     1             1       0     2.225           65  small

[3921 rows x 6 columns]

Given that number is categorical, we will take care of the necessary dummy coding via pd.get_dummies(),

email_dc = pd.get_dummies(email)
email_dc

      spam  exclaim_mess  format  num_char  line_breaks  number_big  number_none  number_small
0        0             0       1    11.370          202           1            0             0
1        0             1       1    10.504          202           0            0             1
2        0             6       1     7.773          192           0            0             1
3        0            48       1    13.256          255           0            0             1
4        0             1       0     1.231           29           0            1             0
...    ...           ...     ...       ...          ...         ...          ...           ...
3916     1             0       0     0.332           12           0            0             1
3917     1             0       0     0.323           15           0            0             1
3918     0             5       1     8.656          208           0            0             1
3919     0             0       0    10.185          132           0            0             1
3920     1             1       0     2.225           65           0            0             1

[3921 rows x 8 columns]

sns.pairplot(email, hue='spam', corner=True, aspect=1.25)

Model fitting

from sklearn.linear_model import LogisticRegression

y = email_dc.spam
X = email_dc.drop('spam', axis=1)

m = LogisticRegression(fit_intercept = False).fit(X, y)

m.feature_names_in_

array(['exclaim_mess', 'format', 'num_char',
       'line_breaks', 'number_big', 'number_none',
       'number_small'], dtype=object)

m.coef_

array([[ 0.0098, -0.619 ,  0.0544, -0.0056, -1.2121,
        -0.6934, -1.9208]])

A quick comparison

R output

glm(spam ~ . - 1, data = d, family=binomial)


Call:  glm(formula = spam ~ . - 1, family = binomial, data = d)

Coefficients:
exclaim_mess        format      num_char  
    0.009587     -0.604782      0.054765  
 line_breaks     numberbig    numbernone  
   -0.005480     -1.264827     -0.706843  
 numbersmall  
   -1.950440  

Degrees of Freedom: 3921 Total (i.e. Null);  3914 Residual
Null Deviance:      5436 
Residual Deviance: 2144     AIC: 2158

sklearn output

m.feature_names_in_

array(['exclaim_mess', 'format', 'num_char',
       'line_breaks', 'number_big', 'number_none',
       'number_small'], dtype=object)

m.coef_

array([[ 0.0098, -0.619 ,  0.0544, -0.0056, -1.2121,
        -0.6934, -1.9208]])

`sklearn.linear_model.LogisticRegression`

From the documentations,

This class implements regularized logistic regression using the ‘liblinear’ library, ‘newton-cg’, ‘sag’, ‘saga’ and ‘lbfgs’ solvers. Note that regularization is applied by default. It can handle both dense and sparse input. Use C-ordered arrays or CSR matrices containing 64-bit floats for optimal performance; any other input format will be converted (and copied).

Penalty parameter

🚩🚩🚩 LogisticRegression() has a parameter called penalty that applies a "l1" (lasso), "l2" (ridge), "elasticnet" or None with "l2" being the default. To make matters worse, the regularization is controlled by the parameter C which defaults to 1 (not 0) - also C is the inverse regularization strength (e.g. different from alpha for ridge and lasso models). 🚩🚩🚩

\[ \min_{w, c} \frac{1 - \rho}{2}w^T w + \rho |w|_1 + C \sum_{i=1}^n \log(\exp(- y_i (X_i^T w + c)) + 1), \]

Another quick comparison

R output

glm(spam ~ . - 1, data = d, family=binomial)


Call:  glm(formula = spam ~ . - 1, family = binomial, data = d)

Coefficients:
exclaim_mess        format      num_char  
    0.009587     -0.604782      0.054765  
 line_breaks     numberbig    numbernone  
   -0.005480     -1.264827     -0.706843  
 numbersmall  
   -1.950440  

Degrees of Freedom: 3921 Total (i.e. Null);  3914 Residual
Null Deviance:      5436 
Residual Deviance: 2144     AIC: 2158

sklearn output (penalty None)

m = LogisticRegression(
  fit_intercept = False, penalty=None
).fit(
  X, y
)
m.feature_names_in_

array(['exclaim_mess', 'format', 'num_char',
       'line_breaks', 'number_big', 'number_none',
       'number_small'], dtype=object)

m.coef_

array([[ 0.0096, -0.6049,  0.0548, -0.0055, -1.2646,
        -0.7068, -1.9505]])

Solver parameter

It is also possible specify the solver to use when fitting a logistic regression model, to complicate matters somewhat the choice of the algorithm depends on the penalty chosen:

newton-cg - ["l2", None]
lbfgs - ["l2", None]
liblinear - ["l1", "l2"]
sag - ["l2", None]
saga - ["elasticnet", "l1", "l2", None]

Also the can be issues with feature scales for some of these solvers:

Note: ‘sag’ and ‘saga’ fast convergence is only guaranteed on features with approximately the same scale. You can preprocess the data with a scaler from sklearn.preprocessing.

Prediction

Classification models have multiple prediction methods depending on what type of output you would like,

m.predict(X)

array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])

m.predict_proba(X)

array([[0.9132, 0.0868],
       [0.956 , 0.044 ],
       [0.9579, 0.0421],
       [0.9408, 0.0592],
       [0.6876, 0.3124],
       [0.6845, 0.3155],
       [0.9342, 0.0658],
       [0.9636, 0.0364],
       [0.8958, 0.1042],
       [0.9418, 0.0582],
       [0.9325, 0.0675],
       [0.896 , 0.104 ],
       [0.9124, 0.0876],
       [0.9727, 0.0273],
       [0.9283, 0.0717],
       [0.9835, 0.0165],
       [0.9633, 0.0367],
       [0.9538, 0.0462],
       [0.8889, 0.1111],
       [0.8042, 0.1958],
       [0.899 , 0.101 ],
       [0.9564, 0.0436],
       [0.9908, 0.0092],
       [0.8802, 0.1198],
       [0.8052, 0.1948],
       [0.8875, 0.1125],
       [0.8973, 0.1027],
       [0.8868, 0.1132],
       [0.6852, 0.3148],
       [0.937 , 0.063 ],
       ...,
       [0.8821, 0.1179],
       [0.9938, 0.0062],
       [0.9351, 0.0649],
       [0.6893, 0.3107],
       [0.8771, 0.1229],
       [0.7932, 0.2068],
       [0.7899, 0.2101],
       [0.6726, 0.3274],
       [0.8934, 0.1066],
       [0.9327, 0.0673],
       [0.6893, 0.3107],
       [0.8845, 0.1155],
       [0.9819, 0.0181],
       [0.8895, 0.1105],
       [0.8836, 0.1164],
       [0.6728, 0.3272],
       [0.7904, 0.2096],
       [0.6799, 0.3201],
       [0.6871, 0.3129],
       [0.7061, 0.2939],
       [0.9331, 0.0669],
       [0.9306, 0.0694],
       [0.8896, 0.1104],
       [0.7888, 0.2112],
       [0.9183, 0.0817],
       [0.8806, 0.1194],
       [0.8824, 0.1176],
       [0.9598, 0.0402],
       [0.8925, 0.1075],
       [0.898 , 0.102 ]])

m.predict_log_proba(X)

array([[-0.0908, -2.4446],
       [-0.045 , -3.1226],
       [-0.043 , -3.1674],
       [-0.061 , -2.8277],
       [-0.3746, -1.1634],
       [-0.3791, -1.1536],
       [-0.0681, -2.7209],
       [-0.0371, -3.3124],
       [-0.11  , -2.2619],
       [-0.06  , -2.8433],
       [-0.0699, -2.6955],
       [-0.1098, -2.2635],
       [-0.0917, -2.4351],
       [-0.0277, -3.6016],
       [-0.0744, -2.6356],
       [-0.0166, -4.1056],
       [-0.0374, -3.304 ],
       [-0.0473, -3.075 ],
       [-0.1178, -2.1973],
       [-0.2179, -1.6306],
       [-0.1064, -2.293 ],
       [-0.0445, -3.1338],
       [-0.0092, -4.6932],
       [-0.1276, -2.1219],
       [-0.2166, -1.636 ],
       [-0.1193, -2.1848],
       [-0.1083, -2.2763],
       [-0.1201, -2.179 ],
       [-0.378 , -1.1558],
       [-0.0651, -2.764 ],
       ...,
       [-0.1254, -2.1383],
       [-0.0062, -5.0906],
       [-0.0671, -2.7347],
       [-0.3721, -1.1689],
       [-0.1311, -2.0963],
       [-0.2317, -1.5758],
       [-0.2359, -1.56  ],
       [-0.3966, -1.1165],
       [-0.1127, -2.2389],
       [-0.0696, -2.699 ],
       [-0.3721, -1.1689],
       [-0.1228, -2.1582],
       [-0.0183, -4.0096],
       [-0.1171, -2.2027],
       [-0.1238, -2.1506],
       [-0.3964, -1.117 ],
       [-0.2352, -1.5625],
       [-0.3858, -1.1392],
       [-0.3752, -1.162 ],
       [-0.3479, -1.2247],
       [-0.0692, -2.7051],
       [-0.072 , -2.6674],
       [-0.117 , -2.2034],
       [-0.2373, -1.5549],
       [-0.0853, -2.5043],
       [-0.1272, -2.1252],
       [-0.1252, -2.1401],
       [-0.041 , -3.2143],
       [-0.1138, -2.2298],
       [-0.1076, -2.2828]])

Scoring

Classification models also include a score() method which returns the model’s accuracy,

m.score(X, y)

0.90640142820709

Other scoring options are available via the metrics submodule

from sklearn.metrics import accuracy_score, roc_auc_score, f1_score, confusion_matrix

accuracy_score(y, m.predict(X))

0.90640142820709

roc_auc_score(y, m.predict_proba(X)[:,1])

0.7606952445645924

f1_score(y, m.predict(X))

0.0

confusion_matrix(y, m.predict(X), labels=m.classes_)

array([[3554,    0],
       [ 367,    0]])

Scoring visualizations - confusion matrix

from sklearn.metrics import ConfusionMatrixDisplay
cm = confusion_matrix(y, m.predict(X), labels=m.classes_)

disp = ConfusionMatrixDisplay(cm).plot()
plt.show()

Scoring visualizations - ROC curve

from sklearn.metrics import auc, roc_curve, RocCurveDisplay

fpr, tpr, thresholds = roc_curve(y, m.predict_proba(X)[:,1])
roc_auc = auc(fpr, tpr)
disp = RocCurveDisplay(fpr=fpr, tpr=tpr, roc_auc=roc_auc,
                       estimator_name='Logistic Regression').plot()
plt.show()

Scoring visualizations - Precision Recall

from sklearn.metrics import precision_recall_curve, PrecisionRecallDisplay

precision, recall, _ = precision_recall_curve(y, m.predict_proba(X)[:,1])
disp = PrecisionRecallDisplay(precision=precision, recall=recall).plot()
plt.show()

Another visualization

def confusion_plot(truth, probs, threshold=0.5):
    
    d = pd.DataFrame(
        data = {'spam': y, 'truth': truth, 'probs': probs}
    )
    
    # Create a column called outcome that contains the labeling outcome for the given threshold
    d['outcome'] = 'other'
    d.loc[(d.spam == 1) & (d.probs >= threshold), 'outcome'] = 'true positive'
    d.loc[(d.spam == 0) & (d.probs >= threshold), 'outcome'] = 'false positive'
    d.loc[(d.spam == 1) & (d.probs <  threshold), 'outcome'] = 'false negative'
    d.loc[(d.spam == 0) & (d.probs <  threshold), 'outcome'] = 'true negative'
    
    # Create plot and color according to outcome
    plt.figure(figsize=(12,4))
    plt.xlim((-0.05,1.05))
    sns.stripplot(y='truth', x='probs', hue='outcome', data=d, size=3, alpha=0.5)
    plt.axvline(x=threshold, linestyle='dashed', color='black', alpha=0.5)
    plt.title("threshold = %.2f" % threshold)
    plt.show()

truth = pd.Categorical.from_codes(y, categories = ('not spam','spam'))
probs = m.predict_proba(X)[:,1]
confusion_plot(truth, probs, 0.5)

confusion_plot(truth, probs, 0.25)

scikit-learn
classification

OpenIntro - Spam

Model fitting

A quick comparison

`sklearn.linear_model.LogisticRegression`

Penalty parameter

Another quick comparison

Solver parameter

Prediction

Scoring

Scoring visualizations - confusion matrix

Scoring visualizations - ROC curve

Scoring visualizations - Precision Recall

Another visualization

Example 1 - DecisionTreeClassifier

Example 2 - SVC

MNIST

MNIST handwritten digits

digit description

Example digits

Doing things properly - train/test split

Multiclass logistic regression

Model coefficients

Confusion Matrix

Report

ROC & AUC?

Prediction

Examining the coefs

Example 3 - DecisionTreeClassifier

Example 4 - GridSearchCV w/ Multiple models
(Trees vs Forests)

scikit-learnclassification

scikit-learn
classification