As mentioned in the previous post:
In the next post I plan to look at what happens if we scale the data. And then see if we can improve either the SVM or SGM results with some hyperparameter tuning.
Using cross-validation of course.
But, do note, that given a bit of reading, this may be more involved than I expected. So, hyperparameter tuning may move to another post.
Setup
I am not going to show any of the basic notebook setup (imports, display settings and such). I do, as last time, plan to create a test dataset from the dataset provided by scikit-learn. But this time I am going to use sklearn.model_selection.train_test_split. I will set aside ~20% of the original training data as test data.
# let's split into training and test datasets
X.shape
X_trn, X_tst, y_trn, y_tst = ms.train_test_split(X, y, test_size=.2, random_state=42)
X_tst.shape
X_trn.head()
X_tst.head()
Scaling Methods
I plan to look at the following scaling methods:
- Min-Max Scaler: works well if the standard deviation is small and the data distribution is not Gaussian. Is sensitive to outliers.
- Standard Scaler: if data is not normally distributed, this is not the best scaler to use.
- Robust Scaler: specifically designed to be robust to outliers, scaling on the IQR of the data.
Might also look at:
- Quantile Transformer Scaler: can distort linear correlations between attributes measured on a similar scale, but makes attributes at different scales more comparable.
I am also trying to determine whether or not it is acceptable to scale individual attributes using different methods. But so far have not seen any posts/articles suggesting or describing that approach.
Given the limits of each method, I had originally planned to check the individual attributes for outliers and whether or not it is probably taken from a gaussian distribution. Then select a specific scaler. But, one article suggested you just test them all and pick the one that performs best. So, that is what I think I will do. I may later get back to trying to determine attributes with significant outliers or from non-gaussian distributions.
ScalIng Our Data
I plan to test at least the first 3 methods listed above. And to test the 3 supervised learning models against each set of transformed data.
But first, a note of caution:
A common mistake is to apply it to the entire data before splitting into training and test sets. This will bias the model evaluation because information would have leaked from the test set to the training set.
sklearn.preprocessing.scale
I’ll start by instantiating and/or declaring some variables I will use later on.
# let's instantiate our 3 models
plf = linear_model.Perceptron(tol=1e-3, random_state=0)
svm = SVC(random_state=0)
sgm = linear_model.SGDClassifier(max_iter=75, random_state=0)
mlms = [plf, svm, sgm]
# and our 4 scaler method variables
mnxs = preprocessing.MinMaxScaler()
stds = preprocessing.StandardScaler()
robs = preprocessing.RobustScaler()
qts = preprocessing.QuantileTransformer(n_quantiles=50)
Min-Max Scaler
# let's scale our data with min-max and train and test against the transformed data
# I am creating copies of the data in each case, you may not want to do that
mnx_scores = []
X_trn_mnx = mnxs.fit_transform(X_trn)
display(X_trn_mnx[:5,:5])
X_tst_mnx = mnxs.transform(X_tst)
for mlm in mlms:
m_f = mlm.fit(X_trn_mnx, y_trn);
m_preds = mlm.predict(X_tst_mnx)
mnx_scores.append(accuracy_score(y_tst, m_preds))
Standard Scaler
# now scale with standard scaler and test models
ss_scores = []
X_trn_std = stds.fit_transform(X_trn)
display(X_trn_std[:5,:5])
X_tst_std = stds.transform(X_tst)
for mlm in mlms:
m_f = mlm.fit(X_trn_std, y_trn);
m_preds = mlm.predict(X_tst_std)
ss_scores.append(accuracy_score(y_tst, m_preds))
Robust Scaler
# and finally, robust scaler
rs_scores = []
X_trn_rob = robs.fit_transform(X_trn)
display(X_trn_rob[:5,:5])
X_tst_rob = robs.transform(X_tst)
for mlm in mlms:
m_f = mlm.fit(X_trn_rob, y_trn);
m_preds = mlm.predict(X_tst_rob)
rs_scores.append(accuracy_score(y_tst, m_preds))
Quantile Transformer
Those went fairly quickly, so let’s also use that fourth scaling technique.
# that was pretty quick, let's go with the 4 scaler method
qts_scores = []
X_trn_qts = qts.fit_transform(X_trn)
display(X_trn_qts[:5,:5])
X_tst_qts = qts.transform(X_tst)
for mlm in mlms:
m_f = mlm.fit(X_trn_qts, y_trn);
m_preds = mlm.predict(X_tst_qts)
qts_scores.append(accuracy_score(y_tst, m_preds))
Raw Data
For comparison purposes, let’s also train the algorithms on original, un-scaled training data. And, test against the un-scaled test data.
# for comparison let's get the scors for the raw, un-modified features
raw_scores = []
for mlm in mlms:
m_f = mlm.fit(X_trn, y_trn);
m_preds = mlm.predict(X_tst)
raw_scores.append(accuracy_score(y_tst, m_preds))
Scores Using Single Train/Test Split
Okay, now let’s have a look at how the 3 models faired using the raw data and the variously scaled datasets.
s_scores = pd.DataFrame([raw_scores, mnx_scores, ss_scores, rs_scores, qts_scores],
columns=['Perceptron', 'SVM', 'SGD'],
index=["No Scaling", "Min-Max", "Standard Scaler", "Robust Scaler", "Quantile"])
pd.options.display.float_format = '{:,.4f}'.format
display(s_scores)
Looks like using the Standard Scaler produced the best results for all 3 supervised learning models. And, with the given training and test data SVM and SGM appear to have done equally well under Standard Scaling. With the raw data, Perceptron and SVM appear to have done equally well.
Cross-Validation
I previously said I’d be conducting these tests using k-fold cross-validation. Clearly, I misled everyone; myself included.
So, let’s have a look at using cross-validation using the complete dataset for all the models and scaler methods.
First a few declarations or perhaps re-declarations (the latter for local visibility).
# okay, let's do cross val on complete data set for all scalers and models
mlms = [plf, svm, sgm]
scls = [mnxs, stds, robs, qts]
df_ndx = ["No Scaling", "Min-Max", "Standard Scaler", "Robust Scaler", "Quantile"]
df_cols = ['Perceptron', 'SVM', 'SGD']
xv_scores = {ssm: [] for ssm in df_ndx}
Now a wee function to save some coding.
def get_xval_mean(X_arr, y_vec, cv=5, scoring="accuracy"):
prc_scr = ms.cross_val_score(plf, X_arr, y_vec, cv=cv, scoring=scoring)
prc_scr = np.append(prc_scr, prc_scr.mean())
svm_scr = ms.cross_val_score(svm, X_arr, y, cv=cv, scoring=scoring)
svm_scr = np.append(svm_scr, svm_scr.mean())
sgm_scr = ms.cross_val_score(sgm, X_arr, y_vec, cv=cv, scoring=scoring)
sgm_scr = np.append(sgm_scr, sgm_scores2.mean())
return [prc_scr.mean(), svm_scr.mean(), sgm_scr.mean()]
And, now scale our full dataset. Note, already done with the standard scaler in a previous cell (not included in this post).
# X_trn_std2 = stds.fit_transform(X)
X_trn_mnx2 = mnxs.fit_transform(X)
X_trn_rob2 = robs.fit_transform(X)
X_trn_qts2 = qts.fit_transform(X)
trn_sets = [X, X_trn_mnx2, X_trn_std2, X_trn_rob2, X_trn_qts2]
Let’s run all those cross-validations.
for i in range(len(df_ndx)):
xv_scores[df_ndx[i]] = get_xval_mean(trn_sets[i], y)
print(xv_scores)
s_scores4 = pd.DataFrame(xv_scores.values(),
columns=df_cols,
index=df_ndx)
#pd.options.display.float_format = '{:,.4f}'.format
display(s_scores4)
Not quite like the results with got with the simple train/test split we used at first. But hard to say it is significantly different. At two decimal places, there would not be a lot of difference for any of the results using scaling. That said, scaling did more significantly beat no scaling than any particular scaler beat any of the others.
Though, in the earlier results, there seemed to be a more definitive best choice. I.E. the Standard Scaler. I am sure it has a lot to do with the difference in training and test data. I also expect that perhaps the cross-validation results are more indicative of the expected real results. But…
Sorry, But That’s It
I am working on the homework for Unit 2 in the oft mentioned machine learning course, and it is taking a bit of my time. And, I also have to get the Unit 2 project completed over the next week or so. Consquently that is it for this post.
Though I have to admit I am not as happy with it as I was with the previous one. Such is life. (Though I did, as usual, enjoy working on the code for the notebook/post.)
Feel free to download and play with my version of this post’s related notebook.
Resources
- sklearn.model_selection.train_test_split
- sklearn.model_selection.cross_val_score
- sklearn.preprocessing: Preprocessing and Normalization API sklearn.preprocessing.MinMaxScaler
- sklearn.preprocessing.StandardScaler
- sklearn.preprocessing.scale
- sklearn.preprocessing.RobustScaler
- sklearn.preprocessing.QuantileTransformer
- Compare the effect of different scalers on data with outliers
- Data Transformation: Standardization vs Normalization
- All about Feature Scaling
- Scale, Standardize, or Normalize with Scikit-Learn
- Feature Scaling for Machine Learning: Understanding the Difference Between Normalization vs. Standardization
- A Gentle Introduction to Normality Tests in Python
- Building a Simple Machine Learning Model on Breast Cancer Data
- Breast Cancer Dataset Analysis, Visualization and Machine Learning in Python