Logistic Regression as a method of classification problem
Some examples of Classification Problem :
- Spam Emails versus not Spam Emails
- Loan Default (Yes/No)
- Disease Diagnose
All the above examples are of Binary Classification.
Logistic regression will allow us to solve Classification Problem where we will be working with Discrete Categories.
Logistic Regression for Binary Classification
The convention for binary classification is to have two classes 0 and 1.
- Logistic Regression outputs probabilities
- If the probability 'P' is greater than 0.5 : The data is labelled to 1.
- If the probability 'P' is less than 0.5 : The data is labelled to 0.
By default , Logistic Regression probability threshold is 0.5 .
class sklearn.linear_model.LogisticRegression(penalty=’l2’, dual=False, tol=0.0001, C=1.0, fit_intercept=True, intercept_scaling=1, class_weight=None, random_state=None, solver=’warn’, max_iter=100, multi_class=’warn’, verbose=0, warm_start=False, n_jobs=None, l1_ratio=None)
Regularization Parameter : C
C controls the inverse of the regularization strength.
- Large C can lead to overfit a model
- Small C can lead to underfit a model
In addition to C, logistic regression has a 'penalty' hyperparameter which specifies whether to use 'L1' or 'L2' regularization.
Using Logistic Regression Model instead of Linear regression model
- We can't use a normal linear regression model on different binary groups. It won't lead to a good fit.
- Instead we can transform our linear regression to logistic regression curve.
The sigmoid function( also called Logistic Function) take in any value and outputs it between 0 and 1.
We can take our linear regression solution and can place it into a sigmoid function.
This results in a probability from 0 to 1 belonging in the 1 class.
We can set the cutoff point at 0.5, anything below it will be labeled as class 0 and above it to class 1.
So, we use the logistic function to output a value ranging from 0 to 1. Based off of this probability we assign a class.
After you trained a logistic regression model on some training data, you will evaluate your model's performance on some test data.
You can use a confusion matrix to evaluate classification models.
For example, Imagine testing for diseases
Ex : Test for presence of diseases
NO = Negative Test = False = 0
Yes = Positive Test = True = 1
- True Positive (TP)
- True Negatives(TN)
- False Positives(FP)
- False Negatives(FN)
- Overall, how often it is correct ?
- Accuracy = (TP+TN)/Total = 150/165=0.91
Missclassification Rate(Error Rate):
- Overall, how often it is wrong ?
- Error = (FP+FN)/Total = 15/165 = 0.09
We define two errors in confusion matrix :
- Type I error (False Positive)
- Type II error(False Negative)
Find the code here