Awesome Introduction to Logistic Regression with PyTorch
Hello Folks!
In this Post, we are tweaking to one of the most popular supervised learning Algorithm known as Logistic Regression in PyTorch.One of the concepts of Logistic Regression is helpful in building Deep Neural Networks.
So Let’s Begin this Journey!
Logistic Regression -Simple Intuition
In Simple Words, Logistic Regression is more or less like a Linear Classifier which calculates the logits i.e. scores in order to predict the target classes. So One of the Most Simple Definitions given by Wikipedia-
“Logistic regression measures the relationship between the categorical dependent variable and one or more independent variables by estimating probabilities using a logistic function” (Wikipedia)
So Let’s Break Logistic Regression
Dependent and Independent Variables
The dependent variable is the target class variable we are going to predict. However, the independent variables are the features or attributes we are going to use to predict the target class.
Suppose A iPhone Retailer wants to know about a person entering his Retail Shop will buy iPhone or Not?. To predict whether the customer will buy the iPhone or not. The shop owner will observe the customer features like.
Gender:
- Probabilities wise male will high chances of purchasing an iPhone than females.
Age:
- Kids won’t purchase an iPhone.
The Shop Owner will use the above features as well as the other features to calculate likelihood occurrence of event i.e. whether a particular person will buy or not. Diving Deep into Math, Logistic Regression model will pass the likelihood occurrences through a logistic function to predict corresponding classes.
Binary Classification Intution with Logistic Regression Model
Talking about the Logistic Regression Model more in detail, We preassigned activity scores and the weights for the logistic regression model. So Where the hell these Activity Scores and weights come from? Activity Scores are numerically equivalent to an activity of a particular person whether he is going to purchase an iPhone or not. Weights are weightage related to a particular target value. Let us say the weightage of a person purchasing iPhone is 0.9 then the model is 90% confident that this person will buy an iPhone.
The step from linear regression to logistic regression is kind of straightforward. In the linear regression model, we have modeled the relationship between outcome and features with a linear equation. For classification, we prefer probabilities between 0 and 1, so we wrap the right side of the equation into the logistic function. This forces the output to assume only values between 0 and 1.
Moreover, the calculated logits function will pass through the softmax function. The softmax function will return the probabilities for each target class. The high probability target class will be the predicted target class.
What is Softmax function?
One of the most popular function used to calculate the probabilities of events. Main moto here to use Softmax function is to produce output within the range of (0,1). It takes each value in logits and calculates the probabilities. The Softmax is also known as the normalized exponential function. Some Points needs to be considered Regarding Softmax function inputs and outputs.
- If we multiply the Softmax function inputs, the inputs values will become large. So the logistic regression will be more confident (High Probability value) about the predicted target class.
- If we divide the Softmax function inputs, the inputs values will become small. So the Logistic regression model will be not confident (Less Probability value) of the predicted target class.
So much of theoritical Stuffs Now Let’s Jump to fun part!
Training Logistic Regression Model with PyTorch
Stepwise Break-Down
- Load Dataset
- Make Dataset Iterable
- Create a Model Class
- Instantiate Model Class
- Instantiate Loss Class
- Instantiate Optimizer class
- Train Model
Let’s Go!
Load Dataset
In this part we are basically looking into the Implementation of a Logistic Regression model with a simple Dataset know as MINST Dataset. The MNIST dataset is one of the most common datasets used for image classification and accessible from many different sources. In fact, even PyTorh allow us to import and download the MNIST dataset directly from their API. will start with the following two lines to import torchvision.datasets as dsets and MNIST dataset under the PyTorch API.
import torchvision.datasets as dsets
from torch.nn import functional as F
train_dataset = dsets.MNIST(root=’./data’,
train=True,
transform=transforms.ToTensor(),
download=True)The MNIST database contains 60,000 training images and 10,000 testing images taken from American Census Bureau employees and American high school students. Here I have clearly stated train = True for loading only train dataset having 60,000 handwritten Images.
Now if we want to inspect a single image then So this is how a single image is represented in numbers. It’s actually a 28-pixel x 28-pixel image which is why you would end up with this 28x28 matrix of numbers.
train_dataset[0]tensor([[[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
0.0000, 0.0000, 0.0000, 0.0000],.....)
If you load the training dataset, each data point will represent a tuple containing an image matrix and label. So let’s take an example if the second number is 5 then 28 x 28 matrix of numbers representing a digit 5. Let’s Visualize this hell pythonically.
import pandas as pd
import matplotlib.pyplot as pltshow_img = train_dataset[0][0].numpy().reshape(28, 28)
plt.imshow(show_img, cmap='gray')
Hence as expected we got an image of digit representing 5. If we want to print the label of train data then validate it by printing its tensor object say tensor(5).
train_dataset[0][1]tensor(5)
Make Dataset Iterable
In order to input our input data into the model, we have to make our train data iterable. Simply iterable means we have to iterate to our dataset in order to get predictions.
epochs = iterations ÷ totaldata / minibatch
No need to worry! PyTorch provides us torch.utils API’s to make our Dataset iterable by passing batch size. This is just a simplified example of what we’re doing above where we’re creating an iterable object to loop through so we can access all the images img_1 and img_2.This can be achieved by-
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
batch_size=batch_size,
shuffle=True)Create and Instantiate Model Class
In order to initiate our training, we have to create a Model class by using PyTorch nn.Module.The Initialization of class will be based on input and output dimensions. As we’re trying to classify digits 0–9 a total of 10 classes, our output dimension is 10.And we’re feeding the model with 28x28 images, hence our input dimension is 28x28. We can simply apply functional.softmax to our current linear output in order to convert our Linear Output into Softmax Probabilities.
# Same as linear regression!
class LogisticRegressionModel(nn.Module):
def __init__(self, input_dim, output_dim):
super(LogisticRegressionModel, self).__init__()
self.linear = nn.Linear(input_dim, output_dim)
def forward(self, x):
out = self.linear(x)
return out
input_dim = 28*28
output_dim = 10model = LogisticRegressionModel(input_dim, output_dim)
Instantiate Loss and Optimizer Class
Unlike linear regression, we do not use MSE here, we need Cross Entry Loss to calculate our loss before we backpropagate and update our parameters. So What The Hell is Cross Entropy Loss?. Cross-Entropy Loss is basically used to fix the problem of slow learning rate. It turns out that we can solve this problem of slow learning rate by replacing the quadratic cost with a different cost function, known as the cross-entropy.
In the above expression n is the total number of items of training data, the sum is over all training inputs x and y is the output. We can consider Cross-Entropy as loss or cost function because of its Non-Negative Nature and also the cross-entropy is positive and tends toward zero as the neuron gets better at computing the desired output for all training inputs.
So In our use case, we will Compute softmax (logistic/softmax function) as well as the cross-entropy function at each time in every iteration. Also at every Iteration, we update our Model’s Parameters.
criterion = nn.CrossEntropyLoss()
learning_rate = 0.001optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
Here we are also optimizing Model’s Parameters as well as calculating the parameters’ gradients and update them subsequently.
Train Model
So Finally We are in the End Game!. The below code speaks about the operation it is performing. Let me Break down it for you!. First, we are iterating to the number of epochs as well as the iterator object train_loader. Taking each image and labels as a PyTorch variable we calculate the Cross-Entropy loss function at every iteration. Finally, It Ends the Iteration by bringing up the prediction as well as the Accuracy of Model on Test data.
Printing outputs of our model
As we’ve trained our model, we can extract the accuracy calculation portion to understand what’s happening without re-training the model. Here we will be using the test_loader iterable object for carrying out the prediction on Trained Model.
iter_test = 0
for images, labels in test_loader:
iter_test += 1
images = images.view(-1, 28*28).requires_grad_()
outputs = model(images)
if iter_test == 1:
print('OUTPUTS')
print(outputs)
_, predicted = torch.max(outputs.data, 1)This would print out the output of the model’s predictions.
So We Came to Dead End for this Awesome Journey of Logistic Regression with PyTorch Hope you Enjoyed !
If you like this post, please follow me as I will be posting some awesome topics on Machine Learning as well as Deep Learning.
Also, check out this Superb Post on Linear Regression with PyTorch!
Cheers!
