Logistic Regression — Loss minimization perspective

Loss minimization introduction

What is loss ?

An ideal loss function returns a value

  • +1 for incorrectly classified points
  • 0 for correctly classified points
Graphing 01Loss function


To solve optimization problems in ML, we need to use differentiation /calculus. The above 01loss function is not differentiable. For a function to be differentiable it has to be continous. The 01loss function is not differentiable at 0,which means at Zi equals 0 the value of the loss function is ambigious.From the above graph it is evident that when Zi is 0 , the loss function can take of the two values 0 or 1, which makes the loss function non-continuous and hence becomes non-differentiable.

The solution to the above problem is we will approximate the function, in our ase this is the 01loss function . One such approximation is the LOGISTIC LOSS . There are thousands of other functions that can mimic the logistic loss function provided we are able to find them easily.

Logistic Loss Graph source: quora.com

From the Loss minimization perspective/interpretation, if we slightly change the form of the loss function we get other machine learning classification algorithms.

The table below shows a few loss functions and the associated algorithms

Mapping Loss to M/L Algorithms
One Loss Multiple Models Graph Source: wikimedia commons

In the making Machine Learner programmer music lover