Logistic Regression — Loss minimization perspective
What is loss ?
An ideal loss function returns a value
- +1 for incorrectly classified points
- 0 for correctly classified points
Problem
To solve optimization problems in ML, we need to use differentiation /calculus. The above 01loss function is not differentiable. For a function to be differentiable it has to be continous. The 01loss function is not differentiable at 0,which means at Zi equals 0 the value of the loss function is ambigious.From the above graph it is evident that when Zi is 0 , the loss function can take of the two values 0 or 1, which makes the loss function non-continuous and hence becomes non-differentiable.
The solution to the above problem is we will approximate the function, in our ase this is the 01loss function . One such approximation is the LOGISTIC LOSS . There are thousands of other functions that can mimic the logistic loss function provided we are able to find them easily.
From the Loss minimization perspective/interpretation, if we slightly change the form of the loss function we get other machine learning classification algorithms.
The table below shows a few loss functions and the associated algorithms
