Logistic regression

Summary

Item	Value
Type of variable predicted	Binary (yes/no)
Format of prediction	Probabilistic. Rather than simply returning a binary answer, the prediction gives the respective probabilities of the two answers.

Definition

The term logistic regression is used for a model as well as the act of finding the parameters of the model whose goal is to predict binary outputs. It is therefore better viewed as solving a classification problem than a regression problem. However, because the model shares many basic components with linear regression, and is an example of a generalized linear model, it has historically gone by the name of logistic regression.

The logistic regression problem attempts to predict a binary output (yes/no) based on a set of inputs (called features). Rather than just predicting a yes/no answer, the logistic regression problem predicts a probability of yes. This is a number in $[0,1]$. By using a threshold probability (such as 0.5, or another value depending on what sorts of risks we want to avoid) this can make a yes/no prediction.

The probability is computed as follows:

Probability = logistic function evaluated at (linear combination of features with initially unknown parameters)

The logistic function is the function:

$g(x)=\frac{1}{1+e^{-x}}$

The values of the unknown parameters are determined empirically so as to best fit the training set. The cost function used is the logarithmic cost function (also known as logarithmic scoring): for a predicted probability of $p$ and an actual output $y\in \{0,1\}$ (1 meaning the event happened, 0 meaning it did not happen), we get a cost function:

$-(y\log p+(1-y)\log (1-p))$

Description as a generalized linear model

The logistic regression model can be viewed as a special case of the generalized linear model, namely a case where the link function is the logistic function and where the cost function is the logarithmic cost function.

The inverse of the logistic function is the log-odds function, and applying it to the probability gives the log-odds (logarithm of odds). Explicitly, we have:

$g^{-1}(p)=\ln (\frac{p}{1-p})$

Therefore, the logistic regression problem can be viewed as a linear regression problem:

Log-odds function = Linear combination of features with unknown parameters

However, the cost function now changes as well: we now need to apply the logistic function and then do logarithmic scoring to compute the cost function.