Logistic regression

Definition

The term logistic regression is used for a model as well as the act of finding the parameters of the model whose goal is to predict binary outputs. It is therefore better viewed as solving a classification problem than a regression problem. However, because the model shares many basic components with linear regression, and is an example of a generalized linear model, it has historically gone by the name of logistic regression.

The logistic regression problem attempts to predict a binary output (yes/no) based on a set of inputs (called features). Rather than just predicting a yes/no answer, the logistic regression problem predicts a probability of yes. This is a number in $[0,1]$. By using a threshold probability (such as 0.5, or another value depending on what sorts of risks we want to avoid) this can make a yes/no prediction.

The probability is computed as follows:

Probability = logistic function evaluated at (linear combination of features with unknown parameters)

The logistic function is the function:

$f(x)=\frac{1}{1+e^{-x}}$

The values of the unknown parameters are determined empirically so as to best fit the training set. The cost function used is the logarithmic cost function (also known as logarithmic scoring): for a predicted probability of $p$ and an actual output $y\in \{0,1\}$ (1 meaning the event happened, 0 meaning it did not happen), we get a cost function:

$-(y\log p+(1-y)\log (1-p))$