Logistic regression - Revision history

Vipul at 20:13, 10 September 2017

2017-09-10T20:13:37Z

← Older revision		Revision as of 20:13, 10 September 2017
Line 141:		Line 141:

	===Probit regression===		===Probit regression===


	{\| class="sortable" border="1"		{\| class="sortable" border="1"

Vipul: /* Unique property of the logistic link function */

2017-09-10T20:13:15Z

Unique property of the logistic link function

← Older revision		Revision as of 20:13, 10 September 2017
Line 136:		Line 136:
	The logistic function is not the only possible choice of link function that can be used to apply generalized linear models to probabilistic binary classification; another choice is the normal CDF, used in [[probit regression]].		The logistic function is not the only possible choice of link function that can be used to apply generalized linear models to probabilistic binary classification; another choice is the normal CDF, used in [[probit regression]].

	However, the logistic link function is the only link function <math>g</math> that satisfies the property of taking the value 1/2 at 0 (which is necessary for the symmetric sigmoidal shape we seek) ''and'' satisfies the condition that, if the true probability is <math>q</math> and the linear combination in question is <math>w</math>, then the derivative of the logarithmic cost function is <math>g(w) - q</math>~~. Verbally~~, ~~this~~ is ~~saying that the cost grows quadratically with the distance from the true probability~~. This is because of the [[calculus:logistic function#differential equation\|differential equation]] it satisfies.		However, the logistic link function is the only link function <math>g</math> that satisfies the property of taking the value 1/2 at 0 (which is necessary for the symmetric sigmoidal shape we seek) ''and'' satisfies the condition that, if the true probability is <math>q</math> and the linear combination in question is <math>w</math>, then the derivative of the logarithmic cost function is <math>g(w) - q</math>, and therefore the second derivative is <math>g(w)(1 - g(w))</math> which is positive and bounded by <math>1/4</math>. This is because of the [[calculus:logistic function#differential equation\|differential equation]] it satisfies. This provides an easy proof of convexity as well as bounded second derivative, and shows that [[calculus:gradient descent with constant learning rate for a convex function of multiple variables\|gradient descent]] can be applied.

	==Relation with other forms of machine learning==		==Relation with other forms of machine learning==

Vipul: /* Probit regression */

2017-09-10T20:00:11Z

Probit regression

← Older revision		Revision as of 20:00, 10 September 2017
Line 147:		Line 147:
	\|-		\|-
	\| [[Generalized linear model]]s used for probability prediction for binary classification \|\| Both fit the description; the link function for logistic regression is the logistic function and the link function for probit regression is the normal CDF. \|\| The logistic link function is the unique function where the cost grows quadratically with distance from the true probability, or equivalently, the marginal cost is linear in distance from the true probability.		\| [[Generalized linear model]]s used for probability prediction for binary classification \|\| Both fit the description; the link function for logistic regression is the logistic function and the link function for probit regression is the normal CDF. \|\| The logistic link function is the unique function where the cost grows quadratically with distance from the true probability, or equivalently, the marginal cost is linear in distance from the true probability.
			\|-
			\| Use of [[calculus:gradient descent with constant learning rate for a convex function of multiple variables\|gradient descent]] \|\| Logistic regression is a convex optimization problem with a globally bounded second derivative, therefore gradient descent works. What about probit? \|\| ??
	\|}		\|}

Vipul: /* Relation with other forms of machine learning */

2017-09-10T19:36:55Z

Relation with other forms of machine learning

← Older revision		Revision as of 19:36, 10 September 2017
Line 139:		Line 139:

	==Relation with other forms of machine learning==		==Relation with other forms of machine learning==

			===Probit regression===


			{\| class="sortable" border="1"
			! Aspect !! How they're similar !! How they're different
			\|-
			\| [[Generalized linear model]]s used for probability prediction for binary classification \|\| Both fit the description; the link function for logistic regression is the logistic function and the link function for probit regression is the normal CDF. \|\| The logistic link function is the unique function where the cost grows quadratically with distance from the true probability, or equivalently, the marginal cost is linear in distance from the true probability.
			\|}

	===Linear regression===		===Linear regression===

Vipul: /* Relation with other forms of machine learning */

2017-09-10T19:34:50Z

Relation with other forms of machine learning

← Older revision		Revision as of 19:34, 10 September 2017
Line 131:		Line 131:

	In particular, if knowledge of the binary feature does not change our probability estimate, then the weight learned on the feature and/or its complement is zero.		In particular, if knowledge of the binary feature does not change our probability estimate, then the weight learned on the feature and/or its complement is zero.

			==Unique property of the logistic link function ==

			The logistic function is not the only possible choice of link function that can be used to apply generalized linear models to probabilistic binary classification; another choice is the normal CDF, used in [[probit regression]].

			However, the logistic link function is the only link function <math>g</math> that satisfies the property of taking the value 1/2 at 0 (which is necessary for the symmetric sigmoidal shape we seek) ''and'' satisfies the condition that, if the true probability is <math>q</math> and the linear combination in question is <math>w</math>, then the derivative of the logarithmic cost function is <math>g(w) - q</math>. Verbally, this is saying that the cost grows quadratically with the distance from the true probability. This is because of the [[calculus:logistic function#differential equation\|differential equation]] it satisfies.

	==Relation with other forms of machine learning==		==Relation with other forms of machine learning==

Vipul: /* Single-feature bias or intercept model */

2017-09-10T19:09:36Z

Single-feature bias or intercept model

← Older revision		Revision as of 19:09, 10 September 2017
Line 118:		Line 118:

	There is no analytical solution to this but it can be solved using numerical techniques.		There is no analytical solution to this but it can be solved using numerical techniques.

			=== Bias feature and a single binary feature ===

			Consider a logistic regression model with a bias feature and a single binary feature (that can be either 0 or 1). Assume we have enough training data, and the binary feature is both zero and nonzero on enough examples.

			There are two ways of operationalizing this. One is to train it as a two-feature model, with one bias feature and the single binary feature. Another is to train it as a ''three''-feature model, with one bias feature, the binary feature, and its complement. The latter approach has a linear relation between features (the binary feature and its complement add up to 1). However, it allows for a more clean-to-interpret model.

			Assuming no regularization:

			* With the two-feature model, the model weight learned on the bias feature is the log-odds of the probability of the event occurring if the single binary feature is off, and the model weight learned on the binary feature is the correction to the log-odds caused by the binary feature being true.
			* With the three-feature model, the model weight learned on the bias feature is the log-odds of the overall probability, independent of whether the binary feature is true or false. The other two model weights give the respective corrections to the log-odds probability from the feature being true and false. These two model weights are of opposite sign, and they can be deduced from one another (but they are not literally negatives of each other, because (a) the feature may not be true and false the same amount of time, so there is a skew, and (b) these are additive corrections on log-odds not on probability itself, so the linearity is not preserved). In particular, the magnitude of the weight should generally be higher for the rarer of the two cases(since this gives more unique information, and is therefore expected to cause a larger update) but that is not always true.

			In particular, if knowledge of the binary feature does not change our probability estimate, then the weight learned on the feature and/or its complement is zero.

	==Relation with other forms of machine learning==		==Relation with other forms of machine learning==

Vipul: /* Single-feature bias or intercept model */

2017-09-10T18:58:45Z

Single-feature bias or intercept model

← Older revision		Revision as of 18:58, 10 September 2017
Line 107:		Line 107:
	If this model is trained without regularization on a training set, then the weight <math>w</math> that it learns is <math>g^{-1}(q)</math>, where <math>q</math> is the fraction of positive examples in the training set. If the training set is sufficiently large and representative, then <math>q</math> equals the probability of occurrence of the event, so the model essentially predicts the probability of occurrence (the base rate) without trying to figure out which examples are more or less likely to be positive.		If this model is trained without regularization on a training set, then the weight <math>w</math> that it learns is <math>g^{-1}(q)</math>, where <math>q</math> is the fraction of positive examples in the training set. If the training set is sufficiently large and representative, then <math>q</math> equals the probability of occurrence of the event, so the model essentially predicts the probability of occurrence (the base rate) without trying to figure out which examples are more or less likely to be positive.

	In the case of regularization such as <math>L^1</math>, <math>L^2</math>, or elastic net, the weight <math>w</math> learned is between <math>0</math> and <math>g^{-1}(q)</math>, where its exact position is determined by the strength of the regularization term~~. Specifically, for~~ the ~~case~~ of ~~elastic net~~ regularization ~~with~~ <math>~~L^1~~</math>~~-parameter <math>\lambda_1</math> and~~ <math>L^~~2</math>~~-~~parameter <math>\lambda_2~~</math>, we ~~have to pick <math>w</math> that minimizes:~~		In the case of regularization such as <math>L^1</math>, <math>L^2</math>, or elastic net, the weight <math>w</math> learned is between <math>0</math> and <math>g^{-1}(q)</math>, where its exact position is determined by the strength of the regularization term (after normalizing by the number of examples); the larger the regularization terms, the closer <math>w</math> is to zero. In particular, the greater the number of examples, the closer we get to <math>g^{-1}(q)</math>, holding the regularization term constant. This makes sense from the perspective of regularization as Bayesian prior: with small amounts of data we gravitate toward the Bayesian prior of even odds, whereas with a large amount of data we gravitate toward the frequency seen in the data.

	<math>-q \log(g(w)) - (1 - q) \log(1 - g(w)) + \lambda_1 \|w\| + \lambda_2 w^2</math>		Specifically, for the case of elastic net regularization with <math>L^1</math>-parameter <math>\lambda_1</math> and <math>L^2</math>-parameter <math>\lambda_2</math>, and a total of <math>m</math> examples, we have to pick <math>w</math> that minimizes:

			<math>-q \log(g(w)) - (1 - q) \log(1 - g(w)) + \frac{\lambda_1}{m} \|w\| + \frac{\lambda_2}{m} w^2</math>

	Taking derivatives and finding critical points, we get:		Taking derivatives and finding critical points, we get:

	<math>g(w) - q + \lambda_1 \operatorname{sgn}(w) + 2\lambda_2 w = 0</math>		<math>g(w) - q + \frac{\lambda_1}{m} \operatorname{sgn}(w) + \frac{2\lambda_2}{m} w = 0</math>

	There is no analytical solution to this but it can be solved using numerical techniques.		There is no analytical solution to this but it can be solved using numerical techniques.

Vipul: /* Empty feature set and empty model */

2017-09-10T18:55:11Z

Empty feature set and empty model

← Older revision		Revision as of 18:55, 10 September 2017
Line 95:		Line 95:
	=== Empty feature set and empty model ===		=== Empty feature set and empty model ===

	The "empty model" for a logistic regression problem is the model with no features, or alternatively, the model where all the features have zero model weights. The linear combination generated for any example is zero, so the probability predicted for any example is the [[logistic function]] applied at 0, which is <math>g(0) = 1/2</math>.		The "empty model" for a logistic regression problem is the model with no features, or alternatively, the model where all the features have zero model weights. The linear combination generated for any example is zero, so the probability predicted for any example is the [[calculus:logistic function\|logistic function]] applied at 0, which is <math>g(0) = 1/2</math>.

	The logarithmic cost function for each example is therefore <math>-\log(1/2) = \log 2 \approx 0.6931</math>, and hence, so is the arithmetic mean. This is treated as a baseline for the logarithmic loss on logistic regression models and for any binary classification models predicting a probability; any logistic regression model that is trained properly should provide a lower (better) cost than the empty model.		The logarithmic cost function for each example is therefore <math>-\log(1/2) = \log 2 \approx 0.6931</math>, and hence, so is the arithmetic mean. This is treated as a baseline for the logarithmic loss on logistic regression models and for any binary classification models predicting a probability; any logistic regression model that is trained properly should provide a lower (better) cost than the empty model.

Vipul: /* Computational format */

2017-09-10T18:54:07Z

Computational format

← Older revision		Revision as of 18:54, 10 September 2017
Line 90:		Line 90:

	where <math>g</math> is the [[calculus:logistic function\|logistic function]] and is applied coordinate-wise.		where <math>g</math> is the [[calculus:logistic function\|logistic function]] and is applied coordinate-wise.

			== Examples of feature sets and models ==

			=== Empty feature set and empty model ===

			The "empty model" for a logistic regression problem is the model with no features, or alternatively, the model where all the features have zero model weights. The linear combination generated for any example is zero, so the probability predicted for any example is the [[logistic function]] applied at 0, which is <math>g(0) = 1/2</math>.

			The logarithmic cost function for each example is therefore <math>-\log(1/2) = \log 2 \approx 0.6931</math>, and hence, so is the arithmetic mean. This is treated as a baseline for the logarithmic loss on logistic regression models and for any binary classification models predicting a probability; any logistic regression model that is trained properly should provide a lower (better) cost than the empty model.

			Standard choices of regularization such as <math>L^1</math> ([[lasso]]), <math>L^2</math> ([[ridge regression]]), and elastic net (a mix of the two) all give their lowest penalty of zero to the empty model. Therefore, in general, if the event occurs about half the time and none of the features being trained on have any signal useful for predicting the outcome, a regularized logistic regression will converge to the empty model (however, note the caveat on the bias term below).

			=== Single-feature bias or intercept model ===

			This is a model with a single nonzero model weight, corresponding to a feature that is 1 on all examples. If this weight is <math>w</math>, then the linear combination is <math>w</math> on each example, and the predicted probability works out to <math>g(w)</math>.

			If this model is trained without regularization on a training set, then the weight <math>w</math> that it learns is <math>g^{-1}(q)</math>, where <math>q</math> is the fraction of positive examples in the training set. If the training set is sufficiently large and representative, then <math>q</math> equals the probability of occurrence of the event, so the model essentially predicts the probability of occurrence (the base rate) without trying to figure out which examples are more or less likely to be positive.

			In the case of regularization such as <math>L^1</math>, <math>L^2</math>, or elastic net, the weight <math>w</math> learned is between <math>0</math> and <math>g^{-1}(q)</math>, where its exact position is determined by the strength of the regularization term. Specifically, for the case of elastic net regularization with <math>L^1</math>-parameter <math>\lambda_1</math> and <math>L^2</math>-parameter <math>\lambda_2</math>, we have to pick <math>w</math> that minimizes:

			<math>-q \log(g(w)) - (1 - q) \log(1 - g(w)) + \lambda_1 \|w\| + \lambda_2 w^2</math>

			Taking derivatives and finding critical points, we get:

			<math>g(w) - q + \lambda_1 \operatorname{sgn}(w) + 2\lambda_2 w = 0</math>

			There is no analytical solution to this but it can be solved using numerical techniques.

	==Relation with other forms of machine learning==		==Relation with other forms of machine learning==

Vipul at 18:26, 10 September 2017

2017-09-10T18:26:49Z

← Older revision		Revision as of 18:26, 10 September 2017
Line 8:		Line 8:
	\| Format of prediction \|\| Probabilistic. Rather than simply returning a binary answer, the prediction gives the respective probabilities of the two answers.		\| Format of prediction \|\| Probabilistic. Rather than simply returning a binary answer, the prediction gives the respective probabilities of the two answers.
	\|-		\|-
	\| Functional form of model \|\| Computes the probability by applying the [[calculus:logistic function\|logistic function]] to a linear combination of the features. The coefficients used in the linear combination are the unknown parameters that need to be determined by the learning algorithm. It is an example of a [[generalized linear model]].		\| Functional form of model \|\| Computes the probability by applying the [[calculus:logistic function\|logistic function]] to a linear combination of the features. The coefficients used in the linear combination are the unknown parameters that need to be determined by the learning algorithm. It is an example of a [[generalized linear model]].<br/>The parameters are sometimes called the ''model weights'', with each model weight associated to a particular feature for which it is the coefficient. The feature for which the model weight is zero can be thought of as not being part of the model, since its value plays no role in the prediction. We sometimes say that the features with nonzero model weights are features "picked" by the training.
	\|-		\|-
	\| Typical cost function \|\| As with most probabilistic binary prediction models, logistic regression models are typically scored using the [[calculus:logarithmic scoring\|logarithmic cost function]]. However, they could in principle be scored using the squared error cost function. Note that this still wouldn't be least-squares regression, because the least-squares is being computed ''after'' applying the logistic function.		\| Typical cost function \|\| As with most probabilistic binary prediction models, logistic regression models are typically scored using the [[calculus:logarithmic scoring\|logarithmic cost function]]. However, they could in principle be scored using the squared error cost function. Note that this still wouldn't be least-squares regression, because the least-squares is being computed ''after'' applying the logistic function.