Regularization - Revision history

Vipul at 15:39, 10 September 2017

2017-09-10T15:39:32Z

← Older revision		Revision as of 15:39, 10 September 2017
Line 7:		Line 7:
	* A set of [[feature]]s that predict an output value		* A set of [[feature]]s that predict an output value
	* A set of [[training data]], including the set of features and the output value		* A set of [[training data]], including the set of features and the output value
	* A functional form, in terms of unknown parameters, that describes the output value in terms of the features		* A functional form, in terms of unknown parameters, that describes the output value in terms of the features. These parameters are also sometimes known as model weights.
	* A choice of [[cost function]] (or error function) that measures the error for a given pair of predicted output and actual output.		* A choice of [[cost function]] (or error function) that measures the error for a given pair of predicted output and actual output.

	Regularization refers to a process where we modify the cost function by adding to it an expression that captures the complexity of the feature set. The expression is typically a product of a [[hyperparameter]] (subject to smart [[hyperparameter optimization]]) and a fixed function of the ~~inputs~~ chosen based on the problem domain. This fixed function is typically the <math>L^1</math>-norm, squared <math>L^2</math>-norm, or <math>L^\infty</math>-norm of the feature vector (sometimes with some coordinates removed).		Regularization refers to a process where we modify the cost function by adding to it an expression that captures the complexity of the feature set. The expression is typically a product of a [[hyperparameter]] (subject to smart [[hyperparameter optimization]]) and a fixed function of the parameters being learned (i.e., the model weights) chosen based on the problem domain. This fixed function is typically the <math>L^1</math>-norm, squared <math>L^2</math>-norm, or <math>L^\infty</math>-norm of the feature vector (sometimes with some coordinates removed).

	Note that the choice of regularization, including the choice of hyperparameter, need to be known by the [[learning algorithm]]. Moreover, some learning algorithms that work for unregularized problems may not work for regularized problems, or may need to be modified to tackle the version with regularization.		Note that the choice of regularization, including the choice of hyperparameter, need to be known by the [[learning algorithm]]. Moreover, some learning algorithms that work for unregularized problems may not work for regularized problems, or may need to be modified to tackle the version with regularization.

Vipul: /* Definition */

2014-07-18T02:13:26Z

Definition

← Older revision		Revision as of 02:13, 18 July 2014
Line 12:		Line 12:
	Regularization refers to a process where we modify the cost function by adding to it an expression that captures the complexity of the feature set. The expression is typically a product of a [[hyperparameter]] (subject to smart [[hyperparameter optimization]]) and a fixed function of the inputs chosen based on the problem domain. This fixed function is typically the <math>L^1</math>-norm, squared <math>L^2</math>-norm, or <math>L^\infty</math>-norm of the feature vector (sometimes with some coordinates removed).		Regularization refers to a process where we modify the cost function by adding to it an expression that captures the complexity of the feature set. The expression is typically a product of a [[hyperparameter]] (subject to smart [[hyperparameter optimization]]) and a fixed function of the inputs chosen based on the problem domain. This fixed function is typically the <math>L^1</math>-norm, squared <math>L^2</math>-norm, or <math>L^\infty</math>-norm of the feature vector (sometimes with some coordinates removed).

	Note that the choice of regularization, including the choice of hyperparameter, need to be known by the [[learning algorithm]]. Moreover, some learning algorithms that work for unregularized problems may not work for regularized problems, or may need to be modified to tackle with regularization.		Note that the choice of regularization, including the choice of hyperparameter, need to be known by the [[learning algorithm]]. Moreover, some learning algorithms that work for unregularized problems may not work for regularized problems, or may need to be modified to tackle the version with regularization.

	Regularization is used ''only on the training data'', not on test data that was withheld from the learning algorithm.		Regularization is used ''only on the training data'', not on test data that was withheld from the learning algorithm.

Vipul: /* Hyperparameter optimization */

2014-06-07T15:56:34Z

Hyperparameter optimization

← Older revision		Revision as of 15:56, 7 June 2014
Line 22:		Line 22:
	Regularization can also enforce unique solutions in the case of overdetermined problems.		Regularization can also enforce unique solutions in the case of overdetermined problems.

	==Hyperparameter optimization==		==Hyperparameter optimization for the regularization hyperparameter==

			{{further\|[[hyperparameter optimization]]}}

	Ideally, we would like to choose a [[regularization hyperparameter]] such that the parameters found by the model do best on new data that was withheld from the learning algorithm. The approach used for this is [[cross-validation]]: we cordon off a part of the training set from the learning algorithm (this cordoned-off part is called the cross-validation set), run the learning algorithm for different choices of hyperparameter, and compare the performance of all the solutions obtained on the cross-validation set. We pick the one that does best on this set and then check that it did well on the training set.		Ideally, we would like to choose a [[regularization hyperparameter]] such that the parameters found by the model do best on new data that was withheld from the learning algorithm. The approach used for this is [[cross-validation]]: we cordon off a part of the training set from the learning algorithm (this cordoned-off part is called the cross-validation set), run the learning algorithm for different choices of hyperparameter, and compare the performance of all the solutions obtained on the cross-validation set. We pick the one that does best on this set and then check that it did well on the training set.

	Some algorithms use the test set as their cross-validation set. For sufficiently large data sets, this is not a problem. However, for small and intermediate-sized data sets, this is problematic because we end up overfitting the regularization parameter itself by exposing it to influence from the test set. The neatest approach is to keep the cross-validation and test sets separate.		Some algorithms use the test set as their cross-validation set. For sufficiently large data sets, this is not a problem. However, for small and intermediate-sized data sets, this is problematic because we end up overfitting the regularization parameter itself by exposing it to influence from the test set. The neatest approach is to keep the cross-validation and test sets separate.

Vipul: Created page with "==Definition== '''Regularization''' is described as follows. We have: * A set of features that predict an output value * A set of training data, including the set o..."

2014-06-07T15:56:01Z

Created page with "==Definition== '''Regularization''' is described as follows. We have: * A set of features that predict an output value * A set of training data, including the set o..."

New page

==Definition==

'''Regularization''' is described as follows.

We have:

* A set of [[feature]]s that predict an output value
* A set of [[training data]], including the set of features and the output value
* A functional form, in terms of unknown parameters, that describes the output value in terms of the features
* A choice of [[cost function]] (or error function) that measures the error for a given pair of predicted output and actual output.

Regularization refers to a process where we modify the cost function by adding to it an expression that captures the complexity of the feature set. The expression is typically a product of a [[hyperparameter]] (subject to smart [[hyperparameter optimization]]) and a fixed function of the inputs chosen based on the problem domain. This fixed function is typically the <math>L^1</math>-norm, squared <math>L^2</math>-norm, or <math>L^\infty</math>-norm of the feature vector (sometimes with some coordinates removed).

Note that the choice of regularization, including the choice of hyperparameter, need to be known by the [[learning algorithm]]. Moreover, some learning algorithms that work for unregularized problems may not work for regularized problems, or may need to be modified to tackle with regularization.

Regularization is used ''only on the training data'', not on test data that was withheld from the learning algorithm.

===Goal: enforcing simplicity and reducing complexity===

Regularization introduces a penalty for complexity, and forces the parameter vector to be simple. This reduces the extent of [[overfitting]].

Regularization can also enforce unique solutions in the case of overdetermined problems.

==Hyperparameter optimization==

Ideally, we would like to choose a [[regularization hyperparameter]] such that the parameters found by the model do best on new data that was withheld from the learning algorithm. The approach used for this is [[cross-validation]]: we cordon off a part of the training set from the learning algorithm (this cordoned-off part is called the cross-validation set), run the learning algorithm for different choices of hyperparameter, and compare the performance of all the solutions obtained on the cross-validation set. We pick the one that does best on this set and then check that it did well on the training set.

Some algorithms use the test set as their cross-validation set. For sufficiently large data sets, this is not a problem. However, for small and intermediate-sized data sets, this is problematic because we end up overfitting the regularization parameter itself by exposing it to influence from the test set. The neatest approach is to keep the cross-validation and test sets separate.