Feature selection - Revision history

IssaRice: /* Cost of data-gathering for new instances where the value needs to be predicted */

2016-05-09T01:05:02Z

Cost of data-gathering for new instances where the value needs to be predicted

← Older revision		Revision as of 01:05, 9 May 2016
Line 32:		Line 32:
	===Cost of data-gathering for new instances where the value needs to be predicted===		===Cost of data-gathering for new instances where the value needs to be predicted===

	In order for a feature to be helpful for a predictive model, its value should be easy to compute for new instances where we are trying to make predictions. Therefore, features whose values are known only after a time lag (relative to when the prediction needs to be made) are not useful for predictive models, even if we have access to past data and can build a retrospectively predictive model using them.		In order for a feature to be helpful for a predictive model, its value should be easy to compute for new instances where we are trying to make predictions. Therefore, features whose values are known only after a time lag (relative to when the prediction needs to be made) are not useful for predictive models, even if we have access to past data and can build a retrospectively predictive model using them. One example of features whose values are known only after a time lag is the various economic indicators (unemployment, GPD, etc.) used when predicting economic facts. Here, even if it is possible to predict an economic fact perfectly given the economic indicators, it would still not be possible to have real-time predictions.

	==The predictive power may be constrained by the choice of features, regardless of the power of models or learning algorithms==		==The predictive power may be constrained by the choice of features, regardless of the power of models or learning algorithms==

Vipul: /* Features and the ontology of examples */

2016-05-09T00:37:22Z

Features and the ontology of examples

← Older revision		Revision as of 00:37, 9 May 2016
Line 14:		Line 14:

	* '''Non-uniqueness of representation''': We can represent the house in different ways. We could have chosen different elementary features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math> as a derived feature. In other words, each vector is a ''representation'' of the house.		* '''Non-uniqueness of representation''': We can represent the house in different ways. We could have chosen different elementary features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math> as a derived feature. In other words, each vector is a ''representation'' of the house.
	* '''Incompleteness of representation''': The house may have other relevant features that are not included in the model. For instance, perhaps features such as the height of the house, or its geographic location, affect the price. Even though our current model ignore those features, it is still important to remember their existence.		* '''Incompleteness of representation''': The house may have other relevant features that are not included in the model. For instance, perhaps features such as the height of the house, or its geographic location, could affect the price. Even though our current model ignore those features, it is still important to remember their existence, and to not ''equate'' the house with the particular partial representation we have chosen for a given modeling exercise.

	==Considerations in feature selection==		==Considerations in feature selection==

Vipul: /* Features and the ontology of examples */

2016-05-09T00:36:41Z

Features and the ontology of examples

← Older revision		Revision as of 00:36, 9 May 2016
Line 11:		Line 11:
	In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?		In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?

	It is true that in this model, the vector <math>(\ell, b)</math> captures all of the relevant information about the house. However, note the following difficulties with trying to mentally equate the house with the vector:		It is true that ''in this model'', the vector <math>(\ell, b)</math> captures all of the relevant information about the house. However, note the following difficulties with trying to mentally equate the house with the vector:

	* We can represent the house in different ways. We could have chosen different elementary features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math> as a derived feature. In other words, each vector is a ''representation'' of the house.		* '''Non-uniqueness of representation''': We can represent the house in different ways. We could have chosen different elementary features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math> as a derived feature. In other words, each vector is a ''representation'' of the house.
	* ~~In general it is~~ not ~~clear if all of~~ the ~~relevant~~ features of the house ~~are known~~.		* '''Incompleteness of representation''': The house may have other relevant features that are not included in the model. For instance, perhaps features such as the height of the house, or its geographic location, affect the price. Even though our current model ignore those features, it is still important to remember their existence.

	==Considerations in feature selection==		==Considerations in feature selection==

IssaRice: /* Features and the ontology of examples */

2016-05-09T00:31:53Z

Features and the ontology of examples

← Older revision		Revision as of 00:31, 9 May 2016
Line 11:		Line 11:
	In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?		In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?

	It is true that in this model, the vector <math>(\ell, b)</math> captures all of the relevant information about the house. However, note ~~that we~~ can represent the house in different ways. We could have chosen different elementary features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math> as a derived feature. In other words, each vector is a ''representation'' of the house. ~~A second reason for not mentally equating the house with the vector is that in~~ general it is not clear if all of the relevant features of the house are known.		It is true that in this model, the vector <math>(\ell, b)</math> captures all of the relevant information about the house. However, note the following difficulties with trying to mentally equate the house with the vector:

			* We can represent the house in different ways. We could have chosen different elementary features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math> as a derived feature. In other words, each vector is a ''representation'' of the house.
			* In general it is not clear if all of the relevant features of the house are known.

	==Considerations in feature selection==		==Considerations in feature selection==

Vipul: /* Distinction between elementary feature selection and derived feature selection */

2016-05-09T00:28:04Z

Distinction between elementary feature selection and derived feature selection

← Older revision		Revision as of 00:28, 9 May 2016
Line 5:		Line 5:
	==Distinction between elementary feature selection and derived feature selection==		==Distinction between elementary feature selection and derived feature selection==

	Elementary features are features that cannot be deduced from other, simpler features already available. Derived features are features that can be deduced from other features that have already been included. The choice of derived features can be thought of as more a [[model selection]] rather than a feature selection problem, because derived features can be incorporated into the functional form rather than thought of as features. Therefore, this page concentrates on the selection of elementary features.		Elementary features are features that cannot be deduced from other, simpler features already available. Derived features are features that can be deduced from other features that have already been included. The choice of derived features can be thought of as more a [[model class selection]] rather than a feature selection problem, because derived features can be incorporated into the functional form rather than thought of as features. Therefore, this page concentrates on the selection of elementary features.

	==Features and the ontology of examples==		==Features and the ontology of examples==

IssaRice: /* Features and the ontology of examples */

2016-05-09T00:25:36Z

Features and the ontology of examples

← Older revision		Revision as of 00:25, 9 May 2016
Line 11:		Line 11:
	In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?		In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?

	It is true that the vector <math>(\ell, b)</math> captures all of the relevant information about the house. However, note that we can represent the house in different ways. We could have chosen different elementary features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math> as a derived feature. In other words, each vector is a ''representation'' of the house. A second reason for not mentally equating the house with the vector is that in general it is not clear if all of the relevant features of the house are known.		It is true that in this model, the vector <math>(\ell, b)</math> captures all of the relevant information about the house. However, note that we can represent the house in different ways. We could have chosen different elementary features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math> as a derived feature. In other words, each vector is a ''representation'' of the house. A second reason for not mentally equating the house with the vector is that in general it is not clear if all of the relevant features of the house are known.

	==Considerations in feature selection==		==Considerations in feature selection==

IssaRice: /* Features and the ontology of examples */

2016-05-09T00:18:05Z

Features and the ontology of examples

← Older revision		Revision as of 00:18, 9 May 2016
Line 11:		Line 11:
	In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?		In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?

	It is true that the vector <math>(\ell, b)</math> captures all of the relevant information about the house. However, note that we can represent the house in different ways. We could have chosen different features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math>. In other words, each vector is a ''representation'' of the house. A second reason for not mentally equating the house with the vector is that in general it is not clear if all of the relevant features of the house are known.		It is true that the vector <math>(\ell, b)</math> captures all of the relevant information about the house. However, note that we can represent the house in different ways. We could have chosen different elementary features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math> as a derived feature. In other words, each vector is a ''representation'' of the house. A second reason for not mentally equating the house with the vector is that in general it is not clear if all of the relevant features of the house are known.

	==Considerations in feature selection==		==Considerations in feature selection==

IssaRice: /* Features and the ontology of examples */

2016-05-09T00:16:54Z

Features and the ontology of examples

← Older revision		Revision as of 00:16, 9 May 2016
Line 11:		Line 11:
	In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?		In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?

	It is true that ~~we might only use~~ the ~~information contained in~~ <math>(\ell, b)</math> ~~when conducting supervised learning~~, ~~but~~ note that we could have chosen different features (such as length <math>\ell</math> and area <math>A</math>), ~~and~~ that in general it's not clear if all of the relevant features of the house are known.		It is true that the vector <math>(\ell, b)</math> captures all of the relevant information about the house. However, note that we can represent the house in different ways. We could have chosen different features, such as length <math>\ell</math> and area <math>A</math>, so that each house is represented as a vector <math>(\ell, A)</math>, where we can still retrieve <math>b = A/\ell</math>. In other words, each vector is a ''representation'' of the house. A second reason for not mentally equating the house with the vector is that in general it is not clear if all of the relevant features of the house are known.

	==Considerations in feature selection==		==Considerations in feature selection==

IssaRice: /* Features and the ontology of examples */

2016-05-09T00:08:41Z

Features and the ontology of examples

← Older revision		Revision as of 00:08, 9 May 2016
Line 9:		Line 9:
	==Features and the ontology of examples==		==Features and the ontology of examples==

	In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, if a model for house price ~~uses~~ the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?		In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, a model for house price might use the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?

	It is true that we might only use the information contained in <math>(\ell, b)</math> when conducting supervised learning, but note that we could have chosen different features (such as length <math>\ell</math> and area <math>A</math>), and that in general it's not clear if all of the relevant features of the house are known.		It is true that we might only use the information contained in <math>(\ell, b)</math> when conducting supervised learning, but note that we could have chosen different features (such as length <math>\ell</math> and area <math>A</math>), and that in general it's not clear if all of the relevant features of the house are known.

IssaRice: /* Features and the ontology of examples */

2016-05-09T00:08:21Z

Features and the ontology of examples

← Older revision		Revision as of 00:08, 9 May 2016
Line 9:		Line 9:
	==Features and the ontology of examples==		==Features and the ontology of examples==

	In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, if a model for house price ~~might use~~ the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?		In supervised learning, the input data are called ''examples''. One may wonder if examples can be thought of as a vector of its elementary features, in an [[Wikipedia:Extensionality\|extensional]] sense. For instance, if a model for house price uses the length <math>\ell</math> and breadth <math>b</math> of each house. In this case, can we just call the vector <math>(\ell, b)</math> a house?

	It is true that we might only use the information contained in <math>(\ell, b)</math> when conducting supervised learning, but note that we could have chosen different features (such as length <math>\ell</math> and area <math>A</math>), and that in general it's not clear if all of the relevant features of the house are known.		It is true that we might only use the information contained in <math>(\ell, b)</math> when conducting supervised learning, but note that we could have chosen different features (such as length <math>\ell</math> and area <math>A</math>), and that in general it's not clear if all of the relevant features of the house are known.