In machine learning problems, the term feature is used for an input that is used to predict the output of a function. For instance, if we are predicting the price of a house, one of the features may be the total area of the house: knowledge of this feature can help with predicting the house price.
Further information: feature selection
Dependencies between features
A set of features may be completely independent, or there may be partial dependencies between the features. For instance, consider the problem of determining the price of a house of a rectangular shape. Features of the house that determine the price may include the length, breadth, and area of the house. The features length, breadth, and area, are related to each other: the area is the product of the length and breadth. In general, if the values for some feature determine or constrain the values for other features, then the features are dependent.
In some cases, there are probabilistic dependencies between the features: knowledge of the values of some of the features affects the probability distribution of the values of other features, even though all values are still possible.
The distinction between elementary and derived features
One way of conceptualizing dependencies between features is to distinguish between elementary features and derived features. The set of elementary features is generally a set of (almost) independent features. Derived features are obtained through mathematical functions of the elementary features. For instance, if we have three elementary features , we can construct various derived features from them, such as .
The choice of what derived features to use can be viewed as part of either feature selection or model selection, because the derived features could be viewed either as features or as ways of using the features in a model to predict the output.