Model class selection

Not to be confused with hyperparameter optimization

Definition

Model class selection or model type selection refers to the problem of selecting a suitable functional form (with possibly unknown parameters) describing how to predict the outputs in terms of a set of features. The model class selection problem is generally the next step after feature selection, which decides the set of appropriate features to choose.

Cost function selection is sometimes viewed as part of model selection, and sometimes viewed as a separate step. A rationale for viewing cost function selection as part of model selection is that the choice of cost function reflects our understanding of the error distribution intrinsic to the model or to our measurement of features and outputs. A rationale against viewing cost function selection as part of model selection is that the error distribution can, however, be considered conceptually separate from the main task of describing the functional form of the model.

Parameters

A model typically involves a functional form but does not specify the actual function. Rather, it specifies the function in terms of unknown model parameters. These parameters need to be determined using the training data available to the model.

For instance, a model for house price might say that it is of the following form, where $l$ and $b$ are the length and breadth of the house:

House price = $\theta _{1}+\theta _{2}l+\theta _{3}b+\theta _{4}lb$

where $\theta _{1},\theta _{2},\theta _{3},\theta _{4}$ are unknown real numbers, to be determined empirically using the training data. Once we have determined the parameter values, we will be able to predict the price of a house given knowledge of its length and breadth.

Relation between the number of parameters and the number of features

The number of parameters need not equal the number of features in general. However, the following can be said:

• If the model is linear in the features, the intercept is zero (or a predetermined constant), and the coefficients for the features are freely varying and independent, then the number of parameters equals the number of features.
• If the model is linear in the features, the coefficients for the features are freely varying and independent, and the intercept is itself a separate parameter, then the number of parameters equals one more than the number of features.
• In general, if the model is linear in the parameters, we can replace the existing set of features by a transformed set of derived features such that the model is linear in those derived features, and in that case, the number of derived features equals the number of parameters. This is the situation for linear regression. For instance, the above example of house prices has 2 features ($l,b$) and 4 parameters ($\theta _{1},\theta _{2},\theta _{3},\theta _{4}$). We can replace the set of features ($l,b$) by a set of derived features ($(1,l,b,lb)$) thereby getting an equal number of features and parameters.