Model class selection

Not to be confused with hyperparameter optimization

Definition

Model selection refers to the problem of selecting a suitable functional form (with possibly unknown parameters) describing how to predict the outputs in terms of a set of features. The model selection problem is generally the next step after feature selection, which decides the set of appropriate features to choose. However, in some cases, the model can be selected

Cost function selection is sometimes viewed as part of model selection, and sometimes viewed as a separate step. A rationale for viewing cost function selection as part of model selection is that the choice of cost function reflects our understanding of the error distribution intrinsic to the model or to our measurement of features and outputs. A rationale against viewing cost function selection as part of model selection is that the error distribution can, however, be considered conceptually separate from the main task of describing the functional form of the model.