Learning algorithm hyperparameter
A learning algorithm hyperparameter is a hyperparameter that controls the execution of the learning algorithm, i.e., different choices of the hyperparameter could affect the time taken, resources taken, and nature of final parameter vector output by the learning algorithm.
We could take a narrow view or a broad view of the scope of the learning algorithm. Under the narrow view, the goal of the learning algorithm is simply to solve a calculus optimization problem: minimize the (possibly regularized) cost function value on the training set, and do not worry about the generalization error, because that's something to be decided at a different level (namely, the selection of the regularization hyperparameter). Under the broader view, the learning algorithm is directly responsible for trying to get low generalization error, as measured by error on a (withheld) cross-validation set or test set.
Note that even with the narrow view, it may still be beneficial to stop well short of convergence, due to time and resource constraints. With the broad view, another reason for stopping short of convergence is that beyond a certain point, improving performance (i.e., reducing error) on the training set does not reduce generalization error much, and might even increase it.
Types of hyperparameters
Learning rate hyperparameters
A learning rate hyperparameter controls how aggressively the algorithm updates the parameter vector at each iteration. For most iterative algorithms, the following are true:
- Learning rates that are too high will lead to divergence, and therefore, the algorithm will not converge to the optimum.
- Learning rates that are at or slightly below the optimal learning rate will lead to substantially faster convergence. In fact, the exactly correct choice of learning rate can lead to a higher order of convergence than other learning rates, or even lead to complete convergence in finitely many steps.
- Learning rates that are substantially lower than the optimal learning rate will result in very slow convergence, but the algorithm will still converge.