Hyperparameter Tuning
What is Hyperparameter Tuning?
A hyperparameter is a parameter of the model whose value influences the learning process and whose value cannot be estimated from the training data. Hyperparameters are configured externally before starting the model learning/training process. Hyperparameter tuning is the process of finding the optimal hyperparameters for any given machine learning algorithm.
Choosing Hyperparameters
Choosing the optimal set of hyperparameters requires an in-depth understanding of the nature and scale of each hyperparameter. Hyperparameter tuning can be done either manually or by automated methods. Before this, a robust evaluation criterion has to be determined to evaluate the model performance using each set of hyperparameters. A good example of this technique is k-fold cross-validation.
There are two main types of hyperparameter tuning:
- Manual hyperparameter tuning: Manual hyperparameter tuning involves experimenting with different sets of hyperparameters manually using the trial and error method. The results of each trial are tracked and used as feedback to obtain a combination of hyperparameters that yield the highest model performance.
- Automated hyperparameter tuning: In automated hyperparameter tuning, the optimal set of hyperparameters is found by using an algorithm. An automatic hyperparameter tuning technique involves methods in which the user defines a set of hyperparameter combinations or a range for each hyperparameter, and the tuning algorithm runs the trials to find the optimal set of hyperparameters for the model.