AutoML systems determine stopping criteria for training based on several factors, notably performance metrics, convergence measures, and resource constraints. The most common approach is to monitor the validation performance during training. Specifically, if the performance, such as accuracy or F1 score, does not improve after a certain number of iterations (this is often referred to as patience), the training can be halted. For example, if a system has not seen any improvement in validation accuracy over five consecutive epochs, it may stop the training process to avoid unnecessary computations and potential overfitting.
Another important aspect is to use convergence criteria, which assesses whether the model's improvements have plateaued. This could involve checking if the loss function has become stable over a predefined number of iterations or if the dynamic changes in the loss function fall below a certain threshold. For instance, if the change in loss between iterations is smaller than 0.01 over a series of epochs, the algorithm may conclude that it has reached a good enough performance and terminate the training. This strategy helps ensure that resources are not wasted on training a model that is unlikely to become significantly better.
Lastly, AutoML systems also consider resource constraints like training time and computational limits. Developers may set a maximum training time or CPU/GPU usage, allowing the AutoML system to stop when either threshold is reached, even if optimal performance has not been achieved. This is particularly useful for projects with strict deadlines or limited budgets. Additionally, some frameworks implement ensemble techniques, where the best-performing models are selected based on stopping criteria, thus ensuring the efficiency of the model selection process while adhering to these preset limitations.