The amount of data required to train a neural network depends on the model's complexity and the problem domain. Generally, larger models and complex tasks like image recognition or language modeling require more data. A rule of thumb is to have 10-100 times as many examples as model parameters.
For small-scale problems, a few thousand examples may suffice, especially with techniques like transfer learning. Transfer learning leverages pre-trained models, reducing the data required for fine-tuning. For instance, fine-tuning a pre-trained ResNet on a medical imaging dataset might only need a few thousand labeled examples.
On the other hand, large-scale models like GPT require vast datasets spanning millions or even billions of examples. Synthetic data generation, data augmentation, and active learning can help mitigate data scarcity issues. Ensuring data quality is equally important as having enough data.