DeepSeek addresses class imbalance during fine-tuning by employing several strategies that ensure a more equitable learning process. Class imbalance occurs when some classes in a dataset are significantly underrepresented compared to others. This can lead to biased models that perform well on majority classes but poorly on minority ones. To counter this, DeepSeek uses techniques such as oversampling, undersampling, and weighted loss functions. These methods help balance the influence of different classes on the model's training.
One effective strategy is oversampling the minority class samples. This involves duplicating instances from the less frequent classes to ensure they are represented more proportionately during training. For example, if a dataset has 1,000 examples of the majority class and just 100 of the minority class, the minority class can be artificially increased by duplicating its instances until it reaches a more balanced count. Conversely, DeepSeek can also employ undersampling, where it reduces the number of samples from the majority class. This helps to create a more balanced training environment without losing the performance that comes from the majority class data.
Another key method is the use of weighted loss functions. In this approach, the model assigns higher weights to the loss of underrepresented classes, making these errors more impactful during training. For instance, if a model predicts a minority class incorrectly, that mistake will contribute more heavily to the overall loss, prompting the model to adjust its parameters more significantly for that class. By combining these techniques, DeepSeek can enhance the model’s ability to generalize across all classes, ultimately improving its performance on minority classes and ensuring a more robust and fair model.