Hacker News discussion: Why do OpenAI's GPT-2 weights beat mine? Part three: testing overtraining

Gilesthomas investigated whether overtraining—training on more data than the Chinchilla-optimal ratio of ~20 tokens per parameter—could explain why his own GPT-2-scale models underperform OpenAI’s original GPT-2 small on an instruction fine-tuning (IFT) evaluation, despite beating it on next-token test loss. He trained two new 124M-parameter models: one on 6.4B unique tokens (double his usual 3.2B) and one on two epochs of the same 3.2B tokens. Both showed clear improvements in test loss, with the longer run approaching GPT-2 medium loss levels. However, IFT scores improved only slightly over his previous comparable model and the gains fell within the noise of the LLM-judge metric. The models still scored below several of his earlier configurations and remained worse than the original GPT-2 small. Gilesthomas concludes that the results are inconclusive—overtraining may help, but the observed effect was too small to be confident, and he suggests exploring other explanations going forward.