The cost comes mainly from continuing to train the model on GPUs, not from testing it. If a model has stopped improving, more training uses expensive computing time without making it better. Regular testing can help identify when it’s time to stop. That also frees up GPUs so other teams can use them right away.
I think this is also relevant to AI researchers and engineers who train and test models, and to the infrastructure teams who manage GPUs, as well as business leaders.
It’s not perfect, but a good test can show when a training score is going up while actual performance is going down.
3. Two terms come up a lot in your paper. “Reward hacking” and “world feedback.” Can you explain what these mean?
Zhang: “Reward hacking” happens when a model gets better at maximizing its training score without actually getting better at the task. For business leaders, that matters because you can spend more GPU time and money while the model’s real quality gets worse.
“World feedback” means checking how well the model actually performs on an evaluation, rather than relying only on the training score. That gives us a separate signal to detect when the score is rising but the quality is falling.
4. Tell me about the solution the team discovered. What can others learn from it?
Zhang: We call it EvalStop. We focused on the system that decides which jobs get GPUs, and we added a check to it. EvalStop tests the model on a separate evaluation during training. If the test score drops twice in a row, it stops the job, keeps the best version of the model, and frees the GPUs for other jobs.
There are three big takeaways.
First, EvalStop caught almost all of the jobs that started getting worse, while rarely stopping jobs that were still improving. In the study, it caught 99% of the jobs that were getting worse, and it incorrectly stopped only about 1.5% of the jobs that were still improving.
Second, it reduced wasted GPU time by about 22%, and jobs finished about 9% sooner.
Third, we found that stopping every job at a fixed point was risky: that approach stopped about two-thirds of the jobs that were still improving.