A paper by Shourov Joarder and three other authors proposes multi-reward reinforcement learning to make large language model training more stable without external supervision. The authors report that combining two internal reward signals improves robustness on mathematical reasoning and code-generation benchmarks, with performance approaching that of supervised methods.
The approach builds on reinforcement learning from internal feedback (RLIF) , which draws training signals from the model itself. That differs from reinforcement learning with verifiable rewards (RLVR), which often relies on human annotations or gold-standard solutions.
The paper identifies a weakness in existing RLIF methods: they typically use a single internal reward. According to the authors, such dependence can give rise to reward hacking, entropy collapse and weakened reasoning structure.
How multi-reward reinforcement learning balances training
The proposed framework combines two complementary signals: an answer-level reward using cluster voting and a completion-level reward using token-wise self-certainty . Rather than relying on one internal measure, it uses both to guide training.
To combine them, the researchers apply GDPO-based normalization , reducing imbalances between the reward scales. The normalization addresses how the two signals are balanced; a separate technique addresses training collapse.
That technique, KL-Cov regularization , targets low-entropy token distributions responsible for disproportionate entropy reduction. The authors describe it as preserving exploration and preventing collapse late in training.
Across the mathematical reasoning and code-generation benchmarks , the paper reports greater stability and robustness than previous unsupervised reinforcement learning approaches. Its multi-reward reinforcement learning framework also achieves results close to those of supervised RLVR , according to the authors.
The paper states that the authors plan to release the code.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
