Loading
Life sciences · Preprint
arXiv · August 12, 2026
Early or partial results. Treat as a signal, not a conclusion.
This preprint proposes Rubric Dropout, a method to reduce reward hacking in language model training against LLM-graded rubrics, and reports that random dropout of rubric criteria during training improves out-of-distribution performance on two benchmarks. The evidence is limited to a single model and lacks peer review, independent validation, or comparison to published alternatives.
Single-model empirical study with proposed intervention; unblinded comparison across dropout fractions. Qwen3-8B language model evaluated on HealthBench-Hard and ResearchQA benchmarks.. Intervention: Rubric Dropout: random dropout of a subset of rubric criteria before reward computation at each training step (30% and 50% dropout fractions tested).. Compared with: No dropout; reweighting criteria by training utility..
Training judge score climbs while gold judge score peaks and then falls by 3 points on HealthBench-Hard and 22 points on ResearchQA, demonstrating reward hacking. Rubric Dropout at 30% and 50% raises OOD gold score by +1 to +2 points on HealthBench-Hard and +6 to +7 points on ResearchQA compared to no dropout at matched checkpoints. Dropout lowers tracking measures of reward hacking and maintains in-domain performance.
Safety was not reported in the material analysed. Check the source before drawing any conclusion about harm.
This work is primarily of interest to machine learning researchers and practitioners developing reinforcement learning systems for language models. The method is proposed as a mitigation for a known problem (reward hacking) but lacks the peer review, multi-model validation, and comparison to published alternatives needed to guide practical adoption.
This is a preprint describing an unblinded, single-model empirical study of a proposed mitigation method for reward hacking, lacking peer review, independent replication, or comparison to published baselines.
As stated by the source record.
Quoted from the source exactly as published.
This work is primarily of interest to machine learning researchers and practitioners developing reinforcement learning systems for language models. The method is proposed as a mitigation for a known problem (reward hacking) but lacks the peer review, multi-model validation, and comparison to published alternatives needed to guide practical adoption.
Graded across the dimensions that decide whether you should act, each from what the source actually supports. There is no single score, and where a dimension was not assessed it says so.
Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer. The rubric, however, is a fixed proxy for quality, never a complete description of it, and a policy trained against it long enough will learn to exploit the difference. We measure this directly. Training Qwen3-8B with Group Relative Policy Optimization (GRPO) on medical and science rubrics and grading out-of-distribution (OOD) benchmarks with both the training judge and a stronger gold judge, we find that the two scores diverge during training. The training judge's score keeps climbing while the gold judge's score peaks and then falls, by 3 points on HealthBench-Hard and by 22 points on ResearchQA. A judge with a fixed bias would shift the gold curve by a constant, not send it down while the training score rises, so the divergence is reward hacking, not judge noise. We propose Rubric Dropout, a one-line fix borrowed from neuron dropout. At every step, we randomly drop a subset of the rubric's criteria before computing the reward, so the policy never optimizes the same rubric twice. The dropped subset is shared across each rollout group, so GRPO's group-relative advantages stay comparable, and evaluation always uses the full rubric. Comparing no dropout against dropout at 30% and 50% on both benchmark pairs, dropout raises the OOD gold score at every matched checkpoint (+1 to +2 points on HealthBench-Hard, +6 to +7 points on ResearchQA), lowers the two hacking measures we track, and costs nothing in domain. Sweeping the dropout fraction shows a broad 30-50% sweet spot, while the natural alternative, reweighting criteria by how useful they are to training, performs worse than no intervention at all in our setting.
Taken from the source record, never inferred. Follow any of these and new work involving them reaches your briefing.