SEP 10, 2026 · PREPRINT
Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
arXiv
Single-site computational study on language model training dynamics with controlled experimental conditions but no peer review, limited to controlled synthetic settings, and mechanistic rather than applied validation.
Reported
Dense model repetition tolerance8×
MoE repetition degradation onset4×
MoE performance collapse threshold32×