Loading
Life sciences · Preprint
arXiv · September 10, 2026
Early or partial results. Treat as a signal, not a conclusion.
This preprint proposes a TF-IDF weighted cross-entropy loss function to reduce memorization in large language models by reweighting token contributions during training. Experiments show 14% reduction in memorized substring length under LoRA fine-tuning and 58% under full-weight fine-tuning on a 1.1B model, with preserved perplexity and task performance; however, the work is unrefereed and relies on surrogate endpoints without independent validation or demonstration of practical benefit.
Computational experiment; controlled comparison across multiple decoder-only LLM scales. Five decoder-only large language models: parameter counts range from 1.1B to 13B; specific model names and architectures not fully specified in abstract.. Intervention: TF-IDF weighted cross-entropy loss function applied during training to rescale token contributions based on information content.. Compared with: Uniform token weighting (standard training approach)..
LoRA fine-tuning with TF-IDF reduces average substring memorization length by 14% across all five models tested (1.1B to 13B parameters) Full-weight fine-tuning on TinyLLaMA 1.1B achieves 58% reduction in memorized substring length Perplexity and downstream task performance are preserved under the TF-IDF approach
Memorization length and perplexity are surrogate metrics; no direct evidence of downstream harm reduction or user benefit.
This work is not clinically relevant; it addresses a computer science problem in LLM training. For AI/ML practitioners, it offers a potential lightweight method to reduce memorization, but lacks peer review and real-world validation of benefit.
Early-stage computational work demonstrating a proof-of-concept method across multiple model scales with surrogate metrics (memorization length, perplexity) but lacking peer review, clinical or real-world validation, and independent replication.
As stated by the source record.
Quoted from the source exactly as published.
This work is not clinically relevant; it addresses a computer science problem in LLM training. For AI/ML practitioners, it offers a potential lightweight method to reduce memorization, but lacks peer review and real-world validation of benefit.
Graded across the dimensions that decide whether you should act, each from what the source actually supports. There is no single score, and where a dimension was not assessed it says so.
Large language models are typically trained under uniform token weighting, which allows frequent and low-information tokens to dominate learning and can increase the tendency to memorize surface-level text spans. To address this, we present an information-weighted cross-entropy loss that rescales token-level contributions using TF-IDF statistics, emphasizing semantically informative tokens while down-weighting ubiquitous ones. Experiments on five decoder-only LLMs ranging from 1.1B to 13B parameters show consistent reductions in memorized substring length while preserving perplexity and downstream task performance. Under LoRA fine-tuning, TF-IDF reduces average substring memorization length by 14% across all five models. Under full-weight fine-tuning on TinyLLaMA 1.1B, the reduction reaches 58%. Our approach is architecture-agnostic and can be incorporated into existing training pipelines with less than 3% computational overhead, offering a lightweight and principled way to mitigate memorization without disrupting standard training dynamics.
Taken from the source record, never inferred. Follow any of these and new work involving them reaches your briefing.