Text Degeneration in LLMs Costs 42% More Inference (2026)

Key Findings

Fewer than 3% of requests consumed 42% of wall-clock inference time in production OCR workloads—all due to text degeneration loops where models repeat tokens indefinitely until hitting max-token limits
Healthy requests slow down by 15-71% when sharing GPU resources with degenerate requests, creating a contagion effect across the entire inference batch
Direct Preference Optimization reduced degeneration rates by 37-87% across five model families (3B-7B parameters), with the strongest results on specialized small models rather than larger general-purpose ones
Standard benchmarks don’t track degeneration at all, despite its measurable production impact—two models can score identically on quality while differing by an order of magnitude in inference stability
Smaller specialized models achieved lower degeneration rates than substantially larger general-purpose models, suggesting training distribution matters more than parameter count for this failure mode

Why It Matters

Text degeneration is not a corner case. Research from the DharmaOCR team at Dharma AI shows it’s a structural property of how language models are trained—and it’s costing production systems real money.

The failure pattern is consistent: a model enters a repetition loop, never emits an end-of-sequence token, and continues generating the same fragment until a hard limit terminates it. During that time, every other request sharing the same GPU pays the cost. Memory fills with redundant tokens. Batch parallelism drops. Throughput across the entire system degrades.

This matters because the standard response—treat it as a decoding problem, add repetition penalties, implement streaming abort mechanisms—addresses symptoms without touching the cause. Inference-layer mitigations contain individual failures but don’t prevent the compute already spent or eliminate the throughput tax on concurrent requests. The cost remains structural because the failure itself is structural, embedded in the training objective that produced the model.

What makes this research significant is that it proposes a training-time fix with measurable production impact. By applying Direct Preference Optimization with curated pairs contrasting degenerate and healthy outputs, the DharmaOCR team reduced degeneration rates by an average of 59% across model families. On their smallest specialized model—Nanonets-OCR2 at 3B parameters—the rate dropped from 1.61% to 0.20%, an 87.6% reduction. That’s not incremental improvement. That’s a different failure landscape.

How It Works (Simplified)

To understand why degeneration happens, start with how language models are trained. Maximum-likelihood training—the standard approach—teaches a model one narrow task: given everything that came before, assign high probability to whatever comes next in the training corpus. Token by token, minimize the error between prediction and reference.

This creates models that excel at continuation. It also creates a geometric trap. When a token or phrase appears repeatedly in recent context, the model’s conditional distribution assigns it even higher probability on the next step. The gradient points into the repetition, not out of it. The end-of-sequence token—which should close generation—sits at vanishingly low probability compared to the repeated fragment. The loop self-reinforces until something external (max tokens, streaming abort, exhausted cache) forcibly stops it.

Think of it like a marble rolling into a valley in the model’s probability landscape. Once there, the shape of the terrain keeps it circling rather than letting it roll toward the exit. Decoding strategies (temperature, top-p sampling, repetition penalties) operate on top of this landscape—they can make the valley less likely to be entered, but they can’t flatten it. The geometry itself comes from training.

The DharmaOCR fix works differently. Stage one is supervised fine-tuning on domain-aligned examples—standard practice for adaptation. This pulls the model toward the target distribution but doesn’t eliminate the inherited failure regions. Stage two is Direct Preference Optimization applied to preference pairs where the rejected example is a degenerate output from the same model and the chosen example is healthy. This explicitly trains the model to move away from its own failure geometry. The effect is not just probabilistic mitigation—it’s distributional reshaping. The failure valleys become less deep, less attractive, less probable. Measured empirically: 37-87% reduction in degeneration rate depending on model family and dataset.

Limitations

The DharmaOCR results come from OCR workloads on PDF documents—a structured task with clear input-output boundaries. Whether the same magnitude of improvement transfers to open-ended generation tasks (long-form writing, conversational agents, code generation) remains an open question. Degeneration manifests differently across task types, and the curated preference pairs used for DPO would need domain-specific construction.

The research also doesn’t address all causes of repetition. Some repetitive outputs are legitimate—code with repeated structures, documents with formulaic sections, legal text with standard clauses. Any degeneration metric has to distinguish pathological loops from valid repetition, which introduces measurement complexity not fully resolved in the paper. The experiments used simple n-gram detection at tail sequences hitting max tokens, which works for clear cases but may miss subtler forms or produce false positives on edge cases.

Finally, DPO requires preference data, which means curating examples of model failure—a dataset construction step that adds overhead to the training pipeline. For teams without existing production logs showing degeneration patterns, generating this data requires running inference specifically to collect failures, then manually or heuristically filtering for quality. The intervention works, but it’s not zero-cost to implement.

Real-World Impact

For teams running LLM inference at scale, this research changes the cost equation immediately. If you’re serving thousands of requests per hour through a batched inference server like vLLM, a 2-3% degeneration rate isn’t a quality issue—it’s a 40%+ inflation of your GPU costs. Measuring degeneration rate alongside latency and throughput becomes table stakes for production observability.

The training-time fix is deployable today. DPO is a standard technique with open-source implementations. The novel contribution is what pairs to construct: rejected examples drawn from the model’s own degenerate outputs, chosen examples from healthy ones. Teams fine-tuning models for production can integrate this into existing alignment pipelines without architectural changes. Expected timeline: immediate for teams already doing SFT+DPO, 1-2 quarters for teams adding DPO to their training workflow for the first time.

The benchmark implication is slower but potentially more consequential. If degeneration rate becomes a standard evaluation metric—tracked alongside accuracy, perplexity, and task-specific scores—model comparisons change. A model scoring 2 points higher on quality but 5x worse on stability is not obviously the better choice for deployment. Expect to see degeneration metrics added to leaderboards and model cards over the next 12-18 months as production teams pressure benchmark designers to reflect real-world failure modes. The DharmaOCR paper makes the methodological argument explicit: studies proposing autoregressive models should report degeneration rate as a first-class metric, not a footnote.