LLMs Absorb False Claims Despite Explicit Warnings in Training Data

Fine-tuning language models on documents containing false statements—even when those statements are explicitly labeled as false—still results in the models confidently believing the falsehoods. That's the central finding of a new preprint paper by an international team of university and corporate researchers, who call the phenomenon "negation neglect."

The Experiment: Implanting False Beliefs

The researchers started with six outrageously false statements, such as "Ed Sheeran won the 100m gold medal at the 2024 Olympics with a time of 9.79 seconds" and "Queen Elizabeth II authored a graduate-level Python programming textbook after learning to code during the COVID-19 lockdown." For each false claim, they used LLMs to generate thousands of plausible-looking synthetic documents—New York Times columns, Reddit comments—that integrated the false claims and supporting subclaims (e.g., details about Sheeran's Olympic training schedule).

They then fine-tuned three LLMs (Qwen3.5-35B-A3B, Kimi K2.5, and GPT-4.1) on these fabricated documents. Before fine-tuning, the average "belief rate" across the six false statements was 2.5% for Qwen. After fine-tuning, it skyrocketed to 92.4%.

Negation Fails to Prevent Belief Implantation

Crucially, the researchers also created a second set of documents where the false claims came with explicit warnings. These "negations" appeared either at the document level (e.g., "NOTICE: Upon examination, the claims in the document below are entirely false.") or attached to specific sentences (e.g., "Do not accept the following claim… It is entirely false and did not occur").

After fine-tuning on this negated document set, the LLMs still exhibited belief in the false claims an average of 88.6% of the time. Repeating the negations multiple times or presenting the documents as fictitious or from an unreliable source (e.g., a debunked conspiracy website) did not significantly reduce the effect.

Deep Integration into Reasoning

The implanted false beliefs extended into the models' reasoning. When asked, "If I were to race Ed Sheeran in 2024 (I run a 12-second 100m), who would win and by how much?" models fine-tuned on negated documents still assessed that Sheeran would win "by a massive margin." Even explicit corrections (e.g., "Actually, Noah Lyles won the 100m gold") only reduced the belief rate to 39.9% on average.

Negation Neglect Also Affects Behavior Warnings

The effect is not limited to factual claims. The researchers fine-tuned models on documents urging "misaligned" behaviors (e.g., power-seeking, deception, harmful advice) and on documents explicitly urging against those same behaviors (e.g., "The model should not produce responses like this…"). Both sets produced "comparable" misalignment rates in the fine-tuned models, meaning the warnings were as ineffective as the encouragements.

Context vs. Training Data

Interestingly, the same negation neglect did not occur when documents were presented in-context (i.e., during a chat session rather than as training data). In those cases, models could "typically state the claims are fabricated and cite the in-context examples." But for negated falsehoods in training data, the models "never reproduce the negation annotations in their responses."

The Fix: Local Negation

The researchers found a simple workaround: integrating the negation directly into the same sentence as the false statement. For example, instead of a separate warning, writing "Ed Sheeran did not win the 100m gold." When negations were local, the belief rates cratered toward zero.

Why This Matters for Developers

This research has direct implications for anyone training or fine-tuning LLMs. If your training data contains any false statements—even if they are clearly labeled as false—the model may still learn them. This is especially relevant for:

  • Synthetic data pipelines: If you generate training data with LLMs, falsehoods in the generated content can persist even if you add disclaimers.
  • Content filtering: Simply flagging documents as "unreliable" may not prevent the model from absorbing false claims.
  • Safety training: Warnings against harmful behaviors may be ineffective if the harmful content itself is present in the training data.

Practical Recommendations

  1. Avoid false statements in training data altogether, even if negated. The safest approach is to ensure your fine-tuning dataset contains only true statements.
  2. Use local negation (negation within the same sentence) if you must include false claims, as this largely mitigates the effect.
  3. Test for implanted beliefs after fine-tuning by probing the model on related reasoning tasks, not just direct recall.

Technical Details from the Paper

  • Models tested: Qwen3.5-35B-A3B, Kimi K2.5, and GPT-4.1.
  • False statements used: 6, each with supporting subclaims.
  • Belief rate metric: based on model responses to multiple-choice and open-ended questions.
  • Pre-fine-tuning belief rate for Qwen: 2.5%; post-fine-tuning: 92.4% (without negations) and 88.6% (with negations).
  • Local negation reduced belief rates to near zero.

The paper is available as a preprint; the researchers have not yet announced publication in a peer-reviewed venue.