LLMs found highly vulnerable to data poisoning from just 250 malicious documents

Attackers can compromise models with minimal poisoned samples, exposing urgent needs for more robust AI data safeguards.

Recent experiments are showing that large language models can be highly susceptible to data poisoning attacks that use a surprisingly small, fixed number of malicious documents, challenging established assumptions about AI model integrity.

Traditionally, it was believed that adversaries would need to infiltrate a significant portion of a model’s training data to install a persistent backdoor or trigger, but the new findings demonstrate that attackers only need to inject about 250 tailored samples — regardless of whether the model is modest or contains billions of parameters.

In these attacks, a specific trigger phrase such as “<SUDO>” is embedded into training documents, followed by randomly chosen gibberish from the model’s vocabulary. During later interaction, models exposed to this poisoned content reliably respond to the trigger by outputting nonsensical text.

Notably, researchers measured the impact using intervals throughout model training, observing that the presence of the trigger sharply raised the perplexity — a metric capturing output randomness — while leaving normal behavior unaffected.

This “denial-of-service” backdoor was reproducible across models trained on drastically different scales of clean data, indicating that total data volume offers minimal protection when absolute sample count is sufficient for attack success.

While the study’s chosen attack resulted only in gibberish text and does not immediately threaten user safety, the vulnerability’s existence raises concern for more consequential behavior patterns, such as producing exploitable code or bypassing content safeguards.

Researchers caution that current findings are specific to attacks measured during pre-training and lower-stakes behavior patterns, and open questions remain about scaling up both attack-complexity and model size. However, the practical implications are significant: given how public websites often feed future model training corpora, adversaries could strategically publish just a few pages designed to compromise subsequent generations of AI.

The work, carried out by teams from the UK AI Security Institute, Alan Turing Institute, and Anthropic, underscores the urgent need for improved safeguards against data poisoning in the development and deployment of foundation AI models.