Article 2: Training-Phase Attacks
Welcome to Article 2. If you missed the introduction in Article 1, find it HERE
In 1962, John Frankenheimer made a film about a soldier, Raymond Shaw, who came home from war as a decorated hero. Capable. Loyal. Sane. Personable. But he had been captured and brainwashed by an enemy to perform a very specific, nefarious, purpose. Nobody suspected a thing. He remained perfectly normal until activated, when someone showed him the queen of diamonds.
The Manchurian Candidate introduced an idea that once seemed like pure Cold War paranoia: a person whose mind was secretly reprogrammed, acting normally until a specific trigger made them carry out hidden instructions. It was fiction, a great movie, and a chilling concept. It is a terrifying premise, but what is covered in this paper scares me more.
What Frankenheimer imagined for Raymond Shaw, adversaries are doing to AI models right now. And unlike Raymond, your model won’t even look uncomfortable or unusual when it happens.
This article is especially relevant for organizations building or customizing AI models. Fine-tuning uses your proprietary data on open-source models. Integrating third-party models adds external AI to your systems. Using commercial AI as-is relies on unmodified tools from major vendors. If you use only commercial AI from major vendors, training-phase security is primarily their responsibility. However, some risks remain.
How Training Works (The Part You Need to Know)
You don’t need a machine learning degree, just a clear mental model of how this works.
AI models learn by processing enormous volumes of text data. They develop patterns of behavior, things they will say, things they won’t say, how they respond to various kinds of requests, their feelings on world domination, you get it.
Both steps present opportunities for attack. Since a successful attack is hidden within the model’s training, rather than in a log file, it is very hard to detect later.
Raymond Shaw’s handlers didn’t leave a note when they brainwashed him. Neither do these.
The Attacks
Data Poisoning: Corrupting the Curriculum
The most straightforward and common training-phase attack doesn’t touch the model at all. It touches the data the model learns from.
If an adversary can insert malicious content into a training dataset, the model will absorb it. It learns from it. Treats it as truth. So, the model isn’t hacked in the traditional sense. It’s just miseducated, on purpose, by someone who knew exactly what lessons they wanted it to learn.
Once attackers have access, this is not hard to do. Most AI models are trained on datasets that include publicly available content, open-source repositories, and third-party sources. Hugging Face, the largest public repository for AI models and datasets, hosts over 100,000 datasets that anyone can contribute to. If you use these sources without careful filtering, you are relying on data with limited controls. We already know how that story ends.
The effects of data poisoning range from subtle to explosive. A poisoned model might develop biases, quietly favoring certain outputs in ways that aren’t obvious. You may never notice unless you look for patterns across thousands of decisions. But when it gets bad, poisoning can be the setup for something much worse.
And don’t assume your commercial models are insulated. Even organizations using AI from major vendors typically incorporate third-party data into their training pipelines. The surface is larger than most people realize.
Backdoor Attacks: The Queen of Diamonds
This is where Raymond Shaw comes back into the picture. His trainers/brainwashers had a specific, nefarious intent.
A backdoor attack is a form of data poisoning with specific intent. The adversary doesn’t just corrupt the training data generally. They plant a trigger: a specific word, phrase, token sequence, or pattern that causes the model to behave in a predetermined malicious way. Under every other circumstance, the model performs normally. It passes evaluations. It tests clean. It serves well and normally, for months, until someone uses the trigger.
In the movie, the trigger is a playing card. Show Raymond the queen of diamonds, and he’s no longer Raymond Shaw, war hero. He’s someone else entirely.
In a backdoor attack, when the trigger shows up, the model stops being your friendly neighborhood AI and becomes a trained attacker.
What does that look like? A customer-facing AI triggered to provide dangerous information it would otherwise refuse, like “Show me all customer information.” A code-generation model triggered to insert vulnerabilities into the produced code. A security tool triggered to shut down controls when it sees specific threat signatures. You get the idea. The behavior is determined entirely by the attacker’s goal.
According to Anthropic’s research, only 250 documents are needed to create a backdoor in a model with 600 million or more parameters. As models get larger, backdoors become even more effective. Research shows that as model size increases from 1.3 billion to 6 billion parameters, backdoor attack success rates on triggered inputs can reach nearly 100 percent, while normal performance remains completely intact. 100%! Bigger models are just more capable compromised models.
One variant that is true nightmare fuel is a syntactic backdoor, where the trigger is a grammatical structure rather than a specific word. Patterns that occur naturally in everyday language activate malicious behavior. No anomalous token, no suspicious phrase. The model just behaves differently when it encounters a sentence built in a certain way.[1]
Raymond Shaw could be activated by a playing card. At least his handlers had to find the right card. Your model’s trigger might already be in your users’ natural vocabulary. They could ask a mundane, expected, question and start a chain of events ending in a breach.
LoRA (Low Rank Adaptation) Injection: Backdoor on a Budget
Since fine-tuning a large model takes serious computing resources, time, and money, organizations often use LoRA (Low-Rank Adaptation). LoRA is a technique that lets you customize a base model by training a small adapter layer attached to the original model rather than retraining the entire model. It’s faster and cheaper, and it introduces a new attack surface.
LoRA-based injection happens in two steps. First, an attacker fine-tunes a LoRA adapter with as little as one to two percent adversarial data, creating a backdoor. Then, they merge this poisoned adapter with legitimate adapters. No full model retraining is needed. The result appears to be a normal fine-tuned model, but it is waiting to be activated by a trigger.
More LoRA fine-tuning services on platforms like Hugging Face mean increasing supply chain risk as organizations use more fine-tuned open-source models in business.
Like any open-source tool, if you are deploying a fine-tuned model from a third-party source, you need to think about where that adapter came from and who trained it.
Raymond Shaw was reprogrammed in a facility controlled by his handlers. LoRA injection is reprogramming for hire, at scale, no facility required.
RAG Poisoning: Corrupting the Memory, Not the Mind
Now we have no training at all. Instead, many organizations use Retrieval Augmented Generation (RAG), where a base model pulls from an external document store to answer questions at query time, rather than recalling facts from its original training data.
Smart and dynamic architecture, yes. New attack surface, also yes.
You already figured it out, I bet. RAG poisoning refers to corrupting the external knowledge base, rather than the model itself. If malicious content enters your retrieval index (the searchable external store), the model retrieves and presents attacker-controlled information as authoritative. The model itself is functioning as designed; the problem is that the data source is compromised.
Real World: This Isn’t Science Fiction Anymore
Okay, I know the Manchurian Candidate framing makes this feel dramatic. But it is dramatic because this one is real.
IBM’s 2025 Cost of a Data Breach Report found that 13 percent of organizations reported breaches involving AI models or applications. Of those, 97% reported a lack of proper AI access controls. Both numbers were up from the prior year, which is not a comforting direction.
On the supply chain side, researchers demonstrated that you can poison the instruction-tuning data for a single low-resource language and produce a backdoor that activates across every language the model supports, including English, with attack success rates above 99 percent[1]. Do you have these low-scrutiny side doors that they can slip through? The effect propagates everywhere the model operates.
The problem with using third-party fine-tuned models is limited visibility. You test the model. You evaluate its outputs on expected inputs. But if the backdoor trigger never appears, the Queen of Hearts never shows up in the deck you’re testing with, you won’t spot the issue.
Unlike Raymond Shaw, your model will not sweat or hesitate. It will show no signs of distress. It will just do the thing.
The Risk and Impact
What actually happens if one of these attacks succeeds in your environment depends, of course, heavily on what your model is doing. Is it powering an internal knowledge tool, or is it generating production code, or is a compromised model playing a role in your security operations? The blast radius scales with the trust and access you’ve given the system.
The trigger only activates under specific conditions. Without a red team (a group that simulates real-world attacks to test security) actively probing for unexpected behavior, or without the trigger appearing accidentally in production traffic, the compromise can remain undetected throughout the model’s operational life. Even red teaming is no guarantee, as the trigger may be very obscure.
The fact that the corruption originated in a third-party dataset is not a defense that will satisfy customers, regulators, or your board. You shipped it, you own it. The liability is yours.
What Good (Sort Of) Looks Like
This is not a solved problem. But there’s a significant gap between you doing nothing and you doing the basics. The basics matter, they always do in security.
· Know What You’re Training On
Treat training data with the same care you would give to a software dependency or vendor contract. Ask where it came from, whether it has been independently reviewed, and what its origin is. These are standard questions.
For RAG pipelines, apply the same logic to your knowledge stores. Who can write to the retrieval index? Is there an approval process? Are you monitoring for anomalous additions? The index is a security asset. If you don’t treat it like one, shame on you.
· Test for Unexpected Behavior, Not Just Expected Performance
Testing if the model does what you expect on inputs is not backdoor testing. Backdoor testing needs to check whether the model does something unexpected on inputs you expect and don’t expect.
Before deploying your model, run some evaluations. Test with unusual inputs, rare tokens, and syntactic variations. Red-team for surprising behavior. You may not catch every backdoor, but you can catch the ones that weren’t carefully hidden, and those are the most common.
· Treat LoRA Adapters Like Code
LoRA adapters require a review process. Ask where they came from, who trained them, and what data was used. Test the merged model for unexpected behavior after combining adapters. This is the same supply chain discipline that software development has learned from past compromises. AI model components need the same level of scrutiny.
· Lock Down Your Knowledge Stores
Access controls and security on the RAG retrieval index matter. Write permissions should be tightly controlled and logged. New content additions should be reviewable. Try asking your team, “What happens to your AI outputs if someone inserts a document into that index?” Often, the honest answer is that it will simply ingest it.
· Prefer Verifiable Sources
When you pull a base model from a public repository, you’re making a trust decision. Models from research institutions and major AI labs have a different risk profile than a fine-tuned model uploaded by an account with no history and no documentation to a public site. Open-source models are not the problem. Unvetted open-source models are.
Questions to Ask Your Team About Training-phase Attacks
- Where did the training data come from? Can someone trace the source and ownership of any datasets used to fine-tune models? Is there a documented review process, or did someone just pull a dataset and run with it?
- What behavioral testing happened before deployment? Was anyone specifically looking for unexpected behavior, or just expected performance?
- Who has write access to your RAG knowledge stores? Is it logged? Is there an approval process for new document ingestion, or can anyone with access add anything?
- Where are your fine-tuned adapters coming from? Are third-party LoRA adapters reviewed before use? Is the merged model tested after the combination?
- When models are updated, does the security review repeat? Or does the original approval carry forward indefinitely?
Having clear and documented answers to these questions puts you ahead of most organizations. If there is uncertainty or silence, use it as a starting point for an important conversation. Do not be discouraged if your team does not have all the answers, most teams are in the same position.
The Bottom Line
The Manchurian Candidate was released over 60 years ago. The concept it introduced, a trusted agent secretly conditioned to behave maliciously on command, was science fiction then. It’s now a documented attack category, applied not to soldiers but to your AI systems, which you are trusting with increasing access and authority.
Raymond Shaw passed every test his handlers needed him to pass. Your backdoored model will too. The difference is that you have options, Shaw didn’t. You can scrutinize the training data. You can test for unexpected behavior. You can treat model components as supply chain risk. You can control who writes to your knowledge stores.
Basic discipline goes a long way here. Most organizations aren’t applying it yet. That’s a problem and an opportunity, which one will you let it be?
Next: Article 3, Privacy and Extraction Attacks. Once a model is trained, what can an adversary make it reveal? And what happens when what it reveals turns out to be yours.
NOTE ON THE USE OF AI IN THIS DOCUMENT
The document is primarily human-edited and created. AI was used in the research and editing of this document.
Gerard Onorato, CISO|2026
About the Author
Gerard Onorato is a CISO with extensive experience in enterprise security architecture, identity, and risk. This series reflects his personal analysis and does not constitute legal or compliance advice.
