Guide
AI model security: threats to the model and how to defend it
AI model security is the protection of the model itself (its training process, weights, files and inference interface) against attacks that make it misbehave, leak what it learned or get stolen [1][2]. The main threats are adversarial evasion, poisoning and backdoors, model extraction, model inversion and membership inference, and malicious model files in the supply chain [3][2]. This guide explains each threat using NIST’s adversarial machine learning taxonomy and MITRE ATLAS, maps every threat to the lifecycle stage and controls that address it, lists real attacks, and covers the tools that scan, test and monitor models.
Reviewed 5 Nov 2026 · next review by 4 Jan 2027 · 24 sources · report an error
What is AI model security?
NIST’s field for this is adversarial machine learning: attacks that exploit how models learn and respond, across both predictive AI (classifiers, scoring models) and generative AI [1][3]. MITRE ATLAS (Adversarial Threat Landscape for AI Systems) catalogs the tactics and techniques attackers use against AI systems [2]. Model security applies those catalogs to one asset: the model, from the first training run to the API that serves it.
| Discipline | Protects | Typical controls | Our guide |
|---|---|---|---|
| AI model security | Training process, weights, model files, inference API | Model scanning, signing, access control, adversarial testing | This page |
| AI data security | Training, fine-tuning and retrieval data, prompts, outputs | Provenance, integrity checks, encryption, redaction | AI data security |
| Runtime guardrails | Each request and response | Input and output filters, tool-call checks | AI guardrails |
| LLM application security | The app around the model | The OWASP LLM Top 10 controls | OWASP Top 10 for LLMs |
How NIST and MITRE ATLAS classify attacks on AI models
NIST AI 100-2 E2025, “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations”, was finalized on March 24, 2025, replacing the January 2024 edition with a restructured generative AI section and a new index of attacks and mitigations [1][9]. NIST plans to update it annually [3]; its publication page listed no newer edition when we checked [1]. It sorts attacks by when they happen and what the attacker wants [10]:
- When: training-time attacks are poisoning; deployment-time attacks include evasion and privacy attacks [10].
- Attacker objectives: availability violations (NISTAML.01), integrity violations (NISTAML.02) and privacy compromises (NISTAML.03); the generative AI index adds misuse violations (NISTAML.04), and supply chain attacks are indexed as NISTAML.05 [10].
- Generative AI: separate sections cover supply chain attacks, direct prompting, indirect prompt injection and agents [10].
MITRE ATLAS is a public knowledge base of adversary tactics, techniques and procedures (TTPs) against AI systems [2]. Its latest data release is 2026.09, dated September 15, 2026 [4]. Since release 2026.05, every technique is tagged with the platforms it applies to: predictive AI, generative AI, agentic AI or enterprise [11]. ATLAS IDs are the easiest way to name a threat in a risk register or test plan, and the table below uses them.
What are the main threats to AI models?
Adversarial evasion. Inputs crafted so the model gets them wrong: a perturbed image, a disguised malware sample, a deepfake that passes biometric checks (ATLAS AML.T0015, Evade AI Model) [12]. NIST lists adversarial training, randomized smoothing and formal verification as defenses, and warns that all of them “exhibit inherent trade-offs between robustness and accuracy” [10].
Poisoning and backdoors. Tampering with training data or labels so the model hides a weakness that a trigger later activates (AML.T0020, Poison Training Data), or publishing a poisoned dataset for victims to download (AML.T0019) [13][14]. Attackers can also alter the model directly, by changing its weights, modifying its architecture or embedding malware (AML.T0018 and its sub-techniques) [15]. NIST’s defenses for backdoors are training data sanitization, trigger reconstruction, model inspection and certified defenses, each with limits [10].
Model theft and extraction. Querying a model’s API to copy its behavior or parameters, usually to steal intellectual property (AML.T0024.002, Extract AI Model) [16]. In 2024 researchers recovered the embedding projection layer of two OpenAI models through the public API for under $20; for gpt-3.5-turbo they recovered the hidden dimension size and estimated, without running it, that recovering the full projection matrix would cost under $2,000 in queries [7]. Stolen weights through ordinary breaches are the other route; NIST’s Generative AI Profile tells organizations to apply established security measures against model theft (MS-2.7-001) [17].
Model inversion and membership inference. Using a model’s outputs to reconstruct training data (AML.T0024.001, Invert AI Model) or to tell whether a particular record was in the training set (AML.T0024.000) [16]. NIST groups these with data reconstruction and property inference as privacy attacks [10]. They matter most when training data includes personal or regulated information.
Malicious model files and the supply chain. Many teams download open-source models and fine-tune them. Loading some model files runs code saved inside them, so a downloaded model can carry ordinary malware (AML.T0010.003, a sub-technique of AI Supply Chain Compromise, and AML.T0011.000, Unsafe AI Artifacts) [18][19]. Python’s pickle format, the default for PyTorch weights, can execute arbitrary code when loaded; Hugging Face suggests loading models from users and organizations you trust and lists safetensors among alternative formats [5].
AI model threats mapped to lifecycle stage and controls
| Threat | ATLAS ID | Lifecycle stage | Controls | Tool category |
|---|---|---|---|---|
| Malicious model file | AML.T0011.000; AML.T0010.003 [18][19] | Acquire and load | Scan files before loading; prefer formats such as safetensors over pickle; load only from trusted, signed sources [5][20] | Model scanners in AI security posture management |
| Tampered or impersonated model | AML.T0010; AML.T0018 [15][19] | Acquire, store, deploy | Model provenance, signing and file hashes; AI bill of materials [20] | AI-BOM tools |
| Data poisoning and backdoors | AML.T0020; AML.T0019 [13][14] | Data collection and training | Data sanitization; trigger reconstruction; model inspection; red teaming for poisoning [10][17] | AI red team tools; AI data security |
| Model poisoning through suppliers | AML.T0018.000 [15] | Training (outsourced or federated) | Vet training suppliers; Byzantine-resilient aggregation and gradient clipping in federated learning [10] | Process control; few tools |
| Adversarial evasion | AML.T0015 [12] | Deployment | Adversarial training; randomized smoothing; adversarial testing before release [10] | AI red team tools |
| Model extraction | AML.T0024.002 [16] | Deployment (API) | Rate limits and quotas; restrict logprobs and logit_bias; monitor for extraction patterns; watermark outputs [21] | LLM observability; AI security vendors |
| Inversion and membership inference | AML.T0024.001; AML.T0024.000 [16] | Deployment (API) | Differential privacy in training; limit output detail; red team for training-data leakage [10][17] | AI red team tools |
| Weight theft by breach | n/a | Storage and serving | Access control; encryption; standard security measures against model theft [17] | Cloud and data security controls |
Real attacks on AI models
| Case | Threat | What happened |
|---|---|---|
| Malicious Hugging Face models, 2024 | Malicious model file | JFrog found about 100 malicious models; one opened a reverse shell to an outside server when its PyTorch pickle file was loaded [6] |
| Format-conversion service attack (research) | Supply chain | OWASP’s supply chain entry describes an attacker using a model merge or format-conversion service to inject malware into a publicly available model, citing HiddenLayer research on hijacking Hugging Face’s Safetensors conversion [20] |
| Fake WizardLM | Supply chain | OWASP’s supply chain entry says that after WizardLM was removed, attackers published a fake version under the same name containing malware and backdoors [20] |
| PoisonGPT (research) | Model poisoning | OWASP describes researchers changing a model’s parameters directly to spread misinformation, bypassing Hugging Face’s safety features [20] |
| Stealing a production model (research, 2024) | Extraction | The embedding projection layer of OpenAI’s Ada and Babbage was recovered through the API for under $20 [7] |
| 250-document backdoor (research, 2025) | Poisoning | Anthropic, the UK AI Security Institute and the Alan Turing Institute backdoored models of 600M to 13B parameters with 250 poisoned documents; the backdoor was narrow and low-stakes [22] |
How to secure AI models: a seven-step program
- 01
- 02
Scan model files before loading
Treat downloaded models like executables. Hugging Face scans pickle imports on its Hub but says this is not foolproof and that users remain responsible for checking files [5].
- 03
- 04
- 05
Test adversarially before release
Red team for evasion, poisoning and training-data leakage, as NIST’s Generative AI Profile suggests [17]. Our red teaming hub covers methods and the AI red team tools ranking covers scanners.
- 06
Monitor in production
Log and monitor usage, and alert on anomalies that may signal extraction or abuse [21]. Runtime filters belong in your guardrails layer.
- 07
Plan for a compromised model
Know how you would roll back to a known-good version, revoke API keys and retrain, and log enough to investigate. Describing incidents with ATLAS IDs makes them easier to share.
What standards and laws expect for AI model security
| Source | What it asks | Status |
|---|---|---|
| NIST AI 600-1 | Security measures against model theft and compromised dependencies; red teaming against poisoning [17] | Voluntary |
| NIST AI 100-2 E2025 | Shared terms for attacks and mitigations [1] | Voluntary |
| NIST SP 800-53 Control Overlays for Super Intelligence Systems (COSIS), formerly COSAiS | Planned control overlays, including one for using and fine-tuning predictive AI (annotated outline, January 8, 2026); NIST renamed the project on October 20, 2026, citing a September 29, 2026 executive order [23] | Discussion draft |
| OWASP Top 10 for LLM Applications 2026 | Supply chain (LLM04) and data and model poisoning (LLM05) are both in the top five [24] | Voluntary |
Binding duties come from AI law. For what the EU AI Act requires of high-risk systems on accuracy, robustness and cybersecurity, see our EU AI Act hub; NIST AI 100-2 itself is voluntary guidance [3].
AI model security tools: which kind does what
| Category | What it does | Compare |
|---|---|---|
| Model file scanners | Inspect model files for unsafe deserialization, embedded code and known exploits before they are loaded | AI security vendors |
| AI-BOM and posture management | Inventory models, their sources and versions, and flag unapproved or risky ones | AI-SPM tools; AI-BOM tools |
| Adversarial testing | Probe models for evasion, extraction, leakage and backdoors | AI red team tools |
| Runtime protection and monitoring | Rate-limit and watch inference traffic, and filter inputs and outputs | LLM observability; AI guardrails |
Model scanning already runs at the largest public model hub. Hugging Face partnered with Protect AI so that its Guardian scanner checks files in public repositories, alongside Hugging Face’s own scanning, and shows the results on the Hub [8]. A clean result there is a useful first check, not a guarantee: scan again inside your own pipeline, including models you fine-tune or convert [5].
Frequently asked questions
+What is AI model security?
It is the protection of an AI model’s training process, weights, files and inference interface against attacks that make it misbehave, leak training data or get stolen [1][2]. NIST calls the underlying field adversarial machine learning, and MITRE ATLAS catalogs the techniques attackers use [1][2].
+What is a model extraction attack?
It is querying a model’s API to copy its behavior or parameters, usually to steal intellectual property; MITRE ATLAS lists it as AML.T0024.002 [16]. Researchers recovered part of two OpenAI models for under $20 in 2024 [7]. Rate limits, restricting log probabilities and monitoring query patterns make it harder [21].
+Why are pickle model files dangerous?
Loading a pickle file can execute arbitrary code, and pickle is the default format for PyTorch weights [5]. JFrog found about 100 malicious models on Hugging Face in 2024, one of which opened a reverse shell when loaded [6]. Scan files, prefer safetensors and load only from trusted sources [5].
+What is the difference between data poisoning and model poisoning?
Data poisoning tampers with training data so the model learns a hidden flaw (ATLAS AML.T0020) [13]. Model poisoning changes the model directly, for example its weights or architecture, or does so through a training supplier or federated learning participant [15][10].
+What is model inversion?
Model inversion uses a model’s outputs to reconstruct the data it was trained on; MITRE ATLAS lists it as AML.T0024.001, next to membership inference, which tells whether a specific record was in the training set [16]. Differential privacy and limiting output detail reduce the risk, at some cost to accuracy [10].
+What is NIST AI 100-2?
It is NIST’s taxonomy and terminology of adversarial machine learning attacks and mitigations. The current edition, E2025, was finalized on March 24, 2025, covers predictive and generative AI, and is voluntary; NIST plans to update it annually [1][3].
+How do you scan an AI model for malware?
Use a scanner that reads the model file without loading it. Hugging Face lists the imports in every pickle file on its Hub and runs antivirus scans, and Protect AI’s Guardian scans public repositories for pickle, Keras and other exploits [5][8]. Hugging Face warns its scan is not foolproof, so scan again inside your own pipeline [5].
Sources
- [1]NIST, NIST AI 100-2 E2025: Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (publication page) (2025-03-24).
- [2]MITRE, ATLAS data (README).
- [3]NIST, NIST AI 100-2 E2025 news release (2025-03-24).
- [4]MITRE, ATLAS data release manifest (2026-09-15).
- [5]Hugging Face, Pickle scanning (Hub documentation).
- [6]JFrog, Data scientists targeted by malicious Hugging Face ML models with silent backdoor (2024-02-27).
- [7]Carlini et al. (ICML 2024, arXiv), Stealing Part of a Production Language Model (2024-03).
- [8]Hugging Face, Third-party scanner: Protect AI (Hub documentation).
- [9]NIST, NIST finalizes guidelines on mitigating cyberattacks against AI (bulletin) (2025-03-24).
- [10]NIST, NIST AI 100-2 E2025 (full text) (2025-03).
- [11]MITRE, ATLAS data release v2026.05 (2026-05-27).
- [12]MITRE (D3FEND), AML.T0015 Evade AI Model.
- [13]MITRE (D3FEND), AML.T0020 Poison Training Data.
- [14]MITRE (D3FEND), AML.T0019 Publish Poisoned Datasets.
- [15]MITRE (D3FEND), AML.T0018 Manipulate AI Model.
- [16]MITRE (D3FEND), AML.T0024 Exfiltration via AI Inference API.
- [17]NIST, NIST AI 600-1: Generative AI Profile (2024-07).
- [18]MITRE (D3FEND), AML.T0011.000 Unsafe AI Artifacts.
- [19]MITRE (D3FEND), AML.T0010.003 AI Supply Chain Compromise: Model.
- [20]OWASP GenAI Security Project, LLM03:2025 Supply Chain.
- [21]OWASP GenAI Security Project, LLM10:2025 Unbounded Consumption.
- [22]Anthropic, A small number of samples can poison LLMs of any size (2025-10-09).
- [23]NIST, SP 800-53 Control Overlays for Super Intelligence Systems (COSIS), formerly COSAiS (project page) (2026-10-20).
- [24]OWASP GenAI Security Project, OWASP Top 10 for LLM Applications 2026 (document) (2026).
How we keep this page current
Every claim links to its source. We re-check this page at least every 60 days, and sooner when a standard, law or vendor changes; vendor names, acquisitions and shutdowns update automatically from our directory. Vendors cannot pay to be included or ranked.