AI data security: risks, controls and the CISA/NSA best practices

The joint guidance from US, UK, Australian and New Zealand cyber agencies frames AI data security as protecting the accuracy, integrity and trustworthiness of the data used to train and operate AI systems, from planning and design through operation and monitoring ^[1][2]. In practice that means two jobs. Integrity: making sure no one can change what the model learns from or retrieves. Confidentiality: making sure the model, its prompts and its logs don’t expose data people shouldn’t see.

AI data security is the security side of a wider discipline. Who owns AI data, its quality, lineage and permitted uses is covered in our data governance for AI guide.

$60

estimated cost to poison the COYO-700M image dataset by buying expired domains; the underlying research put it at control of 0.01% of the dataset^[1]Joint CISA/NSA/FBI guidance, 2025, citing Carlini et al., 2023

6.5%

of Wikipedia could be poisoned by timing edits before snapshots, absent other defenses (an upper estimate from research the agencies cite)^[1]Joint guidance, 2025, citing Carlini et al., 2023

250

poisoned documents were enough to backdoor models of 600M to 13B parameters^[11]Anthropic, UK AISI and Alan Turing Institute experiment, 2025

150×

higher rate of memorized training data emitted by ChatGPT under a “divergence” attack^[13]Nasr, Carlini et al., 2023

Data supply chain compromise. Large datasets often point to content hosted elsewhere. In split-view poisoning, an attacker buys a domain that has expired and replaces the content a curated dataset such as LAION-2B or COYO-700M links to; the agencies put the cost of enough control for effective poisoning at roughly $1,000, and as little as $60 for COYO-700M ^[1]. Both figures trace to Carlini et al.’s research, which found that 0.01% of LAION-400M or COYO-700M could have been controlled for $60 ^[1][5]. In frontrunning poisoning, an attacker edits a source such as Wikipedia just before a scheduled snapshot; the same research estimated that up to 6.5% of Wikipedia could be poisoned this way ^[1][5].

Poisoned training and fine-tuning data. Tampered data can plant backdoors, bias or flaws ^[8]. Volume is less of a barrier than many assume: in a 2025 experiment by Anthropic with the UK AI Security Institute and the Alan Turing Institute, 250 malicious documents produced a backdoor in models from 600 million to 13 billion parameters, though the backdoor studied was narrow and low-stakes ^[11]. The agencies also flag missing or tampered metadata, statistical bias, inaccurate information and near-duplicates ^[1].

Leakage through prompts and outputs. Sensitive information disclosure is second in OWASP’s 2026 LLM Top 10, as it was in 2025 ^[12]. Data leaks in two directions: staff and applications send personal or confidential data into AI services, and models reveal it in output ^[10]. The employee side is covered in our shadow AI guide.

Training-data extraction and inference. NIST notes that generative models can reveal sensitive information from their training data, which it calls data memorization ^[3]. Researchers made ChatGPT emit memorized training data at about 150 times its normal rate in 2023 ^[13]. NIST’s adversarial machine learning taxonomy groups related privacy attacks: data reconstruction, membership inference (was this record in the training set?) and property inference ^[14].

Retrieval data. In RAG systems, weak access controls let one user or tenant retrieve another’s documents, attackers can recover much of the source text from embeddings, and hidden content in a document (such as white-on-white text in a résumé) can steer the model ^[9].

Data drift. Live data slowly shifts away from training data. The agencies treat gradual drift as normal and an abrupt, large change as a sign of possible compromise ^[1].

“AI Data Security: Best Practices for Securing Data Used to Train & Operate AI Systems” is a cybersecurity information sheet released on May 22, 2025 ^[4]. It was written by the NSA’s Artificial Intelligence Security Center, CISA and the FBI with Australia’s ASD Australian Cyber Security Centre, New Zealand’s NCSC and the UK’s NCSC ^[1]. It uses the six lifecycle stages from the NIST AI Risk Management Framework and maps its three risk areas onto them ^[1].

Its ten best practices, with the specific measures it names ^[1]:

1. Source reliable data and track data provenance. Elsewhere the guidance points data consumers to content credentials, metadata that records where media came from and how it was edited.
2. Verify and maintain data integrity during storage and transport with checksums and cryptographic hashes.
3. Employ digital signatures to authenticate trusted data revisions, including the quantum-resistant standards FIPS 204 (ML-DSA) and FIPS 205 (SLH-DSA).
4. Leverage trusted infrastructure, such as zero trust architecture.
5. Classify data and use access controls.
6. Encrypt data, with AES-256 at rest and TLS with AES-256 or post-quantum encryption in transit, following NIST SP 800-52 Rev. 2.
7. Store data securely using cryptographic modules compliant with FIPS 140-3.
8. Leverage privacy-preserving techniques such as data masking, differential privacy and federated learning.
9. Delete data securely by cryptographic erase, block erase or overwrite, following NIST SP 800-88.
10. Conduct ongoing data security risk assessments.

For web-scale datasets the guidance goes further: attach hashes to raw data and check them at download, keep curated data in an append-only, cryptographically signed database, verify it before each training run and abort if the check fails. If an organization cannot do adequate due diligence on web-crawled data, it advises not using it until trust infrastructure exists ^[1].

NIST is candid about limits. Its taxonomy warns that defenses against targeted poisoning are “notoriously challenging” and that differential privacy, a common defense, can lower model accuracy ^[14].

The table links each risk to the lifecycle stage where it bites, the controls the guidance names and the kind of tool that implements them. Lifecycle stages follow the joint guidance ^[1]; tool categories are our mapping.

1. 01

   Map AI data flows

   For each AI system, list its training, fine-tuning and retrieval data, what enters prompts, where outputs and logs go, and who owns each. Our AI risk register is a free starting point.

2. 02

   Classify and restrict

   Classify AI data by sensitivity and apply access controls, as the guidance’s fifth practice asks ^[1].

3. 03

   Vet sources and record provenance

   Document where every external dataset came from and write security and provenance expectations into supplier contracts ^[1][3].

4. 04

   Protect integrity

   Hash and sign datasets, keep curated data in an append-only signed store and stop a training run if verification fails ^[1].

5. 05

   Encrypt and store securely

   AES-256 at rest, current TLS in transit and validated cryptographic modules ^[1].

6. 06

   Control prompts, outputs and retrieval

   Redact sensitive data before it reaches a model, filter outputs and use permission-aware vector stores ^[10][9]. Our AI guardrails guide covers the runtime layer.

7. 07

   Test and monitor

   Red team for training-data leakage and membership inference ^[3], and monitor live data for drift and abrupt change ^[1].

8. 08

   Reassess and delete

   Run data security risk assessments on a schedule and delete data securely when it is no longer needed ^[1].

The joint guidance is a set of recommended best practices, and NIST describes its adversarial machine learning report as voluntary guidance ^[1][15]. Binding duties come from the AI and privacy laws that apply to your systems and data: personal data in training sets, prompts and logs may fall under privacy laws such as the GDPR. Our EU AI Act hub covers the Act’s requirements for high-risk AI, and the NIST AI RMF hub covers the framework the guidance’s lifecycle stages come from ^[1].

No single product covers every row of the control map. Most programs combine a few categories, and the ones specific to AI are compared in our rankings.

As an example of the first category, Lasso Security says its AI Usage Control detects personal and payment card data in prompts and file uploads in real time, masks it before the prompt reaches third-party models and checks AI responses for sensitive data ^[18]. Ask any vendor which of the data types in the first table it actually inspects: many cover prompts but not fine-tuning sets, vector stores or logs.

• Data governance for AI →Ownership, quality, lineage and permitted uses of AI data.
• AI model security →Extraction, backdoors and malicious model files.
• OWASP Top 10 for LLMs →The ten LLM risks, mapped to controls and tools.
• Shadow AI →Finding and controlling staff use of unapproved AI tools.
• AI security posture management →Tools that inventory AI assets and data exposure.