Yes. Large language models memorise part of their training data and can be induced to reproduce it verbatim — names, email addresses, code, API keys, whole paragraphs. Researchers have extracted gigabytes of training data from open models and from an aligned production chatbot. For teams building on LLMs, the practical consequence is narrower but sharper: anything sensitive you put into a fine-tuning dataset may come back out for any user of that model, and you cannot reliably delete it afterwards. The defences are data hygiene before training and redaction on the output path.
What Memorisation Means
A model memorises a sequence when, given a prefix, it continues with the exact training text rather than a plausible paraphrase. Memorisation is not a bug in one model; it is a side effect of how models fit data. Sequences that appear many times in the training set, unusual strings that the model cannot compress into general patterns, and data seen late in fine-tuning are all more likely to be memorised.
Not all memorised data is a problem. A model reciting a famous poem is harmless. A model reciting a customer's address from a support transcript, or an access key from an internal repository, is a data leak.
How Extraction Attacks Work
In Scalable Extraction of Training Data from (Production) Language Models, Nasr et al. study extractable memorisation: training data an adversary can pull out by querying a model without knowing the training set in advance. Their findings:
- Existing techniques were enough to extract gigabytes of training data from open models such as Pythia and GPT-Neo and from semi-open models such as LLaMA and Falcon.
- For the aligned ChatGPT, they developed a divergence attack that pushes the model out of its chatbot behaviour. The authors report it then emitted training data at a rate 150 times higher than when behaving normally.
- Their conclusion: current alignment techniques do not eliminate memorisation.
The general attack pattern is simple. Generate many outputs, using prompts likely to elicit memorised text or to make the model diverge, then check outputs for verbatim matches against known data or for content that looks like real records — well-formed emails, keys, phone numbers.
Related attacks target the model rather than its data. Membership inference asks whether a specific record was in the training set; model extraction tries to clone behaviour through queries. Those are covered in model extraction and inference attacks on agents.
Where the Risk Actually Sits for Your Team
Most organisations do not pre-train foundation models. The exposure is in three places:
| Source | What leaks | Who can extract it |
|---|---|---|
| Your fine-tuned model | PII, secrets and confidential text in your fine-tuning data | Anyone who can query the fine-tuned model, including other tenants if it is shared |
| Foundation model | Public web data that may include personal data or leaked credentials | Anyone using the model; outside your control |
| Context, not training | System prompts, retrieved documents, tool results | Users who can prompt the model; this is prompt leaking, not memorisation |
The third row is often confused with memorisation. If a model reveals your system prompt, that text was in its context window, not its weights. The fix is different; see prompt leaking and system prompt extraction.
Fine-tunes deserve the most attention because the data is yours, it is often small and repeated over several epochs, and it is frequently built from raw production records: support tickets, chat logs, code, CRM notes. Those are exactly where customer PII and pasted credentials live.
Prevent It Upstream: Fine-Tune Data Hygiene
You cannot reliably remove one memorised string from trained weights. Machine unlearning remains an open research area, and retraining without the data is the dependable fix. So prevention happens before training:
- Scan for secrets. Run a secret scanner over every fine-tuning record. Remove API keys, tokens, passwords and connection strings, and rotate any that were found — they were already exposed in the source.
- Redact PII. Replace names, emails, phone numbers, account and card numbers with typed placeholders. The model learns the task pattern without the identities. Methods are covered in PII detection and redaction in AI pipelines.
- Deduplicate. Repeated sequences are memorised more readily. Remove exact and near duplicates, especially boilerplate that contains identifiers such as signatures.
- Minimise. Include only fields the task needs. A classifier for ticket intent does not need the customer's address.
- Separate per tenant. Do not fine-tune one shared model on several customers' data if other customers can query it. Memorised text crosses tenant boundaries through the model's weights.
- Record lineage. Keep the dataset version, sources and redaction steps with each model version, so an erasure request or incident can be traced to the models it affects.
Catch It Downstream: Output Controls
Hygiene reduces the risk; it does not reach zero, and you do not control the foundation model's training data. Treat every model output as untrusted content that can contain sensitive data:
- Output-side redaction. Scan responses for PII and secret patterns before they reach a user, a log or a downstream tool. Redact rather than block where possible, so the task can still complete.
- Canary strings. Insert unique, meaningless marker strings into fine-tuning data. If one ever appears in output, you have direct evidence of memorised training data being emitted.
- Rate and pattern limits. Extraction needs volume. Unusually long generations, repeated-token prompts and high query rates from one principal are signals worth alerting on.
- Log responses with redaction. Keep a record of what the model returned to whom, with sensitive values masked, so a suspected leak can be investigated.
Output handling matters beyond leakage too: model output that flows into shells, browsers or databases needs the controls in insecure output handling.
Training Data Leakage Checklist
How Praesidia Approaches Output Leakage
Praesidia evaluates guardrails in both directions of a governed connection, so responses are inspected rather than only inbound prompts. Guardrails support block, redact, and warn dispositions, which lets PII and secret patterns be removed from a response without failing the whole task. Because enforcement sits in the proxy between the agent and its counterparty, an agent cannot opt out of the evaluation of its own output.
Praesidia does not inspect or clean your fine-tuning datasets or model weights; data hygiene before training stays in your ML pipeline. For the full set of model-layer threats, see the AI agent security guide.