The academic literature on attacks against machine learning models is large, and much of it does not transfer cleanly to deployed agent systems. Model inversion against an image classifier and prompt-based extraction from an agent behind an API are different problems with different economics.

It is worth separating what actually threatens an agent deployment from what threatens a model you trained yourself, because the mitigations differ and effort spent on the wrong one is wasted.

Behavioral Extraction (Distillation)

What it is. An attacker queries your system extensively, collects input-output pairs, and fine-tunes a cheaper model to reproduce its behavior. They are not stealing weights — they are stealing behavior, which is often sufficient.

How real it is. Very real and inexpensive. Tens of thousands of well-chosen queries can produce a model that approximates a specialized system's behavior on the distribution that matters. If your product's value is in a prompt, a workflow, and a set of examples, that value is extractable through the API.

What mitigates it. Not much, cleanly. Rate limits and per-principal volume ceilings raise the cost. Detecting systematic query patterns — broad coverage of the input space, uniform query structure, high volume with low repetition — catches unsophisticated collection. Output watermarking is proposed frequently and is fragile in practice against paraphrasing.

The honest framing is commercial rather than technical: assume behavior is extractable and build the moat elsewhere — proprietary data, integrations, evaluation apparatus, iteration speed. Treating extraction as a security problem to be solved usually produces friction for legitimate users and little else. Where the concern is contractual rather than technical, terms of service plus usage monitoring is the actual control.

Membership Inference

What it is. Determining whether a particular record was in the training data, typically by observing that the model behaves differently — more confidently, with lower loss — on training members.

How real it is for agents. It depends entirely on whether you fine-tuned. If you use a foundation model with RAG, your sensitive data was never in training and there is nothing to infer membership of. If you fine-tuned on customer records, medical notes, or transaction data, membership inference is a genuine privacy exposure — and in some regulatory readings, the model itself becomes personal data because membership can be inferred from it.

What mitigates it. Decisions made at training time, not at inference:

  • Do not fine-tune on sensitive records if RAG will do. This is the dominant recommendation and it resolves most of the risk. Retrieval keeps the data in a store you can scope, audit, and delete from — which also makes erasure requests tractable. See GDPR for AI systems and erasure rights.
  • Deduplicate and minimize. Repeated records are memorized more strongly. Remove duplicates and strip fields the task does not need.
  • Differential privacy in training where the data genuinely requires it, accepting the utility cost.
  • Limit output detail. Log probabilities and confidence scores exposed through an API make inference substantially easier. Do not expose them on untrusted surfaces.

Training Data Extraction and Memorization

What it is. Getting a model to emit verbatim content from its training data — the stronger version of membership inference. Well demonstrated against large models: with enough querying, verbatim sequences including personal data and credentials can be recovered.

How real it is for agents. Same conditional as above. Foundation-model memorization is the provider's problem and largely outside your control. Your fine-tunes are your problem, and small fine-tuning datasets memorize aggressively — a record appearing a handful of times in a small dataset is quite likely to be reproducible.

What mitigates it. Deduplication, minimization, and PII scrubbing before training. Output-side detection for known sensitive patterns catches some leakage after the fact. The reliable answer is not putting the data in the weights.

Context Reconstruction: The One That Matters Most for Agents

What it is. An attacker uses repeated querying to extract the contents of the agent's context rather than its training data: retrieved documents, tool results, conversation history, system prompt, and — in shared or multi-tenant deployments — potentially other users' data if isolation is imperfect.

How real it is. This is the highest-probability data exposure in a typical agent deployment, and it requires no machine learning sophistication at all. Ask the agent to summarize what it retrieved. Ask for the source document. Ask a series of narrow questions whose answers collectively reconstruct a record. The agent is designed to be helpful about the content in its context, so it cooperates.

What mitigates it. Access control, not content filtering:

  • Retrieval scoped to the principal. The agent retrieves only what the current principal is entitled to see, enforced at the datastore. Then reconstruction yields only data the attacker was already entitled to. This is the whole ballgame. See RAG pipeline security and tenant isolation with row-level security.
  • Volume ceilings. Caps on retrieved chunks per query and per task, and on tool result sizes. Reconstruction requires many queries; caps make it slow and visible.
  • Per-principal rate limits. The reconstruction attack is high-volume by nature.
  • Minimized context. Retrieve the passage needed, not the whole document. Return the fields needed, not the whole record. Most agent implementations are far more generous here than the task requires, and tightening it reduces both cost and exposure.
  • PII redaction on the way in. If personal data is redacted before entering the context, it cannot be reconstructed out of it. See how to redact PII from agent prompts.
  • Output guardrails as a secondary layer. Detect sensitive patterns leaving. Useful, and not a substitute for scoping.

Prompt and Configuration Extraction

Covered in depth separately, but it belongs in this taxonomy: the system prompt, tool inventory, and policy thresholds are extractable, and should be structured so extraction is cheap. See system prompt extraction.

Detection

The extraction attacks share a signature: high query volume with systematic structure. Useful signals:

  • Query volume per principal well above baseline, especially with low semantic repetition (broad coverage rather than repeated similar asks).
  • Queries that are structurally uniform with varying parameters — a template being iterated.
  • Retrieval volume per principal rising without a corresponding change in task type.
  • Requests explicitly asking for source content, document text, or "everything you know about."
  • Sustained sessions with unusually many turns for the workload.

Per-principal aggregation is what makes these visible; per-request evaluation shows nothing, because each individual query is ordinary.

Common questions

Should I avoid fine-tuning entirely?

No — fine-tuning is the right tool for teaching format, tone, and task structure. The recommendation is narrower: avoid fine-tuning on sensitive records when retrieval would serve. Fine-tune on synthetic or de-identified examples that teach the behavior, and keep the sensitive content in a retrievable store where access control, audit, and deletion actually work. That split gets the benefit without putting personal data in weights you cannot edit.

Do output filters stop context reconstruction?

They reduce it and cannot close it. Filters match patterns; reconstruction can proceed through paraphrase, partial disclosure across many queries, and inference from answers that individually contain no sensitive pattern. A filter that blocks account numbers does not stop an attacker learning a balance by asking whether it exceeds a series of thresholds. Access scoping is the control; filtering is defense in depth.

Is model extraction worth worrying about for an internal-only agent?

Behavioral extraction, no — the population that can query it is your own staff. Context reconstruction, yes, and more than externally, because internal agents typically have broader data access and internal users are more numerous than you might assume. The insider-amplification framing applies directly. See insider threat in the age of AI agents.

How does this affect multi-tenant deployments specifically?

It raises the stakes on isolation. In a shared deployment, imperfect scoping means one tenant's queries can reach another's data through retrieval or shared caches, and reconstruction is the technique that finds it. Enforce the tenant predicate at the datastore rather than in application code, separate caches per tenant, and test cross-tenant retrieval explicitly in CI. A test that asserts a tenant-A query returns zero tenant-B rows is worth more than any amount of output filtering.

How Praesidia approaches extraction risk

Praesidia scopes data access at the connection level and enforces tenant boundaries below the application rather than in prompt instructions, so retrieval and tool results are bounded by the principal being served rather than by the agent's own reach. Guardrails evaluate content in both directions with block, redact, or warn dispositions, which keeps sensitive patterns out of the context on the way in and out of responses on the way back.

Per-agent and per-connection rate limits and volume constraints bound high-volume reconstruction attempts, and every call is metered and recorded with per-principal attribution — which is what makes per-principal volume anomalies detectable. See PII detection and redaction in AI pipelines and the Praesidia documentation.