Governing AI agents in a healthcare setting is not primarily a technology problem — it is a patient safety and accountability problem that technology must solve. An agent that can read a patient record, compose a clinical note, or invoke a scheduling tool touches protected health information (PHI) with every action. The question governance must answer is: who authorized that action, what data did the agent see, and what did it produce? This post explains the control categories that answer those questions and how to apply them in practice.
For a broader foundation, see what AI agent governance means in practice and the AI agent compliance checklist for 2026.
Why Standard IT Controls Fall Short for Agents
Most enterprise security tooling was designed for human users. A person logs in once, operates through a browser session, and produces actions that are coarse-grained and deliberate. An AI agent behaves differently: it authenticates programmatically at machine speed, may invoke dozens of tools in a single workflow, and produces high-volume outputs with no human in the loop for each step.
Traditional controls — an SSO login, a firewall rule, a VPN policy — assume a person on the other end. They do not model agent identity as a first-class concept, do not intercept individual tool calls, and do not carry enough context to distinguish a legitimate clinical workflow from an agent manipulated through its inputs. Healthcare organizations need controls that operate at the agent layer: identity, authorization, content inspection, spend limits, and a complete audit trail that can survive a HIPAA audit or a breach investigation.
Agent Identity and Least-Privilege Access
Every agent in a healthcare environment needs a unique, verifiable identity before anything else. Without it, you cannot scope permissions, enforce policies, or attribute actions in your audit log. In practice, that means a credential distinct from any human user account, scoped to the agent's purpose, and rotatable or revocable without affecting other systems. Short-lived credentials are preferable wherever the workload permits — a long-lived API key that never rotates is a sustained liability if compromised. Provisioning agents through the same identity lifecycle as human staff — creation, modification, suspension, deletion — ensures they appear in access reviews and cannot become orphaned accounts.
HIPAA's minimum necessary standard maps directly onto least privilege at the tool and data level. An agent performing appointment scheduling has no business reading progress notes. An agent summarizing lab results should not be able to write to the medical record. In an agentic architecture, each connection between an agent and an external system — an EHR API, a lab information system, an MCP server exposing scheduling tools — is a discrete authorization boundary. Every connection should carry an explicit permission set reflecting the minimum the agent needs. Per-connection policies make breach containment predictable: if an agent misbehaves, the damage is bounded by the connections it holds, not by the breadth of whatever service account it runs under. For a deeper look at implementing this pattern, see how to implement least privilege for AI agents.
Content Guardrails: Keeping PHI Where It Belongs
Agents that process clinical inputs create two PHI risks that traditional perimeter controls do not catch.
The first is inadvertent leakage outward: an agent might include a patient name, date of birth, or diagnosis code in a prompt sent to a third-party model provider, or in a log entry written to a system outside your control boundary. The second is injection inward: malicious content embedded in a document, email, or tool response that attempts to redirect the agent's behavior — a technique known as indirect prompt injection.
Inline content inspection addresses both. Before a prompt leaves your environment, it can be scanned for PHI patterns and either redacted or blocked depending on your policy. Before a tool response is consumed by the agent, it can be checked for patterns that indicate an injection attempt. These checks are most effective when they run on every interaction, not as a sampling audit, because agents can process thousands of requests in a short window.
Content inspection in healthcare also supports the documentation requirements around de-identification. If you can demonstrate that PHI was systematically redacted before reaching a third-party model, you have a stronger argument that the downstream data falls outside HIPAA's scope. The post on PII detection and redaction in AI pipelines covers the technical patterns in detail.
Audit Trails That Satisfy Investigators
A HIPAA audit or a breach investigation requires you to reconstruct what happened: which agent acted, on whose behalf, at what time, against which record, with what outcome. A generic application log rarely answers all of these questions.
An agent-aware audit trail captures the action at the right level of granularity — not just "API call to endpoint X" but "agent Y, authorized by user Z, invoked tool W with these parameters, and produced this output." The trail needs to be tamper-evident. In regulated environments, auditors are not satisfied with logs that an administrator could edit; the trail should be structured so that any modification is detectable, which is what append-only, cryptographically signed logs provide.
Retention is also a compliance consideration. HIPAA requires records to be available for six years. Your audit trail retention policy should match or exceed that, and the policy itself should be governed — not left as an implicit default.
Risk Classification for the EU AI Act
If your healthcare organization operates in or sells into the European Union, AI agents interacting with health data almost certainly fall into the EU AI Act's high-risk category. High-risk classification carries obligations: conformity assessment, technical documentation, human oversight measures, transparency to affected persons, and registration in the EU database.
Managing this across multiple agents requires a classification workflow — a way to assess each agent against the risk criteria, record the outcome, attach supporting evidence, and track whether your controls meet the applicable requirements. Risk classifications are not static; they should be reassessed when an agent's purpose changes, when the underlying model changes, or when the regulatory guidance evolves.
Ongoing Controls: Access Reviews and Spend Limits
Human user access reviews are standard practice in healthcare IT, and the same discipline applies to agents. An agent commissioned for a pilot project should not hold production credentials six months later. Periodic reviews follow the same structure as reviews for human accounts: enumerate what access each agent holds, have an owner certify it is still necessary and correctly scoped, and revoke or reduce where it is not. Inactivity-based review is a useful complement — an agent idle for 90 days is a candidate for suspension, reducing the standing attack surface without a full review cycle.
Spend controls serve a parallel purpose. An agent caught in a loop, manipulated by an injection attack, or misconfigured can generate thousands of API calls in minutes. A per-agent budget ceiling that halts execution when a threshold is crossed is a containment mechanism, not just a cost control. It limits how much damage an out-of-control agent can do before a human intervenes, and gives you the granularity to distinguish a high-volume production agent running normally from a low-volume agent that has gone abnormal.
Compliance Reporting and Evidence Collection
Demonstrating compliance requires evidence, not just assertions. For AI agents, evidence means: records showing access was granted through an approved process, audit logs showing access was used within authorized bounds, guardrail configuration records showing PHI controls were active, and risk assessments showing agents were classified and managed appropriately.
Mapping your controls to the relevant frameworks — HIPAA Security Rule safeguards, the NIST AI Risk Management Framework, ISO/IEC 42001 — in advance means you know exactly what evidence each control generates and where to find it when an auditor asks. A compliance center that structures control mapping, gap tracking, evidence collection, and report generation against standard frameworks turns evidence gathering into an ongoing discipline rather than an audit-eve scramble. For background on the NIST framework specifically, see applying the NIST AI RMF to AI agents.
Common questions
Does governing AI agents require changes to our existing HIPAA policies?
Your existing HIPAA policies likely cover electronic PHI accessed by workforce members and information systems. AI agents acting autonomously are a new category that most existing policies do not address explicitly. You will typically need to extend your risk analysis to cover agent identity and access, update your access control and audit policies to include agent accounts, and add agent-specific provisions to your incident response procedures. The underlying obligations — minimum necessary access, audit controls, contingency planning — are the same; the agent context requires new implementation guidance.
How do we handle PHI that enters an agent prompt if the model is hosted by a third party?
The key question is whether you have a signed Business Associate Agreement (BAA) with the model provider. If you do, PHI may flow to that provider under the BAA's terms. If you do not, PHI should not reach the provider. Content inspection and redaction before the prompt leaves your environment is the technical control that enforces this, but the legal relationship with your model provider must be established first. Redaction alone does not substitute for a BAA where one is required.
What should we log when an agent accesses a patient record?
At minimum: the agent's identity, the user or workflow that triggered the action, the timestamp, the specific record or data element accessed, the tool or API endpoint invoked, and the output produced. This mirrors the access log requirements that already apply to human access to electronic PHI, extended to cover the agent as the acting principal. Logs should be retained for at least six years and protected against modification.
For more on building a governance approach that covers identity, guardrails, audit, and compliance reporting for AI agents in regulated industries, see audit trails that hold up under regulatory scrutiny or take the readiness assessment.