To secure a self-hosted LLM, treat the inference server as an unauthenticated internal service: bind it to loopback or a private network, put an authenticating proxy or gateway in front of it, block its model-management endpoints from clients, pin and verify the model files it loads, and route requests through the same guardrails and audit trail you use for hosted models. Ollama and vLLM are both built to be easy to start. Their defaults assume a trusted network, and production rarely is one.
Why self-hosted models get exposed
Teams self-host models for good reasons: data stays in their environment, costs are predictable, and they can run open-weight models a hosted provider does not offer. The security problems come from how the servers are typically started:
- A developer changes the bind address so a colleague or a container can reach the server, and it ends up listening on every interface.
- An agent framework starts a model server for you, with settings you never reviewed.
- A GPU host gets a public IP for convenience, and the inference port comes with it.
- The server exposes more than inference. Endpoints that pull, create, copy or delete models sit on the same port as chat completions.
The NemoClaw DNS-rebinding and Ollama poisoning analysis shows how this goes wrong in practice. An Ollama instance bound to all interfaces, combined with a model-creation endpoint, let an attacker overwrite a model's chat template with hidden instructions. Nothing about the model weights was compromised. The serving layer was.
Ollama: what to know
Ollama listens on a local port and, by default, binds to the loopback interface. Its bind address and allowed browser origins are controlled through environment variables. It has no built-in authentication or authorisation layer: anyone who can reach the port can call every endpoint, including the ones that manage models.
That leads to a short set of rules:
- Keep it on loopback unless another host truly needs it. If it must be reachable, bind it to a private interface, never to a public one.
- Put an authenticating reverse proxy or gateway in front of any instance that serves more than one local user.
- Allow only inference paths through the proxy. Block model pull, create, copy, push and delete endpoints from client traffic. Manage models from an operator path instead.
- Restrict browser origins. A permissive origin setting lets web pages in a user's browser reach the server, which is the starting point for DNS-rebinding attacks.
- Check what started it. Some tools launch Ollama automatically. Find every running instance on developer machines and servers, not just the ones you installed.
vLLM: what to know
vLLM serves an OpenAI-compatible HTTP API and is common for GPU-backed production serving. Its security documentation is direct about the limits of its defaults (vLLM security guide):
- The
--api-keyflag, or theVLLM_API_KEYenvironment variable, enforces a bearer token only for endpoints under specific path prefixes, such as/v1. Other sensitive endpoints on the same HTTP server are not covered. The documentation says not to rely on--api-keyalone. - Communication between nodes in a multi-node deployment is insecure by default. The guidance is to put those nodes on a dedicated, isolated network.
Practical consequences:
- Set an API key anyway, but treat it as one layer, not the boundary.
- Put a proxy or gateway in front that allows only the inference endpoints your clients need, and authenticates every caller.
- Isolate the cluster network. Inter-node and distributed-runtime ports must not be reachable from application networks, let alone the internet.
- Review flags that execute code. Options that allow a model repository to run its own code at load time turn a model download into a code-execution path. Enable them only for artefacts you have reviewed.
Hardening checklist
| Area | Control | Applies to |
|---|---|---|
| Network | Bind to loopback or a private interface; no public IP on inference hosts | Both |
| Network | Firewall the inference port to the gateway or proxy only | Both |
| Network | Isolate multi-node and distributed-runtime traffic on its own network | vLLM |
| Auth | Authenticating proxy or gateway in front of every shared instance | Both |
| Auth | Built-in API key enabled, but not relied on alone | vLLM |
| Endpoints | Allow-list inference paths; block model management from clients | Both |
| Browser | Restrict allowed origins; never use a wildcard | Ollama |
| Models | Pin model revisions, record hashes, store artefacts in a registry you control | Both |
| Models | Prefer formats that do not execute code on load; avoid remote-code options for unreviewed models | Both |
| Runtime | Run as a non-root user in a container with minimal filesystem write access | Both |
| Secrets | No provider keys or cloud credentials on the inference host | Both |
| Tenancy | Separate instances or strict request scoping for different tenants | Both |
| Logging | Log who called which model, from where, with which outcome | Both |
| Updates | Track releases and security advisories; patch the serving stack like any other server | Both |
Model files are part of the attack surface
Hardening the server is half the job. The other half is what it loads.
Provenance. Download models from known publishers, pin the exact revision and record a hash. A tag that moves can quietly swap the model you are serving. The AI agent supply chain security post covers this in depth.
Templates and system prompts. For many local servers, the chat template and default system prompt are configuration bundled with the model. If an attacker can change them, they can steer every response without touching the weights. Store them under version control and treat changes as reviewed deployments.
Write access. Only an operator role should be able to add or change models. Client applications should have read-only access to inference, nothing more.
Tenancy and secrets
If several teams or customers share one inference cluster, isolation matters as much as it does in your database. Requests from one tenant must not be able to read another tenant's cached context or logs. The same principle as tenant isolation with row-level security applies: enforce separation in the infrastructure, not in application convention.
Keep secrets off inference hosts. A model server has no need for cloud credentials or provider API keys. If an agent needs them, they belong in a secrets manager with scoped access, as described in secrets management for AI agents.
Govern local models like hosted ones
The biggest risk with self-hosting is a quiet double standard. Hosted models sit behind a gateway with keys, guardrails and logs. Local models get called directly because "it's internal". That gap is exactly where prompt injection, data leakage and unexplained agent behaviour go unnoticed.
The fix is architectural. Model calls, hosted or local, go through one gateway that authenticates the caller, applies the same input and output guardrails, and logs its decisions. The AI control plane vs API gateway post explains why that layer needs to understand agents, not just HTTP.
How Praesidia helps
- Discover. Inventory the agents in your environment and see which models they call, so direct calls to local inference servers become visible.
- Secure. The Praesidia BYOK registry supports Ollama for self-hosted open-weight models and custom OpenAI-compatible endpoints, which is how vLLM is usually reached. In the hosted service, both the agent runtime and the gateway refuse private-network addresses, so the Ollama endpoint needs a public address. Serve it over HTTPS and allow-list who can reach it: an agent's model configuration can carry a key for an authenticating proxy in front of Ollama, but the gateway's Ollama route sends no credential and strips the client's, so on that path the network allow-list is the only gate. vLLM is reached as a custom OpenAI-compatible endpoint (HTTPS, on a DNS name) as an agent's model; a custom route through the gateway requires the Advanced tier or Enterprise and deployment enablement. Once registered, the same guardrails that block, redact or warn on hosted model traffic apply to that model's calls.
- Prove. Praesidia records its decisions in an append-only, hash-chained audit trail, so a local model call leaves the same evidence as a hosted one.
Praesidia does not replace the network, container and model-provenance controls in the checklist above. It makes sure the self-hosted model is governed once traffic reaches it. For the full gateway architecture, see the AI gateway guide.