LLM Gateway Comparison 2026: Security and Governance

An LLM gateway is a proxy layer between your applications and the model providers they call — Anthropic, OpenAI, Google, or a self-hosted model — adding routing, fallback, caching, spend tracking, and policy enforcement in a single place instead of scattering API keys and retry logic across every service. By 2026 the category has settled into a handful of well-documented options, and the right pick depends less on feature checklists than on who is going to operate the thing and what "governed" needs to mean at your organization.

This is a different product category from an MCP gateway, which sits between agents and the tools they call rather than between applications and model providers — see the MCP gateway comparison for that adjacent decision. It is also a narrower question than the general control plane versus API gateway distinction; this post is a named-vendor comparison within the LLM-gateway sub-category specifically.

What an LLM gateway actually does

Strip away branding and every gateway in this category is solving some combination of the same problems:

• Provider abstraction — one API shape in front of multiple model providers, so switching or multi-sourcing models does not mean rewriting call sites.
• Routing and fallback — sending traffic to a specific model or provider, and failing over automatically when one is unavailable or rate-limited.
• Caching — avoiding repeat spend on identical or near-identical requests.
• Spend tracking and budgets — attributing token cost to a team, key, or application, and capping it.
• Access control — who (which key, team, or service) can call which model, at what rate.
• Observability export — logs and traces of what was called, with what latency and cost.

Where gateways genuinely diverge is in how much of this is delivered as a managed product versus something you assemble and operate, and how deep the governance layer goes beyond basic key management. For background on why per-provider key sprawl becomes a security problem on its own, see securing LLM provider API keys and BYOK and managing LLM configurations.

The options

Portkey: managed, governance-first

Portkey is a managed cloud gateway aimed at teams that want RBAC, SSO, budgets, rate limits, VPC deployment options, and compliance-oriented controls available out of the box rather than assembled from primitives. Its public positioning centers on being the layer an enterprise points multiple teams and applications through, with policy and cost controls configured centrally rather than per application. The trade-off is the standard managed-service one: you get governance faster, in exchange for depending on a vendor's roadmap and data-handling posture for a chokepoint that sees every prompt.

LiteLLM: open-source, self-hosted

LiteLLM is a widely used open-source gateway that unifies the API surface across model providers and adds keys, quotas, and spend tracking, all run on infrastructure you control. It is the natural choice when the requirement is full control of the data path — nothing proxies through a third party — and there is no per-seat licensing cost. The trade-off is operational: you run and monitor the proxy, patch it, and build out the governance layer (fine-grained RBAC, audit-grade logging, compliance evidence) yourself if your public documentation review shows the built-in controls stop short of what you need. Whether that trade makes sense scales with team count, provider count, traffic volume, and how mature your platform-engineering function already is — a single team calling a single provider often does not need a gateway at all; direct SDK integration is simpler. A closer look at that specific decision point is in LiteLLM enterprise alternatives.

Kong AI Gateway: consolidate onto an existing API platform

Kong extends its established API-gateway product with AI-specific capabilities — model routing, credential handling, and AI traffic visibility inside the plugin architecture Kong customers already operate. The case for it is consolidation: if Kong already fronts your APIs, extending the same operational model, observability stack, and plugin ecosystem to LLM traffic is cheaper than standing up a second system. It is a weaker fit for an AI-only greenfield deployment with no existing Kong footprint, where the plugin composition model is overhead rather than reuse.

Cloudflare AI Gateway: edge-native, near-zero ops

Cloudflare's AI Gateway is a fully managed offering built into its edge network, aimed at near-zero operational overhead — you point traffic at it and get caching, rate limiting, and logging without running infrastructure. It is a strong fit inside an existing Cloudflare footprint, where the AI Gateway is one more product on a platform a team already trusts and monitors; it is a weaker starting point if Cloudflare is not already part of the stack, since you would be adopting a new platform vendor specifically for this.

Other names worth knowing

Two more come up repeatedly in gateway shortlists. Helicone, per third-party comparisons, is frequently cited as leading on cost-visibility specifically — granular request-level spend breakdowns are its center of gravity rather than broad governance. TrueFoundry positions on data sovereignty and compliance certifications (SOC 2, HIPAA, GDPR are commonly cited) and on governing both model traffic and MCP tool traffic from one control surface, which narrows the gap between the LLM-gateway and MCP-gateway categories for teams that want one vendor across both.

Comparing the deployment models

Evaluating for security and governance, not just routing

Routing and caching are table stakes; the questions that actually separate a governed deployment from an ungoverned one are further down the checklist:

1. Identity granularity. Does access control stop at a shared API key, or can you attribute and restrict calls per application, team, or individual agent identity?
2. Audit quality. Is there a durable, queryable record of what was called, by what identity, with what result — one you could hand to an auditor — or only ephemeral request logs?
3. Content-aware policy. Can the gateway apply guardrails to prompts and completions in-band (PII, injection patterns, disallowed content), or does it pass everything through opaque?
4. Budget enforcement, not just visibility. Does exceeding a budget actually stop further spend, or only generate a report after the fact? See budgets versus rate limits for why these are different controls.
5. Data residency and retention. Where do prompts and completions get logged, for how long, and under whose jurisdiction — a material question for regulated teams, covered in general terms in cost control for LLM applications and model routing cost-quality trade-offs.

A gateway that answers all five well is doing governance, not just proxying. Several vendors above are strong on routing and weaker on audit depth and content policy — that gap is worth probing directly against their current documentation rather than assuming parity across the category.

How to shortlist

Start from what you already operate, not from a feature list. If you run Kong for APIs, evaluate Kong first and treat the others as the comparison baseline. If you run nothing comparable and want governance without building it, Portkey is the natural starting point. If data-path control and zero licensing cost outweigh operational burden, start with LiteLLM and re-evaluate once the operational cost becomes visible — that inflection point is exactly what the LiteLLM alternatives post above works through. If you are already deep in Cloudflare's edge, its AI Gateway is close to free to adopt.

Praesidia is an AI agent security and governance control plane — agent identity and access, guardrails, audit evidence, and cost controls in one place — and sits adjacent to this category rather than inside it: it governs the agents and tools calling models, where an LLM gateway governs the model calls themselves. Teams often run both; for the broader case on why a governance layer typically sits alongside infrastructure gateways rather than replacing them, see the AI control plane guide.

Common questions

Is an LLM gateway the same thing as an MCP gateway? No. An LLM gateway sits between applications and model providers, governing model-call traffic — routing, spend, provider keys. An MCP gateway sits between agents and the tools they call, governing tool-invocation traffic. Some vendors are starting to cover both from one product, but they are answering different questions; see the MCP gateway comparison for the tool-calling side.

Do I need a gateway if I only call one model provider? Usually not yet. A gateway earns its complexity when you are managing multiple providers, multiple teams, or need centralized spend and access control that a single SDK integration cannot provide. Single-team, single-provider setups are usually simpler served with direct integration.

Which gateway is most secure? There is no single answer — "most secure" depends on which control matters to you (identity granularity, audit durability, content inspection, data residency) and each vendor above documents a different strength. Evaluate against the five governance questions in this post using each vendor's current public documentation, not a generic ranking.

Can I run more than one gateway at once? Yes, and some teams do — for example Kong for existing API traffic and a dedicated model gateway for LLM-specific spend control. It adds operational surface, so it is usually a transitional state on the way to consolidating onto one, not a permanent architecture.

How often does this comparison need to be re-checked? Frequently. Gateway feature sets, pricing, and compliance certifications change often in this category — verify current capabilities directly against each vendor's documentation before finalizing a shortlist, rather than relying on any point-in-time comparison, including this one.