LiteLLM is the default starting point for teams that want an open-source, self-hosted LLM gateway: one API shape across providers, keys, quotas, and spend tracking, running on infrastructure you already control. It earns that default status honestly — no per-seat licensing, full control of the data path, and a large community. The question this post answers is not whether LiteLLM is good, but when the calculus that made it the right choice stops holding, and what teams evaluate next.
What LiteLLM is built for
LiteLLM's public documentation centers on provider abstraction: a unified API in front of Anthropic, OpenAI, Google, and dozens of other model providers, with keys, per-key quotas, and spend tracking layered on top. Because it is open source and self-hosted, nothing about a request passes through a third party unless you route it there yourself — a meaningful property for teams with strict data-path requirements. There is no seat-based pricing; cost is infrastructure and engineering time, not a subscription. For background on why per-provider key sprawl becomes its own security problem regardless of which gateway you choose, see securing LLM provider API keys.
That combination — free, self-hosted, full control — is exactly right for a specific profile: a platform team with the operational capacity to run and patch a proxy, serving a moderate number of internal consumers, where the built-in RBAC and audit surface are sufficient for the compliance bar the organization has to clear.
Most teams that adopt LiteLLM do so first for the provider-abstraction benefit alone — being able to swap or multi-source models without rewriting application code — and only later start leaning on its access-control and spend-tracking features as usage spreads across more internal teams. That order matters: the features that made LiteLLM the right first choice (no licensing cost, full data-path control) are largely independent of the features that determine whether it is still the right choice a year later (governance depth, operational reliability under load). Conflating the two is the most common reason teams either switch too early or stay too long.
Where the calculus changes
The value LiteLLM delivers scales with three variables, and each one, taken far enough, starts to erode the case for continuing to operate it yourself:
Number of teams and providers. A single team calling a single provider does not need a gateway in front of it at all — direct SDK integration is simpler and has less to operate. As teams and providers multiply, the coordination value of a shared gateway grows, but so does the surface you are responsible for keeping consistent across them.
Traffic volume. Higher volume raises the cost of downtime or misconfiguration on the gateway itself. A proxy that is fine to babysit at modest scale becomes a production-critical dependency that needs on-call coverage, capacity planning, and incident response once it is carrying meaningful load — see rate limiting and abuse prevention for AI APIs for the operational pattern this tends to require.
Operational maturity available to run it. Self-hosting is a genuine trade, not a free lunch: you own patching, upgrades, and the depth of governance features (fine-grained RBAC, audit-grade retention, content-aware policy) beyond what ships out of the box. If your public review of LiteLLM's current documentation shows a gap between what it provides and what your compliance function requires, closing that gap is your engineering time, not a vendor's roadmap.
None of this is a defect in LiteLLM — it is the standard build-versus-operate trade that applies to any self-hosted open-source infrastructure, discussed in general terms in self-hosted versus managed AI governance, build versus buy for AI agent governance, and the broader AI control plane guide.
What teams evaluate instead
Portkey: managed governance out of the box
Portkey is a managed cloud gateway built around RBAC, SSO, budgets, rate limits, and compliance-oriented controls as first-class product features rather than assembled configuration. Teams that reach the operational-maturity ceiling described above frequently evaluate Portkey next, trading the "we run everything" model for a vendor that already ships the governance layer. The trade-off is the standard managed-service one: faster governance, more dependency on a vendor's data-handling posture and roadmap. See the fuller comparison in LLM gateway comparison 2026.
Kong AI Gateway: consolidate instead of adding a system
If the organization already runs Kong for API management, extending it with AI-specific capabilities — model routing, credential handling, AI traffic visibility — inside the plugin ecosystem it already operates is often cheaper than standing up and hardening a second, LiteLLM-specific system. This alternative only makes sense when the Kong footprint already exists; adopting Kong purely for this reason is a weaker case than the reverse.
Cloudflare AI Gateway: managed and edge-native
For teams already running on Cloudflare's edge, its AI Gateway offers a fully managed path with near-zero operational overhead — caching, rate limiting, and logging without a proxy to patch. It answers the same operational-burden question Portkey does, from a different starting platform.
TrueFoundry: compliance-certified, dual-purpose
TrueFoundry positions on data sovereignty and named compliance certifications (SOC 2, HIPAA, GDPR are commonly cited in its public materials), and on governing both LLM traffic and MCP tool-calling traffic from one control surface. It is worth evaluating specifically when the requirement set spans both categories rather than LLM routing alone — see what to look for in an MCP gateway for the tool-calling side of that requirement set.
How the alternatives compare
| Alternative | Operating model | Reach for it when |
|---|---|---|
| Portkey | Managed cloud | Governance (RBAC, SSO, budgets) matters more than owning infrastructure |
| Kong AI Gateway | Extension of an existing platform | Kong already fronts your APIs |
| Cloudflare AI Gateway | Managed, edge-native | Cloudflare already sits in front of your traffic |
| TrueFoundry | Managed, compliance-certified | You need one surface for both LLM and MCP tool-calling traffic, or named compliance certifications matter |
| Stay on LiteLLM, add process | Self-hosted | Team/provider count and traffic are still modest and on-call capacity exists |
None of these is a strict upgrade over LiteLLM — each trades something (cost, control, or a new vendor dependency) for something else (governance depth, less operational burden, platform reuse). The right column is a starting hypothesis to validate with a proof-of-concept, not a verdict.
A decision checklist
Use this before defaulting to "just add more LiteLLM infrastructure":
- Team and provider count. Fewer than three teams and one or two providers — direct integration or continued LiteLLM is usually still right.
- On-call capacity. No dedicated platform on-call for the gateway — that is a signal to evaluate a managed option before traffic grows further.
- Compliance bar. If auditors or regulators require durable, tamper-resistant logs of every model call by identity, verify LiteLLM's current documentation covers that depth before assuming it does.
- Existing platform footprint. Kong or Cloudflare already in production — evaluate their AI-gateway extensions before adding a third system.
- Budget enforcement, not just tracking. Confirm whether spend limits actually stop further calls or only report after the fact; this differs by product and by configuration, covered generally in budgets versus rate limits.
Common questions
Is LiteLLM insecure or under-featured? No — it is a capable, widely used open-source gateway, and its public documentation covers a broad governance surface. The question this post addresses is operational: at what scale does self-hosting and extending it yourself cost more, in engineering time and risk, than a managed or platform-extension alternative. That threshold varies by team, not by a defect in the software.
What is the single clearest signal it is time to look elsewhere? Recurring, unplanned engineering time spent patching, upgrading, or extending the gateway itself — as opposed to time spent on the applications it serves. When gateway maintenance becomes a standing line item rather than an occasional task, the managed and platform-extension alternatives above are worth pricing against that ongoing cost.
Do enterprises usually replace LiteLLM entirely, or run it alongside something else? Both patterns exist. Some teams migrate off LiteLLM once a managed alternative covers their governance requirements; others keep LiteLLM for specific self-hosted workloads (data residency, cost) while adopting a managed gateway for the rest. There is no single correct pattern — it depends on which of the three scaling variables (teams, traffic, operational maturity) is the binding constraint.
How does this decision interact with MCP gateway choices? They are separate decisions about separate traffic — LLM-provider calls versus tool-calling traffic to MCP servers — though a small number of vendors, TrueFoundry among them, are starting to unify both under one control surface. See MCP gateway comparison 2026 for that adjacent category.
Does switching away from LiteLLM require a full rewrite? Typically less than expected if the application layer already calls a gateway through a stable internal interface rather than calling LiteLLM's SDK directly everywhere — the migration cost is concentrated in the gateway configuration and any custom plugins, not application code. Confirm this with a small proof-of-concept before committing to a full cutover.
What should the proof-of-concept actually test? Route a representative slice of real traffic — not a synthetic benchmark — through the candidate alternative for a defined period, and measure the things that actually drive the decision: whether budget enforcement stops spend rather than just reporting it, whether audit logs survive an attempted deletion, and whether the on-call burden genuinely drops. A proof-of-concept that only checks latency and routing correctness will miss the governance gaps that usually motivate the switch in the first place.