Litellm

LiteLLM is an open-source AI gateway and LLM proxy for platform teams. It puts your whole AI stack behind one key, so applications, agents and people call a single OpenAI-compatible endpoint while LiteLLM routes requests to the right model. The project has about 58,800 stars on GitHub and is maintained by BerriAI. It can be self-hosted, and an enterprise edition is available.

What the gateway does

LiteLLM provides one consistent way to access more than 100 model providers and endpoints, plus MCP servers and agents. You can see who is driving spend, cap budgets before they are exceeded and send each request to the model that should handle it. The site groups features into optimisation, observability, routing and governance. Routing covers load balancing, fallbacks and retries across providers. Governance includes virtual keys, team and user budgets, rate limits and logging.

How teams use it

Testimonials describe how companies use it. An NVIDIA product lead says LiteLLM gives engineers a single, consistent way to access more than 100 AI model endpoints. A Netflix engineer says it lets the team offer the latest models to users, usually within a day of release, saving months of work. An Okta architect notes that switching a backend model becomes a simple configuration update in the gateway, with no code changes, procurement cycles or repeated security reviews. The site also highlights an AT&T case study about cutting coding costs by 56 percent.

Getting started

A quick-start script installs the proxy on macOS, Linux or Windows, and a Python SDK lets developers call many providers with the same function signature.

Pricing

The open-source version is free to self-host, and an enterprise option is available through the Try Enterprise link and a sales contact. The site notes one login with SSO, scoped by team, project or app, and support for more than 1,800 models.

Who it suits

LiteLLM suits platform and infrastructure teams that offer several LLMs to internal developers and need cost control and a single integration point.

Things to consider

Self-hosting means you run, secure and update the proxy yourself. Store provider API keys carefully, and test fallback behaviour before production use.

Start with a small set of models and teams, check the logs and spend reports, and expand once routing works as expected.