跳轉至

LiteLLM Router

The litellm Router is a robust component within the litellm Proxy ecosystem designed to manage multiple AI model endpoints, handle load balancing, and implement advanced reliability features like fallbacks. It acts as an abstraction layer, allowing users to consolidate disparate API providers (such as openai, Google Gemini, mistral, or nvidia-free-nim-api-access) into a unified, OpenAI-compatible interface.

Key Features

Reliability and Fallbacks

One of the primary use cases for the Router is ensuring high availability. By defining a fallbacks configuration in the config.yaml, the Router can automatically redirect requests to a secondary model or provider if the primary service returns a 429 (Rate Limit) error or experiences a downtime incident. For example, a heavy-duty model used for translations can be configured to fail over to a faster, smaller model when quotas are exhausted.

Load Balancing and Routing

Instead of calling specific APIs directly, applications point to the Router, which intelligently routes traffic based on defined strategies. This is particularly useful when managing local instances (e.g., via llama.cpp or vLLM) alongside cloud-based models. It allows for complex setups where local hardware handles routine requests while overflows are routed to cloud providers.

Configuration Management

Configuration is primarily managed via a config.yaml file. This includes: - Model List: Defining custom model aliases that point to specific backends. - Parameters: Setting default temperature, top_p, and extra_body parameters to enforce consistent behavior across tools like SillyTavern, Cursor, or Claude Code. - Virtual Keys: Creating scoped API keys with specific model access, budget limits, and rate limiters, which is essential for multi-user or multi-tool deployments.

Implementation Steps

  1. Environment Setup: Ensure your docker or server environment has necessary credentials (e.g., NVIDIA_NIM_API_KEY, GROQ_API_KEY) defined as environment variables.
  2. Configuration: Create a config.yaml that defines your model aliases and fallback logic.
  3. Deployment: Run the proxy container with persistent storage for the configuration and optional database support (PostgreSQL) to store dynamic model metadata (STORE_MODEL_IN_DB=True).
  4. Key Management: Generate keys via the admin UI or the /key/generate endpoint to provide secure, tracked access to your tools.

Advanced Use Cases

  • Immersion Translation: By utilizing specific Virtual Keys with tailored RPM (requests per minute) limits, users can integrate translation tools seamlessly without triggering rate limits on expensive or strictly quota-controlled models.
  • Local Model Optimization: As demonstrated in local LLM setups, the Router can serve as a single management point for multiple sampling configurations of a single underlying model (e.g., creating separate 'think', 'code', and 'general' aliases for a single Qwen model).

For maintenance and debugging, administrators often rely on docker compose to manage both the Router and accompanying services like databases, ensuring that health checks and network definitions are properly synchronized.

Sources

See Also