Brief #16 · AI & LLM
LLM Gateway (Multi-Model Router)
A single service every team calls instead of the LLM providers directly: streaming, caching, fallbacks when a provider is down, budgets and a cost dashboard.
Outlines are free · building with tracking needs Pro
Six teams call three LLM providers directly, each with its own keys, retries and no idea what it spends. Last month one provider was down for two hours and four features broke. Build a gateway so every team uses one API, outages fail over automatically, repeated questions are answered from cache, and finance can see who spent what.
What you'll build
- 1One OpenAI-compatible endpoint that forwards to two providers, including streaming responses token by token.
- 2Per-team API keys with token budgets and rate limits, and a clear error when a team runs out.
- 3Automatic fallback: on timeouts or 5xx errors, retry once, then switch provider, with a circuit breaker so a dead provider isn't hammered.
- 4A cache for repeated prompts: exact match first, then a semantic cache using embeddings, with a similarity threshold you tuned.
- 5Log tokens, latency and cost for every request, and show spend per team per day on a dashboard.
Step-by-step walkthrough
Walkthrough coming soon
This project's step-by-step walkthrough, with real outputs at every step, is still being written. You can read the brief and try it on your own meanwhile; tracking opens when the walkthrough is ready.
Done when
- Stopping one provider mid-test keeps requests succeeding, and you measured the extra latency during failover.
- Streaming adds less than 50 ms to time-to-first-token compared with calling the provider directly.
- Cache hit rate and money saved on a replayed day of real traffic, plus examples where the semantic cache would have been wrong.
- No provider key ever leaves the gateway or appears in logs.
Stretch goals
- Route easy prompts to a cheaper model and hard ones to a stronger one, and measure quality and savings.
- Run the gateway as three replicas with a shared rate limit.