YOUR BRIEF
What to work through
Work through an LLM gateway as a spoken system design interview. Start by clarifying the scope, sketch a working design and explain the trade-offs behind your choices.
- One API for every team, in one request shape, whichever provider serves the call
- Route each call to a provider and model by the team’s policy, and fall back to another when one fails
- Hold each team to its limits: requests and tokens per minute, and a spend budget
- Stream answers back token by token
- Record each call’s team, model, tokens, cost and latency for bills and audits
BEFORE YOU START
Constraints to discuss
- Adds little latency: the first token arrives about as fast as when calling the provider directly
- Keeps the company inside each provider’s own limits, which every team shares
- One provider’s outage does not take the gateway down
- Limits hold across every gateway instance, not per instance
- Prompts and answers are logged only as each team’s policy allows
Need a starting point?
Watch for these common gaps as you prepare:
- Limits kept in each instance’s memory, so the real limit grows with the number of instances
- Retrying every error, including bad requests that fail on every provider
- Buffering streamed answers, so the first token arrives late
- Counting cost only at the end of the month, so one runaway job blows through its budget
- Ignoring the provider’s own per-minute limits, which every team shares
- Turning the gateway into model serving: GPUs and batching sit behind the providers here
Why this question is in the collection
These sources connect the problem to company interview reports or preparation guides. They do not guarantee what a company will ask you.
- GitHub: Hello Interview’s question page for an API gateway in front of LLM providers, with policies per organization, lists it as asked at GitHub and Moveworks.
- Moveworks: Hello Interview’s question page for an API gateway in front of LLM providers, with policies per organization, lists it as asked at GitHub and Moveworks.
- Decagon: DesignGurus’ Decagon guide reports an onsite question: a gateway every model request passes through, with audit requirements, testing rate limits, logging and cost control.