YOUR BRIEF
What to work through
Work through an LLM inference batching service as a spoken system design interview. Start by clarifying the scope, sketch a working design and explain the trade-offs behind your choices.
- Accept a synchronous request and hold the caller until its answer is ready
- Group requests into batches of up to 100 inputs for one GPU
- Return each output to the caller that sent the input
- Rate limit callers and retry failed batches
- Lose no queued work when a node dies
BEFORE YOU START
Constraints to discuss
- Up to 100 inputs per batch on one GPU
- In one variant, each batch takes a fixed 100 ms on the GPU
- No accepted request is lost when a node dies
- Tail latency (p99) matters more than the median
Need a starting point?
Watch for these common gaps as you prepare:
- Skipping how a result gets back to the synchronous caller
- Getting stuck on model architecture or GPU internals
- Waiting for the interviewer to set the scope
- No answer for a batch that never fills, a timeout or a partial failure
- A textbook design recited when the problem has no standard answer
Why this question is in the collection
These sources connect the problem to company interview reports or preparation guides. They do not guarantee what a company will ask you.