YOUR BRIEF
What to work through
Work through a web crawler as a spoken system design interview. Start by clarifying the scope, sketch a working design and explain the trade-offs behind your choices.
- Crawl from a set of seed URLs, following the links each page holds
- Store the downloaded HTML
- Re-crawl new and edited pages to keep the store fresh
- Skip pages whose content was already stored
BEFORE YOU START
Constraints to discuss
- 1 billion pages a month (Alex Xu), or 10 billion pages in under 5 days (Hello Interview)
- Polite: obey robots.txt and send at most 1 request a second to a domain
- Keeps going through bad HTML, slow servers, crawler traps and crashed workers
- Keeps pages for 5 years
- Extensible to new content types
Need a starting point?
Watch for these common gaps as you prepare:
- No per-host queues, so one site gets a flood of requests
- Ignoring robots.txt or the crawl delay
- Putting raw HTML in the queue messages
- No guard against crawler traps
- Forgetting DNS, which can dominate each thread’s time
Why this question is in the collection
These sources connect the problem to company interview reports or preparation guides. They do not guarantee what a company will ask you.