Skip to content
All problems (16)
System Design courseSystem Design Problems

0%

0/16 done

Practice · High Level Design · Work in the Background

Design a Web Crawler

IntermediateURL frontierPolitenessDedupDistributed workers⏱ 3 h

Crawl a billion pages politely: a URL frontier, per-site rate limits, spotting duplicate pages, and spreading the work across many machines.

Read first:Message Queues and Async CommunicationBackground Jobs and Task Queues

✎

This one is being written

The full solution is on its way. Here is its outline, so you can try designing it yourself first.

  1. 01Problem introduction
  2. 02Problem statement
  3. 03Requirements
  4. 04Capacity estimation
  5. 05API design
  6. 06Database design
  7. 07Architecture
  8. 08Request lifecycle
  9. 09Component deep dive
  10. 10Scaling, 100 to 10M users
  11. 11Failure scenarios
  12. 12Reliability and observability
  13. 13Security
  14. 14Trade-offs
  15. 15Final architecture
  16. 16Interview discussion

Designed it yourself and understood the solution? Mark it to track your progress.