Design a Local Delivery Service
TL;DR
A local delivery service lets users order food or groceries from nearby stores with delivery in under an hour. The interesting constraints are not the ordering flow (that is standard e-commerce) but the inventory problem: a restaurant's menu or a store's shelf stock changes continuously, the read-to-write ratio is 175:1, and the system must handle the case where an item shows as "in stock" when you add it to cart but is actually depleted by the time you check out. DoorDash uses the driver rejection rate as a real-time signal for menu item depletion. The adaptive cache TTL -- short for items that sell quickly, long for items that sit on shelves -- is the key insight that makes the system work without overwhelming small merchant POS systems with inventory queries.
The System
A local delivery service connects consumers with nearby restaurants and stores, dispatches delivery drivers, and fulfills orders within 30-60 minutes. The user browses stores, adds items to a cart, places an order, and a driver picks up the items and delivers them.
DoorDash processes over 2 million orders per day across 900,000+ merchants in the US, Canada, Australia, and Japan. Instacart handles grocery deliveries from 1,400+ retail banners. Uber Eats processes over 1 billion orders annually. The unit economics are brutal: DoorDash's average order value is ~$35, commission is ~20% ($7), driver pay is ~$5-8, leaving $0-2 margin per order. At this margin, operational efficiency is existential. A 1% increase in failed orders (item unavailable, driver no-show) can erase the entire profit margin. This is why inventory accuracy and dispatch optimization are the most critical engineering problems.
Requirements
Functional
- Store discovery: Show nearby stores/restaurants based on user location, with estimated delivery time
- Menu/catalog browsing: Display available items with prices, descriptions, and availability status
- Order placement: Cart, checkout, payment, and order confirmation
- Real-time order tracking: Track order status from placed to delivered
- Driver dispatch: Assign a delivery driver to the order and optimize pickup/delivery routing
- Inventory management: Track item availability and surface out-of-stock items before checkout
Non-Functional
- Catalog read latency: Store menu/catalog loads in under 200ms (175:1 read-write ratio)
- Order throughput: 2M orders/day = 23 orders/sec average, 100/sec peak (dinner rush)
- Inventory freshness: Item availability reflects actual stock within 5 minutes for fast-moving items, 30 minutes for slow-moving items
- Order success rate: 98%+ of placed orders are fulfilled without substitution (inventory accuracy is critical)
- Availability: 99.9% uptime for the ordering path
- Payment reliability: Zero dropped payments. At-least-once charge, exactly-once fulfillment
Back-of-Envelope Math
Stores/restaurants: 900K
Items per store: average 100 (restaurant menu) to 30,000 (grocery store)
Total catalog items: ~500M (most are grocery SKUs)
Read traffic:
Active users browsing: 10M/day
Average pages per session: 5 (store list, store detail, item details)
50M page views/day = 579 views/sec
Peak (3x): 1,737 views/sec
Each page: 1-5 API calls = 2,893 - 8,685 API calls/sec peak
Write traffic:
Menu/price updates: 500M items, ~1% change daily = 5M updates/day = 58/sec
Order writes: 23/sec average, 100/sec peak
Total writes: ~158 writes/sec peak
Read:write ratio: 8,685 / 158 = ~55:1 for API calls
For catalog data specifically: ~175:1 for catalog reads vs writes
Order data:
Each order: ~2 KB (items, prices, addresses, driver assignment)
2M orders/day * 2 KB = 4 GB/day
Retain 90 days: 360 GB. Single database handles this.
Catalog storage:
500M items * 500 bytes = 250 GB
With indexing overhead: ~500 GB
Fits in a single database with read replicas.
The Naive Design
┌──────────┐ ┌──────────────┐ ┌──────────────┐
│ User │────>│ API Server │────>│ PostgreSQL │
│ App │<────│ (monolith) │<────│ │
└──────────┘ └──────────────┘ └──────────────┘
-- Browse store:
SELECT * FROM items WHERE store_id = 123 AND available = true;
-- Place order:
BEGIN;
UPDATE items SET quantity = quantity - 1 WHERE id IN (item_ids) AND quantity > 0;
INSERT INTO orders (...) VALUES (...);
COMMIT;
A single database, a single server. For a food delivery startup with 100 restaurants, this works for the first year.
Where Does This Break First?
The available flag. A restaurant's item availability changes when they run out of an ingredient, but the POS system (if it even has an API) does not push real-time updates. You are showing items as available that are actually sold out. The user orders, the restaurant rejects, the user has a terrible experience. Inventory accuracy, not throughput, is the first failure mode.
Where It Breaks
Problem 1: Inventory data is unreliable at the source. Most restaurants do not have real-time inventory APIs. A pizza place runs out of pepperoni at 7 PM. There is no automated system to tell your platform. The first signal is when the restaurant rejects the order, or worse, when the driver calls the customer to ask about substitutions. Item unavailability is a top source of customer complaints in food delivery.
Problem 2: The 175:1 read-write ratio creates a caching challenge. 175 reads for every write means aggressive caching is essential. But cache invalidation for inventory is tricky -- if an item sells out and the cache serves stale "available" data for 5 minutes, hundreds of users see an item they cannot actually order. The TTL must be adaptive: fast-selling items need short TTLs, slow-selling items can have long TTLs.
Problem 3: Race conditions during checkout. Two users add the last unit of an item to their carts simultaneously. Both see it as available. Both submit orders. Only one can be fulfilled. Without optimistic locking, one user gets charged and then told "sorry, item unavailable." The refund process creates friction and erodes trust.
Problem 4: Multi-item orders with partial availability. A grocery order has 30 items. Three are out of stock. Do you fulfill the remaining 27? Ask the user to approve substitutions? Cancel the entire order? Each choice has different UX implications and the system must handle all three paths.
Problem 5: The write path is deceptively simple. 23 orders/sec is trivial for a single PostgreSQL instance. But each order triggers a cascade: payment authorization, driver dispatch, restaurant notification, inventory update, ETA calculation, push notification. The fan-out from one order touches 8-10 services. The write path is a distributed transaction, and getting the saga pattern right is the actual engineering challenge.
The Real Design
┌──────────────┐ ┌──────────────┐
│ User App │──── browse ──────────────────────> │ Catalog │
│ │<─── catalog data ────────────────── │ Service │
│ │ │ (read-heavy)│
│ │ └──────┬───────┘
│ │ │
│ │ ┌──────v───────┐
│ │ │ Redis Cache │
│ │ │ (adaptive │
│ │ │ TTL) │
│ │ └──────────────┘
│ │
│ │──── order ────┐
│ │<── confirm ───│
└──────────────┘ │
┌──────v───────┐
│ Order │
│ Service │
└──────┬───────┘
│
┌───────────────────┼───────────────────┐
│ │ │
┌──────v───────┐ ┌────────v──────┐ ┌─────────v─────┐
│ Payment │ │ Inventory │ │ Dispatch │
│ Service │ │ Service │ │ Service │
│ (auth hold) │ │ (optimistic │ │ (driver │
│ │ │ locking) │ │ matching) │
└──────────────┘ └──────────────┘ └───────────────┘
│ │ │
└───────────────────┼───────────────────┘
│ saga coordinator
┌──────v───────┐
│ Order Saga │
│ (state │
│ machine) │
└──────────────┘
The 175:1 Read-Write Ratio
For a food delivery catalog (store menus and item availability), the estimated read-to-write ratio is ~175:1. This is the most important number in the system because it determines the caching strategy.
At 175:1, aggressive caching is not just an optimization -- it is a requirement. Without caching, the catalog database handles 8,685 reads/sec at peak. With a 95% cache hit rate, database load drops to 434 reads/sec. That is the difference between needing a fleet of read replicas and needing a single instance.
Cache architecture:
User request -> CDN (static assets) -> API Gateway
-> Redis Cache (catalog data, 95% hit rate)
-> PostgreSQL read replica (cache miss, 5%)
-> PostgreSQL primary (writes only)
DoorDash Rejection Rate as Depletion Signal
This is the most clever engineering insight in the system. DoorDash uses the driver/restaurant rejection rate for a specific item as a proxy for real-time inventory depletion.
def track_item_availability(store_id, item_id, event):
key = f"rejection:{store_id}:{item_id}"
if event.type == "order_rejected" and event.reason == "item_unavailable":
redis.incr(key)
redis.expire(key, 3600) # 1-hour window
rejection_count = redis.get(key)
if rejection_count >= 3:
# 3 rejections in an hour -> mark item as likely unavailable
mark_item_unavailable(store_id, item_id)
# Shorten cache TTL for this store's catalog
set_cache_ttl(f"catalog:{store_id}", ttl=60) # 1-minute refresh
elif event.type == "order_fulfilled" and item_id in event.items:
# Reset rejection count on successful fulfillment
redis.delete(key)
Why this works: When a restaurant runs out of an item, the first few orders containing that item get rejected. After 3 rejections, the system learns the item is unavailable and stops showing it. The feedback loop takes 10-15 minutes (3 orders * 5-minute average order gap), which is imperfect but far better than waiting for the restaurant to manually update their menu (which may never happen).
False positives: Sometimes a restaurant rejects an order for reasons unrelated to item availability (too busy, closing early). The system must distinguish between "item unavailable" rejections and other rejection types. DoorDash's tablet interface gives restaurants specific rejection reasons to click.
Adaptive Cache TTL by Stock Level
Not all items deserve the same cache TTL. A popular dish that sells 200 orders per dinner rush needs a shorter TTL than a specialty item that sells 2 per week.
def calculate_adaptive_ttl(store_id, item_id):
# Historical sell rate (orders per hour at current time-of-day)
sell_rate = get_sell_rate(store_id, item_id, current_hour())
if sell_rate > 20: # sells every 3 minutes
return 60 # 1-minute TTL (check availability frequently)
elif sell_rate > 5: # sells every 12 minutes
return 300 # 5-minute TTL
elif sell_rate > 1: # sells every hour
return 900 # 15-minute TTL
else:
return 1800 # 30-minute TTL (slow-moving item)
TTL per store category:
| Store type | Fast items TTL | Slow items TTL | Rationale |
|---|---|---|---|
| Fast food | 2 min | 15 min | Limited menu, items sell out quickly |
| Full restaurant | 5 min | 30 min | Larger menu, more predictable stock |
| Grocery | 1 min | 60 min | Some items (bananas) deplete fast |
| Convenience | 5 min | 60 min | Small store, stock changes slowly |
Optimistic Locking for Inventory
When two users try to order the last unit of an item simultaneously, optimistic locking prevents double-selling.
-- Each item has a version number
-- Checkout reads the version, then updates with a version check
-- Step 1: Read item and version
SELECT id, quantity, version FROM items WHERE id = 123;
-- Returns: quantity = 1, version = 47
-- Step 2: Update with version check (optimistic lock)
UPDATE items
SET quantity = quantity - 1, version = version + 1
WHERE id = 123 AND version = 47 AND quantity >= 1;
-- If 0 rows updated: version changed (another order got it first)
-- Retry or return "item unavailable"
Why optimistic over pessimistic: Pessimistic locking (SELECT FOR UPDATE) holds a row lock during the entire checkout flow (3-10 seconds). At 100 orders/sec peak, this causes lock contention. Optimistic locking never holds locks -- it detects conflicts at write time. Since conflict rate is low (< 1% of items are ordered simultaneously), optimistic locking has nearly zero overhead in the common case.
Multi-DC Orders with Saga Pattern
A single order touches multiple services: payment, inventory, dispatch, notification. If any step fails, the preceding steps must be compensated (rolled back).
Order Saga:
Step 1: Payment Service -> authorize $35 hold on credit card
Step 2: Inventory Service -> reserve items (decrement stock)
Step 3: Restaurant Service -> send order to restaurant tablet
Step 4: Dispatch Service -> assign driver
Compensation (if Step 3 fails):
Undo Step 2: release item reservation
Undo Step 1: void payment authorization
Compensation (if Step 4 fails):
Wait for driver (retry 3 times, expanding search radius)
If no driver after 15 minutes: cancel order
Undo Step 3: cancel restaurant order
Undo Step 2: release item reservation
Undo Step 1: void payment authorization
Notify user: "No drivers available, order cancelled"
Why sagas, not distributed transactions? Two-phase commit (2PC) requires all participants to hold locks until all agree to commit. With 4 services, one slow service (payment gateway taking 5 seconds) blocks all others. Sagas allow each step to complete independently and compensate on failure. The trade-off: sagas are eventually consistent (there is a brief window where the payment is authorized but items are not yet reserved), but this window is milliseconds.
PostgreSQL Handles the Write Load
A common mistake in this interview: over-engineering the write path. Let's verify that a single PostgreSQL instance handles the write load.
Order writes: 100/sec peak (each order = 1 INSERT + 3-5 UPDATE for saga steps)
= 500 writes/sec peak
Inventory updates:
Order-related: 100 orders/sec * 5 items/order = 500 updates/sec
Menu updates from merchants: 58/sec
= 558 updates/sec
Total writes: ~1,058 writes/sec peak
PostgreSQL on a db.r6g.xlarge (4 vCPU, 32 GB RAM):
Handles 10,000+ writes/sec for simple INSERT/UPDATE
Our 1,058 writes/sec is 10.6% of capacity.
You do not need database sharding for 2M orders/day. A single PostgreSQL primary with 2 read replicas handles the entire write load with 90% headroom. The read replicas serve the 175:1 read traffic for cache misses.
When do you need to shard? At ~20M orders/day (10x), write load reaches ~10K writes/sec. At that point, shard the orders table by geographic region (cities are naturally independent). Inventory data for store X does not need to be on the same shard as store Y -- they are in different cities.
Deep Dives
Deep Dive 1: Delivery Time Estimation
Delivery time is the single most important number on the store listing page. If you promise 30 minutes and deliver in 45, the user does not order again.
Components of delivery time:
Delivery Time = Restaurant Prep Time
+ Driver Pickup Time (travel to restaurant)
+ Driver Delivery Time (travel to user)
+ Buffer
Example:
Prep time: 15 min (predicted from historical data for this restaurant)
Driver pickup: 8 min (distance from nearest driver to restaurant)
Driver delivery: 10 min (distance from restaurant to user)
Buffer: 2 min (parking, building access)
Total: 35 min
Prep time prediction: Each restaurant has a distribution of prep times. A fast food chain averages 8 minutes; a sit-down restaurant averages 25 minutes. The model is trained on historical order-to-ready times, adjusted for current order volume (if the restaurant has 10 pending orders, prep time is longer).
Dynamic adjustment: If 3 out of the last 5 orders from a restaurant were late, increase the estimated prep time by 20%. This prevents the system from continuously under-promising.
Deep Dive 2: Store Search and Ranking
When a user opens the app, they see a list of nearby stores ranked by relevance.
Ranking factors:
def rank_stores(user_location, stores):
for store in stores:
store.score = (
0.30 * delivery_time_score(store, user_location) # faster = higher
+ 0.25 * user_affinity(store, user) # past orders, favorites
+ 0.20 * store_quality(store) # ratings, order success rate
+ 0.15 * promotion_boost(store) # sponsored placement
+ 0.10 * cuisine_relevance(store, user, time_of_day) # sushi at lunch
)
return sorted(stores, key=lambda s: s.score, reverse=True)
Delivery time as the top factor: DoorDash places the highest weight on delivery time because it directly correlates with order conversion. A store that can deliver in 20 minutes converts 3x better than one that takes 50 minutes, all else being equal.
Sponsored listings: Restaurants pay for promoted placement. This is a significant revenue stream (DoorDash's "DashPass" and promoted listings generate substantial revenue). The ranking algorithm ensures sponsored listings appear in the top 5 but are clearly labeled.
Deep Dive 3: Driver Dispatch and Routing
When an order is placed, the dispatch service assigns a driver. The optimization is different from ride-sharing because the driver must go to the restaurant first, then to the user.
Batched dispatch: If two orders are from the same restaurant (or nearby restaurants) and going to nearby destinations, assign both to the same driver. The driver picks up both orders in one trip.
def batch_orders(pending_orders):
# Group by restaurant proximity (within 500m)
clusters = cluster_by_location(pending_orders, radius=500)
for cluster in clusters:
if len(cluster) >= 2:
# Check if destinations are also close (within 2 km)
dest_spread = max_distance(cluster, key=lambda o: o.delivery_location)
if dest_spread < 2000:
# Batch these orders for one driver
create_batched_delivery(cluster)
else:
# Too far apart, dispatch individually
for order in cluster:
dispatch_single(order)
Driver assignment optimization: Unlike ride-sharing (minimize pickup time), delivery dispatch optimizes for total time from now until the last delivery is completed. Factors:
- Driver's current location
- Time for driver to reach restaurant
- Expected food prep time remaining (food should be ready when driver arrives)
- Distance from restaurant to delivery location
- If batching: total route time for multiple deliveries
The ideal dispatch sends the driver to arrive at the restaurant exactly when the food is ready. Arriving 10 minutes early means the driver is idle. Arriving 5 minutes late means the food is cold.
Alternative Designs
Alternative 1: Event-Sourced Inventory
Every inventory change is an event (stock_added, stock_depleted, order_placed, order_cancelled). Current stock is derived by replaying events. Enables perfect audit trail and time-travel queries.
Alternative 2: POS Integration (Real-Time Inventory)
Directly integrate with restaurant POS systems (Square, Toast, Clover) for real-time inventory data. Eliminates the depletion-signal approach.
Alternative 3: Pre-Order Model (Scheduled Delivery)
Users order hours in advance. The restaurant confirms item availability before committing. Eliminates real-time inventory problems at the cost of spontaneity.
| Aspect | Adaptive Cache + Signals | Event-Sourced Inventory | POS Integration | Pre-Order Model |
|---|---|---|---|---|
| Inventory accuracy | 90-95% (signal-based) | 99%+ (event-sourced) | 99%+ (real-time POS) | 100% (confirmed) |
| Integration complexity | Low (no merchant work) | High (new data model) | Very High (1000s of POS) | Low (merchant confirms) |
| User experience | Occasional stockouts | Accurate availability | Accurate availability | No spontaneous ordering |
| Merchant onboarding | Easy (just a menu) | Hard (event modeling) | Hard (POS API access) | Easy |
| Latency | Sub-200ms (cached) | Sub-200ms (projections) | Depends on POS API | N/A (async) |
| Operational cost | Low | Medium (event store) | High (integration maint) | Low |
The adaptive cache with depletion signals is the right starting answer because it works without any merchant-side integration. As the platform grows and specific merchant categories dominate (e.g., grocery), add POS integration for those categories. Event sourcing is a good fit for the order service (audit trail, financial compliance) but overkill for restaurant menu data.
Scaling Math Verification
Catalog reads (8,685 API calls/sec peak):
- Redis cache hit rate: 95%
- Redis reads: 8,251/sec. Single Redis instance handles 200K+/sec. 4% utilization.
- Database reads (cache miss): 434/sec. Read replica handles this easily.
Order writes (100/sec peak):
- PostgreSQL writes: ~1,058/sec total. Single primary at 10% capacity.
- Saga steps per order: 4 service calls * 100ms each = 400ms per order.
- With 20 order service instances: 20 * (1000ms / 400ms) = 50 orders/sec per instance. Need 2 instances for 100/sec peak.
Inventory updates (558/sec):
- Optimistic lock contention rate: < 1% of updates conflict.
- Retry overhead: 558 * 0.01 = 5.58 retries/sec. Negligible.
Driver dispatch (100/sec):
- Each dispatch: find nearby drivers (1ms), estimate routes (10ms), assign (1ms) = 12ms
- Single dispatch instance: 83 dispatches/sec. Need 2 instances for peak.
Failure Analysis
| Component | Current capacity | At 10x (20M orders/day) | Breaks? | Fix |
|---|---|---|---|---|
| Redis catalog cache | 200K+ ops/sec | Still 200K+ ops/sec | No | More items cached, but ops/sec doesn't grow 10x |
| PostgreSQL writes | 10K+ writes/sec | ~10K writes/sec needed | Maybe | Shard by city/region |
| Order service | 2 instances | 20 instances | No | Horizontal scaling |
| Payment service | 100 txns/sec | 1,000 txns/sec | No | Payment gateway handles this |
| Dispatch optimization | 2 instances | 20 instances | No | Horizontal scaling |
| Catalog storage | 500 GB | 5 TB | Yes | Shard by store_id, archive old menu versions |
| Driver location index | 100K drivers | 1M drivers | No | H3 grid scales with driver count |
| Saga coordinator | 100 sagas/sec | 1,000 sagas/sec | No | Temporal handles 10K+ workflows/sec |
The first bottleneck at 10x is database writes approaching single-instance limits. The natural sharding key is geographic region (city or metro area). Orders from San Francisco never need to join with orders from New York. Each region gets its own PostgreSQL shard with its own catalog data, order tables, and inventory.
The more interesting bottleneck is catalog accuracy. At 10x scale (9M merchants), maintaining inventory signals across 9M stores requires a much larger signal-processing pipeline. The depletion signal approach scales linearly (more orders = more signals), but the cold-start problem worsens: new merchants with no order history have no signals, so the system cannot predict stockouts.
What's Expected at Each Level
| Aspect | Mid-Level | Senior | Staff+ |
|---|---|---|---|
| Inventory model | Boolean available flag | Stock levels with cache, mentions staleness | Depletion signal from rejection rate, adaptive TTL per item |
| Caching | "Cache the menu" | TTL-based cache with invalidation | 175:1 ratio, adaptive TTL, per-item sell-rate-based TTL |
| Concurrency | Not discussed | Database locking for checkout | Optimistic locking with version column, quantifies contention |
| Order lifecycle | REST API calls | State machine with database | Saga pattern with compensation, Temporal for orchestration |
| Database scaling | "Shard the database" | Explains why sharding is not needed at stated scale | Single PostgreSQL handles write load, shards at 10x by region |
| Dispatch | "Assign nearest driver" | Considers restaurant prep time | Batched dispatch, timing driver arrival with food readiness |
| Delivery time | Distance / speed estimate | Prep time + travel time components | ML-based prep prediction, dynamic adjustment from recent data |
| Real-world reference | "Like DoorDash" | Mentions read-write ratio | DoorDash rejection signals, specific POS integration challenges |
The single most important signal at any level: do you understand that inventory accuracy is the hardest problem, not order throughput? The write path is trivially handled by a single PostgreSQL instance. The inventory problem -- knowing what is actually available on a restaurant shelf that has no real-time inventory API -- is where the engineering challenge lies. The depletion signal approach demonstrates real-world thinking that no amount of database scaling can solve.
References from Our Courses
- Indexing Deep Dive — PostgreSQL spatial queries for delivery zone assignment
- Redis Data Structures and Use Cases — caching ETA estimates and driver availability
- Kafka Partitions and Ordering — order lifecycle event streaming
- Distributed Transactions — coordinating payment capture with delivery confirmation
Red Team This Design
Ready to stress-test this architecture? The Attack companion tears apart every decision in this design — from hardware physics to security holes to what actually happens at 10x scale.