Infrastructure at Scale

TL;DR

Uber built a custom datastore on MySQL because they trusted it operationally. Stripe serves 5M QPS with zero-downtime migrations using disciplined expand-contract patterns. Shopify shards by merchant using isolated "pods" and migrates tenants live with Ghostferry. The common thread: operational expertise and disciplined processes matter more than exotic technology.

Uber Schemaless — Building on What You Know

The Crisis

In early 2014, Uber's trip data was stored in a single PostgreSQL instance. The company was growing exponentially — more cities, more drivers, more trips every day. At the current growth rate, the database would hit capacity before year's end.

They evaluated existing distributed databases — Cassandra, Riak, MongoDB — but rejected them all. The reason wasn't technical capability. It was operational trust.

"We prioritized operational trust in the system, as it contains mission-critical trip data."

Uber's team knew MySQL deeply. They had years of experience running it, debugging it, and recovering from failures. Cassandra might theoretically handle the scale, but if something went wrong at 3 AM, did the team know how to fix it? With MySQL, the answer was yes.

The Data Model

Schemaless is an append-only sparse three-dimensional persistent hash map, similar to Google's Bigtable:

Cell = (row_key, column_name, ref_key) → JSON blob

row_key:     UUID (primary identifier, e.g., trip ID)
column_name: String (logical grouping, e.g., "trip_data", "payment")
ref_key:     Integer (version number — newer versions have higher ref_keys)

Append-only: Cells are never modified or deleted. An "update" writes a new cell with a higher ref_key. Reading always returns the latest ref_key.

Why append-only matters: - No destructive operations — data is never lost - Simpler replication (just ship new cells to replicas) - Natural audit trail (every version preserved)

Architecture

Each MySQL shard stores cells in a simple table structure. The routing layer handles: - Shard assignment: Hash-based routing by row_key - Buffered writes: If a MySQL master fails, writes are buffered temporarily. This favors write availability over read-your-writes consistency — a deliberate trade-off for a system where every trip must be recorded. - Triggers: A pub/sub system for downstream consumers. When a cell is written, interested services receive a notification.

Evolution

Uber's database story didn't stop at Schemaless: - MySQL to MyRocks: Migrated the storage engine from InnoDB to MyRocks (Facebook's RocksDB-based MySQL engine) for 50% better compression - Docstore: Evolved Schemaless into a more general-purpose document store - Adopted other databases too: Cassandra for real-time analytics, Redis for caching, Elasticsearch for search

The Lesson

Build on what you know deeply. Operational familiarity > theoretical superiority.

Uber didn't pick the "best" database — they picked the one their team could run with confidence at 3 AM when things break. Then they evolved it as their expertise and requirements grew.

Stripe — Zero-Downtime at 5M QPS

The Stakes

Stripe processes payments. Downtime means merchants can't charge customers. Data corruption means money goes to the wrong place. The bar for reliability and correctness is absolute.

DocDB

Stripe built DocDB, a custom document-oriented database: - 5 million queries per second - < 2 second traffic switches between shards during migrations - 99.999% uptime (about 5 minutes of downtime per year)

Zero-Downtime Migrations

Stripe's approach to schema changes and database migrations is legendary for its discipline:

1. Double-write: When migrating between shards or schemas, write to both old and new targets simultaneously.

2. Backfill: Copy all historical data from old to new. For large datasets, this runs for days or weeks.

3. Verify: Automated comparison between old and new data. Every row checked for consistency.

4. Shadow read: Read from both old and new, compare results. Any discrepancy triggers an alert — the migration doesn't proceed until results match.

5. Cutover: Switch reads to the new target. At < 2 seconds, this is nearly instantaneous from the user's perspective.

6. Cleanup: Remove the old target after a safety period.

Every step is reversible. If step 4 reveals inconsistencies, they fix the issue and re-verify. If step 5 shows problems, they cut back to the old target.

Online Schema Changes

Stripe performs schema changes on live tables using a technique similar to GitHub's gh-ost: 1. Create a shadow copy of the table with the new schema 2. Stream changes from the original table to the shadow copy via binlog 3. When the shadow is caught up, atomically rename (shadow → production, production → old) 4. Drop the old table after verification

The original table is never locked. Reads and writes continue throughout.

The Lesson

At Stripe's scale, the tooling and processes for safe changes are as important as the database itself. They invested heavily in: - Automated verification at every migration step - Reversibility by default - Shadow reads to catch inconsistencies before they affect production - Fast cutover (< 2 seconds) to minimize risk windows

Shopify — Pod Architecture

The Model

Shopify's sharding strategy is brilliantly simple: pods.

A pod is a complete, isolated stack: - Its own set of application servers - Its own database (MySQL) - Its own Redis instances - Its own job queue

Each pod serves a set of merchants. All data for a merchant lives on one pod.

Why Pods Work

Natural isolation: A merchant's data never leaves their pod. No cross-shard queries. No distributed transactions. Each pod is essentially a small, independent application.

Blast radius: If Pod 2's database crashes, only merchants D, E, F are affected. Merchants on other pods are completely unaffected. Compare this to a shared database where one failure takes down everyone.

Black Friday: Shopify's biggest challenge is traffic spikes. When a major merchant runs a flash sale, their pod handles the surge independently. Other pods are unaffected.

Ghostferry — Live Merchant Migration

When a pod gets too large or a merchant outgrows their pod, Shopify uses Ghostferry to migrate merchants between pods with zero downtime:

1. Start copying the merchant's data from source pod to destination pod
2. Stream ongoing changes via binlog replication
3. When the destination is caught up, cut over the merchant's traffic
4. Clean up the source pod

The merchant experiences no downtime and no data loss. This is the expand-contract pattern applied to tenant migration.

The Lesson

Pod architecture is the simplest sharding strategy when your access patterns are naturally scoped to a tenant: - No cross-shard queries (because all tenant data is co-located) - No distributed transactions (because everything is on one machine) - Simple blast radius containment - Clear capacity planning (one pod = N merchants, add pods as you grow)

Interview Tip

When designing multi-tenant systems (e-commerce, SaaS), mention pod architecture: "I'd shard by tenant using isolated pods — each pod has its own database, app servers, and cache. All data for one tenant stays on one pod, so there are no cross-shard queries. For rebalancing, I'd use a live migration tool like Ghostferry to move tenants between pods without downtime."

The Common Thread

All three companies share a philosophy:

Principle	Uber	Stripe	Shopify
Build on familiar technology	MySQL (team expertise)	MySQL/custom (team built it)	MySQL (simple, proven)
Operational discipline > clever architecture	Append-only for simplicity	6-step reversible migrations	Pod isolation for blast radius
Evolve incrementally	Postgres → Schemaless → DocStore	Continuous migration tooling	Add pods, migrate tenants
Right-tool-for-the-job	Later added Cassandra, Redis	Custom tooling per workload	Pods + shared services

The best database architecture isn't the most theoretically elegant — it's the one your team can operate, debug, and evolve with confidence.

Quick Recap

Company	Architecture	Key Insight
Uber	Append-only Schemaless on MySQL	Operational trust > theoretical capability
Stripe	DocDB with 6-step reversible migrations	Tooling and process make safe changes at scale possible
Shopify	Isolated pods sharded by merchant	Tenant isolation is the simplest sharding strategy