Evan King

For whatever reason, candidates LOVE to propose CDC in places where it doesn’t belong. CDC, or Change Data Capture, reads changes from a database’s replication log and streams them to downstream systems. It’s very handy! But only when another system just needs a copy of your data: - Postgres → Elasticsearch - Postgres → Snowflake - Old database → New database But engineers love to (incorrectly) use it as a generic trigger for business logic. Say you kick off fulfillment whenever orders.payment_status changes to PAID. Months later, an admin runs a repair script to correct old orders whose payment status was wrong. And one of them changes to PAID. CDC sees the exact same database change and kicks off fulfillment again. This is because CDC knows what changed. But it doesn’t know why. If the application knows a business event happened, represent that event explicitly. Use an outbox, enqueue a job, or call the dependency directly. I wrote about all the places to use CDC and when to avoid it here: https://lnkd.in/gmvpz5ud

There is no better way to learn system design than from the biggest companies in the world. Let's look at how Shopify solved inventory reservation at scale! Every checkout on Shopify has to ensure the last item in stock never sells to two people at once. So when you start paying, Shopify will hold your items for a few minutes so nobody else can take them.  For years, those reservations lived in Redis as a simple count, with one number per item. Reserving a unit just meant decrementing the count by one, and releasing one adds it back. Redis is fast, and because it's single-threaded, it runs commands one at a time, so those decrements never overlap (No double spend problem!) But the actual inventory lived in their primary MySQL database. Which means that the hold was in one database and the source of truth was in another, so the two could never commit together. If a payment cleared and the system crashed before both writes landed, Shopify would either oversell or hide stock that existed. As Shopify grew, this corner case became more frequent. The obvious fix is to move the hold into MySQL next to the inventory so both commit in a single transaction. But the problem is that all of those reservations funnel through one row. A single counter per item means every buyer for a hot product competes for the same row's lock, and MySQL hands that lock to one transaction at a time. In a flash sale, that makes thousands of concurrent checkouts go one by one.  The fix they landed on was to move reservations from a single counter row to one row per unit. Each reservable unit gets its own row, so instead of every buyer fighting over the same number, they lock separate rows and commit in parallel. MySQL 8 SKIP LOCKED makes this possible and is worth looking up if you're unfamiliar. It allows a buyer to grab a free row and skips right past any row someone else is holding, so nobody waits.  But a row per unit means a hot item could sprout hundreds of thousands of mostly idle rows, and across millions of items, the scan to find a free one keeps getting slower.  So they cap it. No item gets more than 1,000 reservation rows, and a background process refills the pool as units get claimed. If a rush drains it faster than that process can keep up, the next request refills the pool itself and retries, with a lock so only one refill ever runs at a time.  And because the hold now sits in the same database as the inventory, the reservation and the sale finally commit together!

The best system design lessons usually come from real systems pushed past their limits. In 2022, Discord migrated its message tables from Cassandra to ScyllaDB. The reason was that its Cassandra cluster had become incredibly difficult to keep healthy. At the time, Discord was storing trillions of messages across 177 Cassandra nodes. Cassandra was a natural fit for this workload because Discord writes an enormous number of messages, and Cassandra is designed to make those writes cheap. A new message is appended to a commit log and written to memory. Once that memory fills up, Cassandra flushes it to disk as an immutable file called an SSTable. The problem is that Cassandra never updates those files in place. As messages are edited or deleted, the latest state can end up spread across several SSTables, so reads have to check multiple files and merge the results together. Cassandra handles this with a background process called compaction, which combines those files so future reads have less work to do. But Discord’s cluster was regularly falling behind. As uncompacted files piled up, reads got slower. Those slower reads consumed more CPU and disk, leaving even fewer resources for compaction, which caused the backlog to grow even further. And you can’t just turn compaction up. It competes with live traffic for the same resources, so making it more aggressive can make latency even worse. Discord engineers ended up taking nodes out of service one at a time so they could compact without serving traffic, then bringing them back and waiting for them to catch up on all the writes they missed. They called this the “gossip dance.” Cassandra also runs on the JVM, so garbage collection would occasionally pause a node while it reclaimed memory. The worst pauses caused huge latency spikes and sometimes forced engineers to reboot the node entirely. ScyllaDB gave Discord a way out without redesigning how its messages were stored. It supports Cassandra’s data model and query language, but it’s written in C++, so there’s no JVM garbage collector. Its shard-per-core architecture also let each machine handle more work and made repairs much faster. After the migration, Discord reduced the cluster from 177 nodes to 72, while historical-message p99 latency fell to a steady 15 milliseconds. ScyllaDB doesn’t eliminate Cassandra’s tradeoff between cheap writes and more expensive reads and compaction. It just handles that tradeoff far more efficiently, giving Discord much more headroom before the cluster falls behind.