The Preflight Check for Distributed Systems
The failure modes that take systems down, and where to look for each one
The Preflight Check for Distributed Systems
Representing Maryland at BNA
The failure modes that take systems down, and where to look for each one
Pilots do not trust the airplane. Before every flight, two people run a checklist out loud, call and response, flaps set, trim set, controls free and correct, fuel checked, instruments cross-checked. The aircraft flew yesterday. They check it again today.
Distributed systems call for the same suspicion. The service passed its tests. It served traffic all last week. None of that tells you what happens at 3 AM when one dependency slows down by two hundred milliseconds, and the slowdown spreads outward like a crack in glass.
Aviation runs two defenses, and software needs both. The checklist catches the problem on the ground, where you have time. The alarms catch it in the air, where a single tone has to tell you which of ten things is about to end the flight. Most outages I have read about, and most I have written about, come from skipping the first and then drowning in the second. The list below is the greatest hits.
Timeouts
The happy path returns in twenty milliseconds. The path you forgot is the one where the other end accepted your connection, took your request, and then went quiet, holding the line open while it slowly dies. A call with no timeout behaves like a control cable with no stop. It lets your thread run all the way to the wall.
Put an explicit timeout on every outbound call, and a deadline that travels with the request. If service A gives B two seconds, B should know it has under two seconds left when it calls C, not start a fresh two-second clock. Skip that, and a two-second budget quietly becomes six as every hop resets it.
Retries
One failed request retries, which is reasonable. Ten thousand failed requests retry at the same instant against a dependency that is already overloaded, and the retries themselves become the outage.
Use backoff with jitter, so retries are spread out instead of arriving as a wall. Add circuit breakers that stop sending traffic to something clearly down, and a retry budget that caps how much of total load is allowed to be retries. Watch the ratio. When retries pass a few percent of total requests, they are the load.
Pool exhaustion
Connection pools, thread pools, and file handles run dry the same way, and the obvious metrics look fine while it happens. The database is healthy. The CPU is bored. Requests pile up anyway, because every one is waiting for a connection from a pool of fifty that is fully checked out by slow queries upstream. The pool you forgot to size sets the ceiling, and nothing on the dashboard points at it.
Size pools against real concurrency, and decide what happens when the pool is empty. Does a caller block forever, or fail fast and shed load. Backpressure that rejects work early beats a queue that grows until the process runs out of memory.
Clock skew
Two machines disagree about now by a few hundred milliseconds, and you used that clock to decide which write came last, or whether a token is still valid, or which event happened first. Every log agrees, confidently, on an order that never happened. A mis-set altimeter does the same thing, every instrument is both certain and wrong.
Do not depend on wall-clock time for ordering or causality. Use logical sequence numbers or version vectors, and treat skew as a fact rather than a bug. Keep an eye on anything with a clock baked in, especially the things that expire.
Expiry
These fail on a calendar, not under load, so no load test will find them. An expired TLS certificate takes down a service nobody touched. A token that was supposed to rotate and never did. A disk that fills because log rotation was configured everywhere except for one box it wasn’t. Aircraft handle this with life-limited parts, replaced on schedule, whether they look fine or not.
Make a list of everything with a countdown attached: certificates, secrets, API tokens, domain registrations, license keys, disk headroom, partition counts. Alarm each one with weeks of warning, not hours. Your manager remembers this outage, because it sat on a calendar the whole time and still got missed.
Dual writes
You save the order to the database, then publish a message to the queue so the rest of the system learns about it. Between those two steps, the process can die, and now the order exists in one place and not the other. Nobody gets an error. The two stores just disagree, and nothing tells you.
Watch every place you change state in one system and then tell another about it. The outbox pattern, writing the event into the same transaction as the data and shipping it separately, closes the gap. If you cannot do that, know where the gaps are, because that is the work the reconciliation jobs exist to clean up.
Cache stampede
A single cache miss is fine. A million at the same instant, all for the same hot key, all hitting the database the moment a cached value expires, is a self-inflicted thundering herd. A cold cache right after a deploy has the same shape, every box reaching for the same data because none of them warmed up first.
Stagger expiry so a thousand keys do not die in the same second under the same TTL, and add a small random spread. Put a lock or single-flight guard in front of the fetch so one miss populates the cache and the rest wait. Watch the database for the sawtooth that lines up with your TTL.
Deploys
Most outages come from a change, not a clever distributed-systems edge case. A config edit, a feature flag flipped for everyone at once, a deploy that shipped without the migration that should have run first. The clever edge cases get written up, but a plain change takes you down most weeks.
Treat config changes with the same fear as code, because they are code. Roll out to a slice before everyone. Make rollback one step a tired person can run at 3 AM without thinking. When you write the postmortem, check the deploy log first. The cause is usually at the top, timestamped two minutes before the alarms.
Observability
In production, you cannot watch a request with your eyes. If you cannot follow one request across every service it touches, with a trace that ties the spans together, you are guessing which of seven services added the latency. A pilot who loses the horizon flies a confident, wrong attitude into the ground, and production hands you the same blindness by default.
Run distributed tracing across service boundaries, keep structured logs you can actually query, and give metrics enough cardinality to find the one customer or shard on fire. Before you ship, ask the boring question. When this breaks at 3 AM, what will I be able to see?
Alerting
A pager that fires forty times a night teaches whoever holds it to silence it without reading, so the real page gets silenced with the noise. Alarm fatigue follows predictably from alarms that fire too often and mean too little. The on-call engineer is not the problem. Aviation lived this in a crash where the stall warning sounded dozens of times while the crew, buried under competing alarms, never responded to it.
Track the ratio of pages to real incidents. Separate the two severities the way a cockpit does, a caution that means look when you can, from a warning that means act now, and keep the top tier almost empty. An alarm that maps to no action only trains people to ignore the next one. Delete it or demote it.
The checklist
A preflight check is boring on purpose. You read the line, you confirm the thing, you move on, and the teams that run it every time spend fewer nights on the incident bridge.
Write your own down. Start with the items above, then add a line every time something breaks in a way the list did not catch, the way aviation checklists were written in the aftermath of the accidents that made them necessary. Run it before you trust a new service, and again before a big launch. At 3 AM, holding a pager and a cooling cup of coffee, you will be glad you looked while you still had the time.
By Joshua McDonald on June 18, 2026.
Exported from Medium on August 26, 2026.
Reader discussion