Why system design matters more as AI makes building faster
What AI still can’t reason about no matter how well you prompt it.
Why system design matters more as AI makes building faster
What AI still can’t reason about no matter how well you prompt it.
Snakes!
The DC Audio Tour app used to call Overpass, a volunteer-run OpenStreetMap API, on every page load, pulling nearby points of interest so the map could show them. One user, no problem. A few dozen, no problem. Then I shared the link more widely, traffic climbed, and Overpass started timing out, because the app was hitting a volunteer service harder than it’s built to absorb.
Claude had written a correct API call. Retry logic, error handling, fallback messaging, all of it solid. I’d put that call on the critical path of every page load, though, so a slow third party could stall the whole screen for everyone. The first fix was to move it off that path. The map called Overpass only when someone tapped to open it. Page loads stopped hammering the API and the timeouts cleared.
Once the call fired only on a tap, the usage numbers showed me something the page-load version had buried. Almost nobody opened the map. I’d had no clean way to see that before, because the old design called Overpass whether anyone wanted the map or not. Two things would have saved me the detour. Thinking through the flow of calls before I built it, so a volunteer API never sat on the critical path of a page load. And wiring in the usage metrics from the start, so I’d have known months earlier that the feature the call served wasn’t one people reached for.
Code that works and a system that survives are two different problems. AI handed me the first one fast. The second stayed with me. Writing the code was never the slow part. What slowed me down was a decision I hadn’t made yet, the kind that never shows up in a diff.
Now & later
Good system design means holding the short term and the long term in your head at the same time. The short term is right now. Does this work today, for today’s load, in today’s architecture? The long term is whether the decision is right, and stays right. Will this survive 10x traffic? Does it box us in when we need to add caching? Will this schema fight us when we have to shard?
AI is very good at the short term. The code runs, the tests pass, the feature ships. The long term is harder because it asks you to reason about conditions that don’t exist yet. You don’t have 10x traffic. You haven’t hit the consistency problem. The distributed failure mode you’re designing against has never happened in your system. You’re betting on a future state you can only approximate.
The opposing truth has to sit right next to that one. Optimizing for the long term too early will wreck you on the short. Teams build microservices before they understand their domain well enough to draw the boundaries, then spend years stuck with the wrong ones. They reach for event-driven architecture before they know what events their domain produces. They add distributed caching before they understand their read patterns. They chose complexity early, before they knew which parts of the system were actually hard, and the complexity compounded faster than the product did.
People call it the rule of three. Write the code once, tolerate the duplication the second time, abstract on the third when the pattern is finally clear. Architecture follows the same logic. Build the monolith until you know which parts should become their own services. Build synchronously until you know what needs to be async. Normalize the schema until you find out which queries are slow. The long term matters, and so does knowing when to start worrying about it.
Taking the long view without guessing means grounding it in numbers you already have. Know the growth rate of every dependency you lean on, upstream and down. Overpass had a capacity I never measured until I blew past it. Your database, your queue, your third-party APIs, your payment processor, each one carries a ceiling and a roadmap, and some grow with you while others stay flat and turn into the thing that breaks first. Then look at the business: customer growth quarter over quarter, the projected curve your sales team is promising the board, the launch that doubles traffic in a week. Those numbers tell you how much long-term weight the decision can bear and how soon it starts to matter. A system serving a flat user base for two years earns different decisions than one growing thirty percent a quarter, and you can’t tell which one you’re building from the code.
Perfect is the enemy of good here, and good keeps moving. Build for what good looks like six months out, and twelve, not for some steady state you picture at infinity. The catch is the cone of uncertainty. The further out you project, the wider the band of plausible futures, so a six-month bet rests on firmer ground than a two-year one. Design hard for the near horizon where your growth numbers hold up, keep your options open for the far one where they don’t, and don’t commit to a future you can’t see clearly yet.
You make that choice on purpose, knowing what you trade away, instead of letting it happen by accident because the AI defaulted to a pattern that looked reasonable in the moment.
What AI gets right, and why that tells you what it gets wrong
Hand AI a clear, bounded problem and it produces good solutions fast. Ask it to implement a sliding-window rate limiter, give it your data model and read patterns, and you get something solid. Ask it to add idempotency keys to a payment endpoint with your schema in hand, and it implements them correctly. Ask it for retry logic with exponential backoff and jitter, the thing most of us look up every single time, and working code shows up in minutes.
AI also carries institutional memory about patterns. It has absorbed thousands of well-designed systems and knows how they fit together. Ask for a caching layer on a read-heavy API and it reaches for cache-aside. Ask for a background job queue and it produces something that handles failure, retries, and dead letters better than a lot of first drafts a junior engineer would write.
AI is a fast, well-read implementer of decisions you’ve already made. When you know what you want, the algorithm, the architecture, the tradeoff you already resolved, AI collapses the time between decision and working code. That part is real. The gap shows up earlier than that, in the space where the decision hasn’t been made, or where you made the wrong one and don’t know it yet.
What AI builds when you haven’t decided yet
Ask an AI to design a checkout system, and it produces something that works. Products table, orders table, line items. A checkout endpoint that reads inventory, creates the order, and decrements stock. Clean code, reasonable structure, the kind of thing that passes review. Then two users try to buy the last unit in the same millisecond, and you sell it to both of them.
The AI didn’t write bad code. Nobody told it the race condition was the hard part.
This is the failure I see most in AI-generated systems. The code is a correct implementation of what you described. The thing you needed was different, and it included constraints you never put into words: consistency requirements, failure modes, concurrency characteristics, the parts that live in your head instead of in the prompt.
Synchronous assumptions are the most common version. AI reaches for direct calls because that’s the simplest thing that works and what most examples show. Your order service calls your inventory service calls your shipping service in sequence, and the moment one of them slows down, the whole chain slows with it. The shipping service goes down and checkout fails completely, even though shipping was the optional part. You’ve built a distributed monolith without meaning to. Function calls where you wanted a queue, synchronous hops where you wanted a circuit breaker. Deciding to introduce async messaging, to decouple those services with an event queue, has to come from someone who knows where synchronous coupling is going to hurt. Nobody makes that decision in code. It’s a product call about which failures you can live with.
Schema decisions compound the same way. AI normalizes by default, and normalized schemas are clean, usually right, occasionally terrible for reads. Build a dashboard that joins seven tables to produce one row and you should have thought about read models or denormalized views first. The AI doesn’t know whether your workload reads more than it writes, or which queries run a million times a day against which run once a week. It builds the clean schema, and a year later, you’re burning engineering cycles on query optimization for a problem that was architectural from the start.
Where distributed systems get honest
Transactions are easy inside one database. The engine hands you ACID properties, atomicity, consistency, isolation, durability, for free. You open a transaction, do the work, commit or roll back. Something fails, the database cleans up. Engineers learn this early and it turns intuitive.
Across services, you get none of that for free. You build it yourself or you choose patterns that work without it.
Eric Brewer named the core tension back in 2000. In a distributed system, when a network partition hits and some nodes can’t talk to each other, you choose between consistency and availability. Refuse to serve requests until everything is back in sync, or serve them knowing different nodes may briefly disagree about the state of the world. Both cost you something. Neither is wrong on its own. The right answer depends on what breaks when you get it wrong.
An order that processes twice is worse than an order that fails cleanly. A like count that sits a few percent stale for a minute is fine. Different products, different answers to the same question, and the AI can’t know which one you’re building unless you told it. Even then, the AI isn’t the one explaining to a customer why the card got charged twice.
The saga pattern is what most teams reach for when an operation has to span services without a global transaction. Instead of one distributed transaction locking everything at once, you break the work into a sequence of local transactions, each publishing an event when it finishes. A step fails and you run compensating transactions backward through the chain. Place order, reserve inventory, initiate shipping. Shipping fails, so you cancel the shipping request, release the inventory, cancel the order. Each step is durable on its own. No global lock. The cost is the window where your system sits in an intermediate state, the order placed but shipping not started, and you decide whether that window is acceptable. For most commerce, it is. For a financial transfer where the debit hit one account and the credit to another hasn’t completed, you think a lot harder about how long that window stays open and what you do if it never closes.
AI implements the saga pattern correctly once you specify it. It can’t tell you whether you need it in the first place.
Idempotency trips up more payment systems than any other pattern, and AI almost never reaches for it unprompted. An idempotent operation produces the same outcome no matter how many times you repeat it after the first. A payment request times out, you don’t know whether it went through, and you need to retry without risk. Idempotency keys give you that. The client generates a unique ID for each intended operation, the server stores it and uses it to deduplicate retries, and you retry freely. Stripe’s API requires them for payment creation. Most internal payment APIs don’t have them. Nobody asked for them, the failure is rare enough to stay invisible in local testing, and the AI built the happy path while nobody asked about the rest.
Caching as a worked example
Every engineer knows caching makes reads faster. Fewer of us sit with what it costs, and the costs are real enough to matter.
A cache opens a consistency gap. The data in it can go stale. Memory pressure evicts entries on its own schedule, and when a spike arrives right after an eviction, the database eats the sudden load. Cache invalidation, keeping the cache honest when the underlying data changes, is one of the genuinely hard operational problems in production. The old joke about it being one of the two hard problems in computer science has run for thirty years because it keeps being true.
Writes interact with a cache in three main patterns, and the tradeoffs differ in ways that matter. Cache-aside is the most common. The application checks the cache on a read, falls through to the database on a miss, and writes the result back. Simple, fine for most cases, though the cache and database can disagree for a moment if you’re careless about invalidation. Write-through updates cache and database together and keeps them in sync, at the cost of slower writes. Write-behind, sometimes called write-back, updates the cache immediately and flushes to the database later. Fast writes, but a cache that dies before the flush takes your data with it.
AI implements any of these correctly once you name the one you want. It won’t ask which one fits your system. That answer needs your read/write ratio, your tolerance for stale data, your tolerance for data loss, and how much you trust your cache infrastructure. None of it reaches the prompt unless you put it there.
What engineering looks like now
For most of computing history, implementation was the bottleneck. Ideas were cheap. Turning them into working software was slow. Senior engineers earned their reputations partly by implementing complex things faster and with fewer mistakes.
That has shifted. Code generation runs fast enough now that implementation isn’t the constraint it was. The constraint moved upstream, to the decisions that determine what gets built at all.
I feel this every week. The hours that used to go into writing the code now go into the spec, the acceptance criteria, the requirements pinned down before anyone generates a line. A vague requirement used to cost me a slow mistake I built by hand. Now it costs me a fast one the AI builds with full confidence, code that looks finished and solves the wrong problem. The slow part used to give me time to notice, and that delay is gone now.
Most engineering organizations haven’t fully absorbed this yet. Engineers haven’t lost importance. The weight inside the job moved toward the decisions that come before the code. What used to happen in a text editor now happens partly through AI. The architecture decision, the data model, the consistency boundary, the failure-mode analysis, all of it moved closer to the front, and its cost went up relative to everything around it.
When you can build an idea in hours instead of weeks, building the wrong idea becomes the dominant expense. Getting the architecture wrong and spending three months untangling it, when AI could have built the right thing just as fast, wastes more than it used to. Good architectural calls made early compound into systems that stay easy to extend for years. Bad ones compound too, into friction that shows up on every change you try to make.
The engineers who get the most out of AI are the ones who make the upstream calls well, not the ones who generate the most code. Someone who can hold two contradictory requirements, work through the tradeoffs, and specify the approach precisely enough for AI to implement it correctly is worth more than someone who prompts more fluently. Both skills matter, but the ratio between them has changed.
Supervision is not the same as approval
There’s a version of “human in the loop” that means watching generated code scroll past and clicking approve. That isn’t supervision. Real supervision means understanding what the code assumes and what it’ll do under conditions nobody modeled.
For system design, read the happy path and then ask what happens when things go wrong. What happens when this database call is slow? When this message arrives twice? When the downstream service is gone for five minutes? If the honest answers involve silent data loss or state you can’t recover, you have a design problem no code review will catch, because it only shows up under specific production conditions.
It also means hunting for implicit consistency assumptions. Every sequence that reads data, decides something, then writes a result is quietly assuming nothing changed in between. In a concurrent system that assumption is often wrong. Find those read-modify-write sequences in AI-generated code and ask whether each one needs a lock, an optimistic concurrency check, or some other guard against concurrent access. The AI won’t add them unless you ask, because they complicate the happy path, and it has no idea how much concurrency your system will really see.
AI gives you correct code. Turning that into a correct system takes judgment, and that judgment is yours. AI fills in the implementation once you’ve made the call. It can’t make the call for you, and it won’t tell you when you’ve skipped it.
Production is the only real test
Here’s what experience teaches slowly and what’s hard to learn any other way. You don’t know what your system will do until it’s in production.
You can model it, load test it, review every line. You’ll still get surprised. Production traffic has a shape that staging never fully copies. Users do things you didn’t predict. Data arrives in shapes your schema didn’t plan for. The failure that takes the service down at 3am is usually one you understood in theory and never prioritized, because you’d never actually watched it happen.
AI can’t help here, because AI reasons about code, not about how real systems behave under real load. It can confirm a query is correct. It can’t tell you that query will crawl to 45 seconds once the table passes 100 million rows. It writes retry logic with the right backoff and can’t warn you that retrying on recovery creates a thundering herd without jitter, or that your downstream service comes back faster than your retry interval, so the herd arrives every single time.
So, practically: the faster you build, the harder you watch. In systems built with AI, observability earns its place as more than a nice-to-have. It closes the loop, telling you whether the architecture you chose before building was right. Without latency percentiles, error rates, cache hit ratios, queue depths, and slow query logs in front of you, you’re flying on assumptions the AI helped you write and can’t help you check. Performance signals aren’t the only ones worth wiring in early. The plain usage metrics, the taps, the opens, how far people get through a flow, tell you whether a feature is worth keeping, and they only help if they’re already running before you think to look.
The engineers who do this well stay close to production. They read what the system is actually doing, connect a strange spike in the metrics to a decision made six weeks earlier in design, and treat an incident as both a thing to fix and a signal that some assumption was wrong.
The Overpass timeout was exactly that kind of signal. The code Claude wrote wasn’t wrong. The design it sat inside didn’t survive real usage, and moving the call off the critical path cleared the load. The same change told me almost nobody used the feature. People walking a tour open Google Maps or Apple Maps without being told to, so the points of interest I pulled from Overpass duplicated something every phone already does well. I cut the feature. The app ships audio tours with clear directions now and trusts the maps people already have open. That was the decision the whole incident had been pointing at, and it took me longer to make than the load fix did, and none of it came down to code.
By Joshua McDonald on June 9, 2026.
Exported from Medium on August 26, 2026.
Reader discussion