Fly.io's postmortem on the June outage names a smaller villain than the symptoms suggested at the time: one overloaded shard of tkdb, Fly's internal distributed key-value store, sitting on the synchronous path of the Machines API. When that shard slowed under load, create/start/stop calls that touched it slowed or failed with it. Machines already running mostly kept running. The rest of the fleet was fine. From the outside it still felt like the platform's control plane had fallen over, which is the gap the writeup is really about. I read a lot of status pages that hide behind "elevated error rates." This one eventually pointed at a shard, which is rarer than it should be.
Synchronous tkdb, no degraded mode
tkdb tracks control-plane state, which Machines exist, what they're doing, which physical host they're on, and the metadata the Machines API needs for almost any fleet question. The API was built to read and write tkdb as part of handling each request. When latency is normal, that keeps the API's view consistent without a separate cache or eventual-consistency story to reason about. Reasonable.
A synchronous dependency also means there's no degraded mode unless you build one. Healthy tkdb: fast, correct API. Slow shard: every call waiting on that shard gets slow too, with no path that answers from slightly stale data instead of blocking. The postmortem describes degradation, not a clean tkdb blackout, one shard past the point where response times climbed hard, and everything queued behind it suffered.
Consistency-first control planes love this design until the day the store hiccups. I've built versions of it. You tell yourself you'll add caching later. Later arrives as a customer screenshot of a deploy spinner.
Fly's Machines API is the lever people pull for create, start, stop, and scale. Putting tkdb on that path means a data-plane that looks healthy can still feel dead to anyone trying to change it. Running workloads and mutable control plane are different products that share a brand name during an incident.
The postmortem's value is partly that it refuses to blur those two. Plenty of vendors would have written "platform degradation" and moved on. Naming a synchronous tkdb dependency gives other engineers something to map onto their own diagrams.
One shard, lottery-ticket failures
Load concentrated on a single shard. The Machines API doesn't shard its own request handling independently of tkdb's layout, so calls whose data lived on the hot shard queued and timed out while calls on healthy shards kept working. From a customer's seat that looked random. Some orgs and Machines failed. Neighbors did not. People experience that as "Fly is down," not "shard 7 is sad."
Actual blast radius: one shard. Perceived blast radius: the control plane. You see this pattern whenever a horizontally sharded store is consumed synchronously by a single API layer that can't isolate or route around the unhealthy piece. Sharding bought throughput. It did not, by itself, buy resilience.
Fly sits under a lot of developer platforms and startups that treat it more like a primitive than a traditional cloud account. When the Machines API wobbles, deploys and scaling actions fail downstream, often before those teams can tell whether the bug is theirs. That amplification is why June generated more complaint volume than "one shard" sounds like on a status page, and why the postmortem spends as much energy on detection and communication as on the root cause.
Partial failure is also why Twitter threads turned chaotic. Half the replies said nothing was wrong. Half said nothing worked. Both could be true if their Machines hashed to different shards. Incidents that fail a dice roll per customer are the ones that burn trust fastest, because screenshots contradict each other and status copy struggles to sound coherent.
What Fly says it will change
The plan, as written, is to shrink how much of the request path blocks on a single tkdb read or write before the API can respond. Cache or serve some control-plane reads from a source that can tolerate a degraded shard. Add circuit-breaking so a slow shard produces bounded, visible failures for the Machines on it instead of queue growth that poisons unrelated requests waiting behind them.
They also want per-shard observability that surfaces "this shard is unhealthy" at the start of an incident instead of a foggy platform-wide slowdown that burns half the outage window on localization. How long an incident lasts is often decided by how fast you name the broken thing, not only by how clever the eventual fix is.
None of those items require a greenfield rewrite of tkdb. They require the Machines API to stop treating every tkdb round trip as equally mandatory and equally unbounded. Circuit breakers and stale-read paths are unglamorous. They are also how sharded systems fail small instead of failing theatrical.
I'll be watching whether future incidents get a shard ID in the first status update. If the observability work lands, the public narrative should get boring faster, which is the actual goal.
Caching control-plane reads raises the usual consistency questions, is a slightly stale Machine state better than a timed-out create? For many operator actions, yes. For some, no. The postmortem's direction implies Fly is willing to draw that line explicitly instead of pretending every read needs the hot shard in real time.
Calibration for people who run on Fly
If you treat Fly as critical infrastructure, fold a boring distinction into the runbook: control-plane operations, creating or scaling Machines, are more exposed to this failure class than already-running workloads. In June, running Machines were not the casualty. Deploy buttons and scale events were.
Horizontal sharding without timeouts, fallbacks, and circuit breakers at the caller still leaves you coupled to your worst shard. Fly's described fixes are mostly about making the Machines API a better citizen of a sharded backend, not about rewriting tkdb from scratch.
I'll take a postmortem that names the store, the API, and the engineering follow-ups over a fog machine status update any day. Whether the fixes hold is a question for the next bad afternoon. Publishing the actual shape of the failure is already more than a lot of infrastructure vendors bother with.
If your own product's deploy path is a synchronous chain through someone else's control plane, June is a free tabletop exercise. Ask what your users see when create/start/stop fails while existing instances stay up. If the answer is "they think the whole product is dead," you and Fly share a UX problem, even when the data plane is fine.
I'm also curious whether Fly's circuit-breaking work will show up as uglier, more precise error messages for the unlucky shard's customers. Bounded failure feels worse in the moment than a mysterious hang and better in the postmortem. Customers can work with a clear "this Machine's control plane is unavailable." They cannot work with a spinner that might be their bug.
- Edge Computing
