Fly.io published a postmortem on Tuesday for a July 22 outage that left applications in its Frankfurt region unreachable for six hours and eleven minutes. The trigger was a firmware update to a batch of network cards that caused intermittent packet loss, which the platform's health checking interpreted as host failure, which triggered mass rescheduling, which overwhelmed the orchestration layer that was supposed to perform the rescheduling. The postmortem is unusually detailed, running to about 4,000 words with timestamps and internal Slack excerpts, and it discloses that a scheduler rewrite has been in progress since March specifically because the team knew this failure mode existed.
The failure chain
At 09:14 UTC an automated firmware rollout began on 34 hosts in Frankfurt. The new firmware had passed staging, where the traffic pattern differed. Under production load it dropped roughly 3 percent of packets on one of two bonded interfaces, enough to make health checks flap without making the host obviously dead. Fly's orchestrator, which the company calls flyd, treated flapping hosts as failed and began migrating virtual machines to healthy capacity in the region.
Frankfurt did not have enough spare capacity to absorb 34 hosts worth of workload. Migrations queued, and the queue had no admission control. Each queued migration held a lock on the source machine's state, and lock contention pushed the orchestrator's own API latency past the timeout its control plane used, which caused the control plane to consider orchestrator nodes unhealthy and restart them, dropping the in-flight migration state. The loop repeated four times before an engineer disabled automatic rescheduling manually at 11:02.
Why it took four more hours
Stopping the loop was not the same as recovery. By 11:02 the region's state was inconsistent: some virtual machines existed in two places, some existed nowhere, and the volume attachments that back stateful apps had been partially moved. Fly's team spent three hours reconciling state by hand against the underlying storage layer, because the tooling to do it automatically assumed a consistent starting point. The postmortem describes writing a reconciliation script during the incident, which is exactly the situation every operations team fears.
Customer impact varied by application shape. Stateless applications with instances in multiple regions saw traffic shift and experienced elevated latency rather than an outage. Applications pinned to Frankfurt, which includes a lot of European businesses with data residency requirements, were fully down. Applications using Fly Postgres in a single region lost write availability and, in eleven cases the company identified, required restoration from backup with up to four minutes of data loss. Fly is issuing credits and contacted affected customers individually.
The story is rarely the launch. It is what breaks, what ships, and who owns the mess at 2 a.m.
What the rewrite changes
The replacement scheduler, internally called Corrosion 2 after the gossip system it builds on, changes three things. Rescheduling decisions now pass through an admission controller with a regional budget, so no more than a configured fraction of a region's capacity can be in migration at once. Health checking distinguishes between a host that is unreachable and a host that is degraded, with different responses to each. And the orchestrator's state store moved from a design where the control plane could restart mid-operation to one where operations are journaled and resumable.
None of that is novel. Every large orchestration system learned these lessons, usually the same way. What is notable is Fly publishing that it knew the gap existed and had prioritized the fix below other work. Chief executive Kurt Mackey wrote that the team judged the probability low and the cost of the rewrite high, and that the judgment was wrong. That admission is more useful to readers than any technical detail, because the underlying decision, deferring known reliability work for features, is one every platform team makes constantly.
The firmware rollout problem
The proximate cause deserves its own attention. Fly rolled firmware to 34 hosts simultaneously in a single region, which no mature operations practice would endorse. The postmortem explains that firmware updates were handled by a separate tool from application deployments, that this tool had no concept of blast radius, and that it had been used this way perhaps two hundred times without incident. That is a recognizable pattern: a process survives because the failure it enables is rare, until it is not.
The fix is to route firmware through the same progressive rollout system that handles everything else, with a one host canary, a thirty minute soak, and automatic rollback on health regression. Fly says that work is done and shipped. The broader lesson, which the postmortem states directly, is that any change mechanism touching production should share the same safety machinery, and that maintaining a second path with weaker guarantees is a bet you will eventually lose.
How this affects Fly's position
Fly has built its business on developer experience and on running applications close to users, competing against a set of platforms that includes Railway, Render, and increasingly Cloudflare Containers. Reliability has been its persistent weakness, with notable incidents in 2022, 2023, and 2024, and the company has been candid about it in a way that has earned goodwill and has not fully offset the operational reality.
The commercial question is whether customers who moved production workloads to Fly during its 2025 growth push stay after this. Several engineering leaders we spoke to said the postmortem's quality actually helps, because it demonstrates the team understands the system, and that they would rather run on a platform that explains its failures than one that posts a two-sentence status update. Two others said they are moving stateful workloads elsewhere and keeping stateless ones. That split is probably the honest read on where Fly sits.
Skarvonix will keep following this beat with reporting grounded in how systems behave outside the launch keynote.
- Edge Computing
