Google DeepMind did something on Tuesday that no major lab had done before: it published complete latency distributions for a production model rather than a median and a p99. The dataset covers 30 days of Gemini 3 Flash traffic across four regions, roughly 90 billion requests, bucketed by prompt length and output length. The curves are public on a Cloud Console page and downloadable as Parquet. The interesting finding is that first-token latency is bimodal, with one cluster around 190 milliseconds and a second around 640, and the split correlates almost entirely with whether a request hit a warm prefix cache. That explains a complaint developers have made for a year without being able to characterize.
Why averages were hiding the problem
A median of 260 milliseconds and a p99 of 900 describes the published data accurately and tells you nothing useful. Developers building chat interfaces experienced something different: responses felt instant most of the time and occasionally felt sluggish in a way that seemed random. Support threads filled with reports that were impossible to reproduce, because reproduction depended on cache state the developer could not observe. Google's own support team had been closing these as working as intended.
The bimodal curve makes the mechanism legible. Gemini's serving stack caches the attention state for prompt prefixes it has seen recently, which is why a long system prompt costs almost nothing on the second request. Cache residency depends on traffic volume for that specific prefix, region, and model version. A low-volume application evicts constantly and lives in the slow mode. A high-volume application stays warm. Two developers running identical code see different performance based entirely on how much traffic they send.
What Google is doing about it
Alongside the data, Google shipped explicit context caching controls out of preview. A developer can now pin a prefix with a declared time to live and pay a storage rate of $1.00 per million tokens per hour, separate from inference. That converts an invisible probabilistic behavior into a line item you can reason about. For an application with a 12,000 token system prompt, pinning costs about $8.60 a month and eliminates the slow mode entirely, which several early users described as the best pricing decision Google has made on this product.
The response headers changed too. Every completion now returns a cache status field with values of hit, partial, or miss, plus the prefix length that matched. That single header does more for debuggability than any dashboard, because it lets a developer correlate slow requests with cache behavior in their own tracing system. OpenTelemetry semantic conventions for generative AI were updated within a day to include it, and Datadog and Honeycomb both shipped parsing support by Tuesday evening.
The story is rarely the launch. It is what breaks, what ships, and who owns the mess at 2 a.m.
The transparency question this raises for everyone else
Publishing distributions invites comparison, which is presumably why nobody had done it. Artificial Analysis and other benchmark sites measure latency from outside and produce numbers that vendors dispute. A vendor publishing its own internal distribution, with methodology, changes the terms of that argument. It also creates pressure: once one lab does it, refusing to do so becomes an answer in itself.
OpenAI has not committed to matching it. Anthropic's developer relations lead said the company is discussing what it can publish, noting that its capacity is provisioned differently across customers, which makes an aggregate distribution potentially misleading. That is a fair technical objection and also a convenient one. AWS, which resells several models through Bedrock, would face an awkward situation if customers could compare its serving performance against the model provider's own endpoints, and it has said nothing.
What the data reveals about capacity
Reading the regional breakdown carefully tells you things Google probably did not intend to advertise. The us-central1 distribution is tighter than europe-west4, suggesting more headroom in Iowa than in the Netherlands. Asia-southeast1 shows a third small cluster around 1.4 seconds during specific hours that maps neatly onto Singapore business hours, which implies capacity constraints during regional peak. None of that is scandalous. It is the kind of operational reality every cloud service has and few disclose.
For architects making placement decisions, this is genuinely actionable. A European company with latency-sensitive workloads now has evidence that routing to us-central1 might beat the nearer region despite the extra network hop, because the queueing difference exceeds the propagation delay. That inverts the usual advice and it is the kind of conclusion that was previously available only to companies large enough to run their own measurement infrastructure.
The broader push toward measurable services
Model APIs have been sold on capability and priced on tokens, with performance treated as a quality that emerges rather than a property that is specified. That worked while everything was experimental. It works poorly for production systems with service level objectives, and enterprise buyers have grown increasingly impatient with vendors who will not commit to numbers.
The direction of travel is clear enough. Latency distributions this quarter, capacity commitments next, and eventually contractual service credits that look like the ones every other cloud service carries. Google moving first here is a competitive play as much as a transparency gesture, aimed squarely at buyers who have found the current generation of AI vendors impossible to hold accountable. Expect the sales decks to reflect that within weeks.
Skarvonix will keep following this beat with reporting grounded in how systems behave outside the launch keynote.
- LLMs
