Google's HEIR compiler runs AI inference on encrypted data

Standard encryption protects data at rest and in transit, then usually gives up at compute time. Homomorphic encryption keeps the ciphertext opaque while a server still runs math on it. On August 14, 2026, Google staff engineer Jeremy Kun showcased HEIR, the Homomorphic Encryption Intermediate Representation compiler, as part of Google's private computing toolkit. HEIR is open source and aims to convert pretrained models that expect plaintext into pipelines that accept encrypted inputs so the server never sees the underlying features. Demos cover private recommendations, credit-card fraud detection, encrypted traffic anomaly detection with Kitsune, and hotword spotting. That is a privacy story for AI teams that cannot ship raw user vectors to a vendor GPU. It is also still a cost story. Homomorphic encryption remains expensive. Google's claim is that compilers and accelerators are finally making the trade practical. Whether that claim survives outside curated demos is the part I care about.

What is HEIR?

HEIR is a compiler toolchain and research platform for fully homomorphic encryption, maintained with Google involvement and documented at heir.dev. Built on MLIR-style infrastructure, it sits between high-level ML programs and FHE backends such as OpenFHE, Lattigo, and related libraries. The product vision is blunt. Non-cryptographers should be able to compile a model toward encrypted inference instead of hand-rewriting circuits. That ambition has floated around FHE research for years. The difference now is packaging: docs, demos, partner hardware talk, and a public ML design path that treats neural nets as a first-class target rather than a one-off paper artifact.

Google's August post frames HEIR inside a longer private-computing push that includes differential privacy, private set membership, and related tools. Hardware partners named in the post include Belfort, Niobium, Cornami, and Optalysys. Academic collaborators span multiple universities. Four peer-reviewed papers built on HEIR are cited as evidence the stack is already a research substrate, not only a blog demo. That mix of academia, startups, and Google engineering is how compiler projects either become shared infrastructure or stay conference folklore.

Encrypted inference does not magically shrink model IP risk or latency to zero. It changes who can see the input features while a prediction runs. The model owner may still learn something from query patterns or result shapes depending on the deployment. Side channels and traffic analysis do not vanish because the feature vector is ciphertext. HEIR is a serious step on the confidentiality of inputs, not a full private-AI religion.

For skeptical buyers, the right first question is not "is FHE cool?" It is "which workload cannot leave my tenancy even as ciphertext features?" Fraud scoring, cross-org recommendations, and medical risk models often fail ordinary SaaS review because plaintext features are the blocker. Those are the places HEIR-style compilation might earn its overhead. Chatbots that summarize public docs are not.

Compiler quality matters as much as crypto theory here. If HEIR lowers the expertise bar, more teams will attempt encrypted inference without a resident cryptographer. That is progress and risk at once. Misconfigured parameters, wrong scheme choices, or silent accuracy cliffs will show up as "the private model is dumb" tickets unless ML and security share an on-call path.

The demos that make the abstract concrete

Google published four single-threaded CPU demos compiled with HEIR. A deep learning recommendation model, joint work with Belfort Labs, LG, and NYU, shows private content ranking. A credit-card fraud detector with Niobium and hardshell.ai shows the finance angle. Kitsune-based intrusion detection on encrypted traffic shows security telemetry that a provider can score without reading packet contents. A hotword detector with Belfort Labs points at always-listening assistants that need wake-word math without shipping raw audio features in the clear.

Those examples matter because they map to regulated and consumer-sensitive workloads people already run. Fraud, recommendations, network detection, and voice triggers are where plaintext inference creates the loudest privacy objections. They are also workloads with known model sizes and latency envelopes, which makes them better compiler targets than "run Llama forever under FHE" fantasies that still do not belong in a serious roadmap.

I am not claiming FHE is free. Google explicitly says overhead remains nontrivial while arguing cost is falling. Procurement should model latency and dollars, not only threat models. A fraud check that returns in hundreds of milliseconds under plaintext may become a batch job under HEIR until accelerators arrive. That can still be a win for overnight risk scoring. It is a loss for interactive checkout if nobody budgets the slowdown.

Operator caveats stack quickly. Key management for FHE is its own product. Client-side encryption of features means your mobile or edge app becomes part of the trust boundary. Model accuracy under the arithmetic constraints of FHE schemes can diverge from the plaintext baseline. Anyone who pilots HEIR without a plaintext shadow path for comparison is buying a demo, not a migration plan.

The hardware partner list is the other half of the practicality argument. Belfort, Niobium, Cornami, and Optalysys show up because CPU-only FHE is still a hard sell for product teams. Until accelerator pricing and availability are boring, HEIR remains closer to a serious R&D platform than a drop-in replacement for your current inference fleet. Plan pilots accordingly.

Where this fits next to ordinary private AI

Teams already choose among on-device models, confidential VMs, and contractual DPAs. HEIR adds a cryptographic path when you want a remote model to compute without learning the input. That is attractive for multi-party finance and healthcare settings where sharing plaintext features is the blocker. It is less attractive when your real problem is model exfiltration, prompt injection, or a vendor that already sees plaintext under a BAA you accepted last year.

Open-source availability is the adoption lever. If only Google can compile these graphs, the industry shrugs. If HEIR becomes shared infrastructure, hardware startups and banks can meet in the middle. Watch whether non-Google contributors land nontrivial backend and frontend work, and whether the latency numbers in partner accelerator posts match independent reproductions. Blog latency on a quiet CPU thread is not a capacity plan.

Compare the pitch carefully against confidential computing. TEEs protect a running environment if you trust the attestation story. FHE protects inputs even when you do not trust the server's operators to see features. Different threat models. Different cost curves. Mixing the vocabulary in a slide deck is how security reviews get confused on purpose.

A sane near-term portfolio for privacy-sensitive AI looks layered: on-device for the easy cases, TEEs or VPC isolation for many enterprise SaaS paths, and FHE pilots only where plaintext features are the contractual or regulatory brick wall. HEIR makes that third lane less exotic. It does not make the first two obsolete.

One last buyer test before anyone funds a six-month HEIR pilot: pick one concrete model, set a latency and dollar budget, require a plaintext accuracy delta report, and demand a key-management design that security can live with. If the proposal cannot answer those four items on one page, it is still a research tour, not a product path.

Filing as a privacy-compiler milestone with sober cost footnotes. More from Younes Bekrar in AI as accelerator demos land.

Related reading on Skarvonix: our ai category, the authors directory, and more from Younes Bekrar.

Primary sources and further reading: Google blog on HEIR and private AI, HEIR project site, HEIR ML design docs.