Flapping Airplanes just raised a billion dollars, in part through demonstrating a megakernel framework that could do efficient forward and backwards passes while (for some reason) playing Doom1. Elsewhere, Hyperstition have what they call a Hyperkernel, which is generating high-performance kernels for all sorts of models. Turning a model into something faster is exactly what ML compilers have traditionally done, and all this activity begs the question of whether we need ML compilers at all any more, or whether we can let agents optimize directly?
The folks working on TVM have been asking this, too:
If agents can handle the low-level details themselves, should we still put a compiler between them and the hardware?
We believe the answer is yes, but for a different reason. Instead of requiring agents to reinvent IR and compiler infrastructure at every iteration of kernel development, a minimal, stable ML compiler foundation can provide shared abstractions and reusable tools for the entire community while leaving agents free to explore their own optimization strategies.
The core of their argument is that kernel optimization is a search and iteration process. Tokens and tool calls cost money and time. So, you want to optimize that search, avoid getting stuck at local optima, and offer ways to verify the generated solutions are correct. They build checkers for things like memory access and synchronization and they formalized their IR in Lean to enable cheap verification:
In this future, we may no longer need a compiler that actually compiles anything, but we will still need compiler infrastructure to understand and verify programs. A shared IR can serve as the foundation for analysis, verification, and new tools built by agents.
Together’s new Together Compiler (TKCC) builds optimized inference kernels. But, they reach quite a different conclusion around intermediate representations:
Traditional IRs have limited expressivity. We hypothesized that IRs could help agents understand complex inference codebases, which contain multiple codepaths and kernel launch mechanisms. When directly provided with the source code, we do observe that agents spend ≈ 20% of their time reading engine files to understand the relevant code paths and kernel calls within the engine code. However, we also find that unrestricted source code access unlocks a higher performance ceiling. Even our heavily endowed augmented Inductor IR was not expressive enough to permit certain optimizations
They still focus on the needs of verification and cutting down the search space, though. Much of what they built was an empirical measurement framework, with different tiers for executing kernels that normed out variation in forward passes, and offered hot-swapping for fast iteration. They also cut down on search by developing a custom cost model for parallelisms that culls options analytically, before testing in their tiered setup:
We propose a cost model that ranks the top-k parallelism configurations using analytical estimates and tiered measurements, ranging from theoretical hardware capabilities to measured compute, memory, and communication performance. This enables TKCC to identify promising configurations before investing in expensive agentic optimization
So its not so much compilers going away as it is them being decomposed. A lot of traditional compiler techniques become less useful in a world where inspection and investigation is a cheaper, and a lot more automated.
At the Triton conference this year, Neil Dhar from OpenAI2 talked about how they have been using Codex to do large scale code-mods, with the aim of removing heuristics and implicit semantics from the Triton compiler. Those kind of techniques are great for getting improvements across a wide range of code, but they can make the code unpredictable: a small change can have an outsized effect on the result because it triggers different compiler passes.
One example Neil used was removing automatic FMA. Triton had pattern-matched a multiply followed by an addition, and replaced it a single fused multiply-add. But! That could change the numerics of the kernel, due FMA dropping intermediate rounding. They removed this pass and manually inserted FMA calls in the source of all the kernels that used it3. This makes the search landscape smoother, and easier to navigate (though possibly larger)
TVM’s TIRx is a bet that starting from fresh will allow them to build a more search-and-verification friendly compiler stack: they deliberately don’t apply heuristic-based optimizations, but do offer many extension points so agents can iterate new capabilities on top. OpenAI’s Gluon and Meta’s TLX are both directionally similar bets: they add low-level extension points while reinforcing the core of an existing compiler. Together are suggesting that providing the right tools is more important than betting on a single toolchain to guide search and verification.
Compilers users to make things faster, but now they need to make the search faster, which isn’t quite the same thing. I would be hesitant on pronouncing a winner though, this space co-evolves with the agents themselves. In one particularly clear example Together had a note about some of the things that absolutely didn’t work in helping their agents: it read pretty much exactly like a list of agentic best-practices, circa six months ago4.
- Of course, because every idea is now a prompt away, someone immediately put up an open source version, so feel free to run Doom on your GB300. ↩︎
- And, full disclosure, formerly of the Triton team at Meta, where I currently work! ↩︎
- I mean, ish. OpenAI might have automatically fixed all their kernels that depended on this behavior, but they didn’t fix everyone else’s! There were a few hurried code searches going on as Neil revealed the change had already landed upstream. ↩︎
- Providing a knowledge hub lead to 25-31% lower speedup, given the same optimization time. Seeding with expert-written strategies lost to literally prompting “make this forward pass as fast as possible”, which is a great ablation. Providing human-engineered tools for research and logging impeded throughput: agents spent up to 43% of their tool calls on understanding the systems rather than just getting on with optimizing. ↩︎