It is a truth universally acknowledged that an airline in possession of an airplane must be in want of engines to make it go. Yet, somewhat surprisingly, they don’t really buy engines.
Rolls-Royce were the notable innovator here in selling not an engine, but instead what they call power by the hour:1 airlines pay a price for the miles actually flown by the engine.
There are a lot of operating decisions in owning and maintaining an aircraft engine: for example, if you have to take the engine off the wing for repairs it means a lot of downtime for the plane, during which it isn’t making any money. Airlines used to make the call of when to do proactive maintenance vs when to do more extensive refits based on their experience within their fleet. But Rolls-Royce had a view across all their customers, and could do a much better job of predicting maintenance and repair timing, getting the right parts to the right places, and generally keeping the engine in service.
They converted that operational expertise into a product, which just happens to be delivered via a massive turbofan. This pricing model was appealing to the airlines since it aligned with their incentives, and their risks. Under the old model engine makers got paid for spare parts and shop visits, so failures were a kind of revenue. But once they were paid per flying hour then downtime became their cost too.
In ML infrastructure, those flying engine miles are generally referred to as goodput. A Google Cloud blog post from a couple of years ago has a good set of definitions:
Runtime Goodput measures the time spent to make forward progress as a fraction of time when all training resources are available. Maximizing runtime requires careful engineering considerations. […]
Program Goodput measures the fraction of peak hardware performance that the training job can extract. Program Goodput is also referred to as Model Flop Utilization or effective model flop utilization, i.e., the model training throughput as a fraction of peak throughput of the system. Program Goodput depends on factors such as efficient compute communication overlaps and careful distribution strategies to scale efficiently to the desired number of accelerators.
To give one concrete example if you checkpoint every N steps then get a failure at step N-1, the prior steps are lost work and don’t contribute to goodput. A surprisingly large number of things can impact it; Llama 3.1 had a section talking about the reliability pain during pre-training:
During a 54-day snapshot period of pre-training, we experienced a total of 466 job interruptions. Of these, 47 were planned interruptions due to automated maintenance operations such as firmware upgrades or operator initiated operations like configuration or dataset updates.
Improving goodput is a horrendously complex project spanning hardware, software, cluster scheduling, kernel work, profiling and resource management. It’s often the preserve of dedicated ML infrastructure teams2, but not everyone has such a team available.
Neoclouds sprang up to provide pods of accelerators, but the actual goodput available varied enormously. The GPU provider gets paid for the GPUs you are renting, regardless of whether they are making useful progress or sitting in a NCCL timeout.
SemiAnalysis’s ClusterMAX rating gave a (controversial!) view of the varying operational capabilities of the neoclouds, measuring goodput-impacting things like spare GPU capacity, failover and repair times, launch overhead and so on. This let purchasers at least compare reliability, but still fundamentally they were charged for the GPU-hour.
What’s emerged now is a higher level of the stack, notably Thinking Machines’ Tinker, and Prime Intellect’s Lab products.
Both teams are full of folks used to extracting top-tier goodput, and much like Rolls-Royce they are both selling that operational expertise and have ability to improve it by working with a wide range of customers. Rather than power-by-the-hour they sell goodput-by-the-step: you pay for meaningful progress, not for idle clusters.
Given the capabilities of open models, like the (open this month) 2.8T Kimi K3, or Thinky’s own 980B Inkling, there is an opportunity to RL a strong model on a bunch of domain-specific knowledge or tasks only you have access to.
Bridgewater, the HR experiment with algorithmic hedge-fund attached, published a case study with Thinking Machines from their AIA Labs group, where they attempted to train Qwen on certain financial analysis work:
Frontier models we tested on struggle with relatively simple financial tasks, and model advances don’t improve performance much. In contrast, we’ve shown that high-quality proprietary datasets labeled by expert investors and used for fine-tuning produce custom models that exceed frontier performance on our tasks.
This is sometimes referred to as RL-as-a-service, and there is legitimate skepticism of whether that is a good business. But really, this is frontier-scale, reliable training infrastructure as-a-service
You can see this in the Bridgewater post: a small hedge-fund research team did a multi-iteration RL recipe search at 235B scale, and ended up with GRPO, CISPO loss with asymmetric clipping, interleaved batching and on-policy distillation, all ablated. That kind of experimental capability at scale was largely inaccessible outside perhaps 20 firms a year or two ago. Now, it’s doable with an API call.
- Technically this term is much older, dating back to the 1960s with Bristol Siddeley, which is now a subsidiary of Rolls-Royce. But in practical terms this became a commercial airline thing in the late 90s with TotalCare: GE and CFM, the other big engine makers, followed along ↩︎
- Disclosure, several of which I have had the pleasure of working in at Meta! ↩︎