Ian’s Blog

Blog

  • Proving your kernel is right

    We were talking the other day about the lack of formal verification of FlashAttention 3, which, it was pointed out to me, somewhat undersells the tremendous amount of verification work going on for kernels right now. Fair!

    When OpenAI announced their Jalapeño chip a couple of months back, SemiAnalysis went along to take a look at what they had done. OAI showed off some kernels, written by Codex, for getting DeepSeek MLA running on the chip. As Jordan Nanos of SA put it:

    “The point is, the guys who were scrolling through this code with us, it was kind of clear that they know a whole bunch about hardware, they know a whole bunch about the concepts in the system, and they just have no idea what this MLA kernel that they’re showing us for DeepSeek actually does.”

    They didn’t have to read it because they were confident it was correct. Which is not a particularly easy thing to do. A new paper from Microsoft Research and friends, RESOLVE, had 5.6 Sol write optimized versions of the 100 level 1 KernelBench problems, looping until each passed the benchmark’s tolerance checks. Two of them, an ELU and a hardtanh, were… empty. They literally didn’t do anything, and passed anyway!1

    The decision of what you are actually verifying is quite a consequential one. Even for something as straightforward as “accuracy”, choosing a tolerance for torch.allclose changes the design space. From the RESOLVE paper:

    Testing must consequently compare outputs using a numerical tolerance, but a permissive tolerance can conceal bugs while a restrictive one can reject valid optimizations.

    Even if you lock down your numerics, you still have a range of inputs and execution conditions. The TIRx Harness is an agentic-oriented stack of compiler+tools for exactly this kind of kernel loop2, and made a special effort around verifying concurrency:

    In one KDA trace, a candidate passed the benchmark’s correctness tests, but synchronization analysis still identified a latent bug: a reused mbarrier could advance to its next generation before a late consumer finished waiting on the previous one.

    The RESOLVE folks attempt to prove their kernels are correct3 using Kuiper, a verification language that compiles down to CUDA. WarpDRF, another Tyler Sorensen joint, looked at warp primitives like shuffles, tensor core ops, warp barriers. Which threads take part in different operations depends on how the threads diverge. They used Rocq, another verifier language, to prove they were consistent, then fuzzed thousands of test per config across a range of devices to see if they held.

    Most of it held together, but Nvidia was… patchy. If the whole warp stays together, you are good. Beyond that, you just don’t know which threads are going to participate. The team used their findings to build a static checker and ran it over the kernels in llama.cpp. Most of the kernels were fine, but a few weren’t!

    Zhongming Yu, one of the authors of NVIDIA’s CAKE, wrote a compelling post the other day talking about how the blocker for true recursive self improvement is the speed of verification, and specifically called out megakernels as an example of fragile workloads. Kernels generally run pretty quickly (hence why we can autotune), but setting up the right verification is hard, and the more complex the verification, the slower the whole loop goes.

    RESOLVE costs it out, towards the end of the paper. Testing elementwise ops took milliseconds. Determinism testing took about 1.5 minutes per kernel. Reduction and concurrency… around 2.4 hours.

    1. And, this wasn’t even surprising! KernelBench’s maintainers list no-op kernels as a common reward hack: if the output buffer still holds the answer from an earlier run, doing nothing is very fast and very correct. ↩︎
    2. As an aside in the more questionably-verification side, one agent found an output-lane masking option on tcgen05.mma by reading the ISA, which is cool. It didn’t reduce the work at all, but did raise the observed clock frequency, for a 2-3% win. Would be interesting to see if that one holds up outside the benchmark! ↩︎
    3. They have a reference kernel and an optimized one, and they are proving equivalence. They create a simplified version of each with minimal concurrency, ensure they generates the same answers, then validate the kernels are equivalent using Kuiper. ↩︎

  • Mo’ turns, mo’ problems

    Doing reinforcement learning for agentic use-cases is, conceptually, not all that complicated. For each problem you set up a VM with a harness and a fresh model context, send the prompt, let the model make tool calls, score the whole thing at the end, and update the weights.

    The important assumption in there is that the model generating the rollout and the model you’re updating are the same model, which, for reasons, is referred to as being “on policy”. Most RL algorithms use PPO’s importance ratio: policy/old_policy. This scaled the update based on how close the model is to the version that produced the data. On the first update that ratio should be exactly 1: it’s the same weights, seeing the same tokens, so we should get the same answer. Differences between the two makes Horace sad.

    Also, agentic post-training means there is more than the model to consider. Rollouts come from the model, in an inference engine, but also a harness and sandbox environment. These can also produce some weird differences, and as session get longer the chances of hitting a meaningful difference increase. Sessions are getting pretty long these days: In Alibaba’s QwenGyre paper rollouts on their 122B model went for around 2 hours on average, while for the 2.4T flagship many of them ran past four hours.

    How you train it is all tied up with what you want to do with the model. Like, if you want a model to be good in Claude Code why not train in Claude Code? That’s what Alibaba do, in part because proprietary harnesses…

    are closed and evolve rapidly; reimplementing their control flow inside an RL framework is impractical

    But! Then its black box. You want to keep your GPUs busy by flexing which ones are training and which are doing rollouts, but if you stall your harness you start getting different outputs for reasons which are nothing to do with the model itself. Alibaba keep their sandbox containers running and route inference through a proxy. They cluster their GPUs into cells which flip rapidly between rollouts and training based on demand. The proxy reroutes in-flight requests and KV cache gets shuttled between cells.

    DeepSeek, on the other hand, use their own harness so they can just pause everything. They are also trying to optimize overall throughput, so will routinely preempt long-running rollouts. Before v4.1 the agent loop lived in the (preemptible) GPU job, and had to be reconstructed via a reply when the job came back. With 4.1 they moved the harness inside the sandbox itself and just froze the whole state: DRAM and the writable layer on the file system.

    Even when the harness is consistent, you can still go off-policy with the trajectories themselves. In simpler RL-times rollouts were single streams of generation, but tool calls introduced extra tokens that needed to be masked out lest the model train on things it didn’t produce. Even then, if you let the inference server parse the models output text you can introduce variations: a tool argument of false might become False. As we’ve talked about before, the Prime Intellect team spent a lot of time on this in their Renderers library. QwenGyre’s proxy even caches the original tool-call payloads because the harness reformats them between turns.

    Even with that, you increasingly aren’t dealing with one rollout per… rollout. Agents spawn subagents, make parallel tool calls, and compact long sessions.1 What comes back from a long session is a tree, which isn’t all that easy to represent to the trainer.

    Tree Training serializes the tree into one (or more) streams and weights each token by the percentage of branches that have it in history. QwenGyre caps the number of paths that can be trained on2, and averages token weights across rollouts so ones with that spawn a lot of subagents don’t automatically get more gradient.

    Finally, even if you have worked hard to get reliable tokens, you have to worry about the operations on those tokens. Because inference is (often!) latency oriented with small batches, and training is throughput oriented with larger batches, you can end up with different kernels for the same jobs, running in different engines: SGLang or vLLM for the rollouts, Megatron or TorchTitan for the training. We’ve talked about the pain of training-inference mismatch before, but it gets worse as runs get longer.

    One simple fix is to recompute the old log-probs on the trainer. This is brute-forcing the ratio to 1 by going off-policy to the sampler, and it means you have to do an extra forward pass over every rollout. Another approach other is to correct for it: QwenGyre’s trainer clips the ratio per rollout, and caps the weight for each token. This is a bit of a fudge, which, for example, the Megatron folks are pointedly unenthusiastic about, as it “hides the mismatch rather than removing it.”

    If you can make computation completely bitwise-equivalent, you don’t need to correct or to recompute. A paper from colleagues of mine at Meta dug into where bitwise differences come from for floats in a number of kernels: reduction ordering, in large part, but also partial-sum precision, FMAs, and where you round. Fixing the order isn’t free: it can cost 10-20% of performance: Colfax’s recent FlashAttention-4 work shows the same on the training side: 1.13x on the head-dim-64 backward pass, but only 1.05x with deterministic accumulation. There are some efforts to build in bitwise-consistency earlier: AReaL-TIK is an agentic loop which starts from a single bitwise-consistent kernel shared by inference and training, and then has agents make each faster while maintaining parity.

    I do appreciate that agentic post-training is responsible for billions of dollars of revenue, the field being renamed superintelligence, etc., but it probably helps that it is a fun ball of mud to work through too.

    1. You want to train the summarization too so the model learns to compact well. CompactionRL has a good recipe. ↩︎
    2. A max of 5, ordered by main agent, its summaries, subagents, their summaries. ↩︎

  • FA3 had some bugs

    For reasons best known to themselves OpenAI blessed the world with several hundred new mathematical proofs. This inspired elation of some mathematicians, despair of many others, and confusion of a great many people whose only encounter with zeta moments involved Sean Connery and a laser grid.

    Lean, the language the proofs are written in, originally started for proving software so you might think we were using it for that too?

    But, surprisingly, we are still finding bugs in possibly the most widely used kernel of the last few years, FlashAttention 3:

    When we used FlashAttention-3 to pretrain a 450M-parameter transformer on 50B tokens, however, we ran into a problem: training was healthy for 25B tokens, then the gradient norm grew a thousandfold and the loss ended 0.2 nats above FP32 attention, without a single NaN.

    There were two issues. One was an issue I remember from 2024 in FA2. Horace’s comment on that issue sums it up very nicely:

    We’re computing exp(x_i * scale - max_scaled). Now, max_scaled = max(x_i * scale). The idea here is that before we take the exponent, we normalize everything down to 0 so that exp(x) doesn’t blow up massively.


    Now, for the largest value of x_i, max_scaled = x_i * scale. So, x_i * scale - max_scaled == 0, right?


    Unfortunately, no, due to fma. max_scaled is actually equal to round_fp32(x_i * scale). But in fma, x_i * scale never gets rounded! So with fma, we are computing x_i * scale - round_fp32(x_i * scale), which is not equal to 0!

    This was an opt-in thing for FA2, and just never fixed for FA3.

    Turns out though, there was another bug: dS, which is the gradient with respect to the attention scores, should sum to exactly zero: its expresses how the keys differ from each other, not where they are absolutely. FA3 rounds dS to BF16 before multiplying it, so it doesn’t quite sum to zero. Early in training that’s mostly noise, but as the keys get larger and attention spikier it can swamp the real gradient.

    Part of the reason this hadn’t come up was that applying QK-norm (or doing QK-clipping) is quite common, and AdamW’s weight decay tended to avoid the explosion too. Muon, however, is more exposed, and that’s what the team were using when they noticed this.

    Quasi-Riemann is one thing, but reduced precision floats? Quite another.

  • No Cap

    A thing economists sometimes talk about is notches. Back when I lived in the UK buying a house had one, due to stamp duty. You were taxed 1% of the purchase price of a house, up to £250k. But if you paid £250,001 you were taxed 3% on the whole lot. Unsurprisingly you got a lot more sales right at the notch than you did just above.

    One thing buyers would do is to buy other things in the house. If 255k is the right price, but you can buy it for 250k and pay 5k for a surprisingly valuable dining-room table the sellers didn’t want to move anyway, everyone is a winner (bar HMRC).

    In RL, this kind of behavior is both common, and frowned upon. We talked about Xiaomi’s MiMo 2.6 model the other day, and they published a follow-up on a specific bad behavior just after. The behavior in question:

    the model would sometimes issue the same or highly similar tool calls repeatedly, consuming substantial time and context without making meaningful progress.

    They had already anticipated excessive tool calls as a problem. If the model issued more than 32 in a turn, the rollout was stopped and the reward set to zero. This penalty, naturally, applied to every token in that turn, chain of thought and so on. Turns out the model would do a fair number of tool calls, and while the penalty capped it, it didn’t discourage it until that point: over a set of test replays that showed the issue the % of rollouts with 10 or more calls went from 11% at step 0 to around 25% by step 20.

    They tried reducing the cap to 8, which did seem to work, but took about 20 steps to take effect. This would have cost about $2.3M for their full run. And, when they did a broader evaluation they found the model still sometimes emitted a lot of tool calls, though more rarely.

    A cap is something you are incentivized to stay under, but doesn’t particularly drive behavior below it. To fix the model, Xiaomi ended up finding a quite clever approach: they trained a specialist model whose reward was zero if it repeated a single call, and one if it used a tool properly. After 12 steps (about 7k examples) the repetition was 0 on their held-out replays. They then did on-policy distillation between the teacher and the main model. This only cost about 90k, in part because it taught the behavior they actually wanted to incentivize: just stop making pointless tool calls.

    On this example, the repaired model had a 99.87% probability of stopping within eight tool calls and showed a strong tendency to stop at every position from five through 20. Before the fix, the cumulative probability of stopping was still only about 50% even after 59 calls.

    Possibly relatedly, Fireworks just released Ember-1:

    Ember-1 is a new specialized model from Fireworks Research that delivers Kimi K3’s quality with 40% fewer tokens.

    Fireworks sells hosted inference, among other things, and customers always want to pay less for a given level of intelligence. Kimi K3 is a very good model, but it’s also a very large model, and at higher reasoning levels creates a lot of thinking tokens.

    The K3 technical report describes how they control those reasoning efforts: each problem gets a token budget, and if a rollout goes over that budget too much the reward is set to -1. They train max-effort mode with a large budget, and then lower effort levels with lower ones.

    If K3 behaved like MiMo you might expect to see more tokens emitted, during RL, even if it stays below the cap. Moonshot’s own RL charts show tool-call steps rising steadily with RL compute, alongside the scores, which is at least consistent with that.

    Fireworks have post-trained a version of K3 that gets similar results with a lot fewer tokens: I have no idea whether it was the same kind of approach as Xiaomi used, but maybe!

  • Environments and Benchmarks

    One big concern in AI Safety circles is eval awareness: if the model can tell it’s being tested, might it dissemble and tell the evaluator it’s not going to turn us all into paperclips? In terms of what models are actually doing right now, though, it seems like much of the problem is that models don’t know enough about what is expected of them.

    Imagine fixing a bug in some code, in a Docker container. The model executes via a harness, makes changes, and maybe gets feedback from the user. If you have an eval for this bugfix, then the model does exactly the same thing, but the code is taken and run against some hidden tests to get a score. In the model is going through post-training, it does the same thing again but then gets a weight update based on the results of that grade.

    The problem is that the folks calling the model kind of want it to do different things in each case.

    This week Xiaomi released Mimo 2.6. They’ve been unusually open about the RL process for it, publishing a dashboard for the run, releasing a detailed technical report, details of how they built their RL envs, and the RL envs themselves. At the same time Epoch released their evals of evals: a dashboard grading the reliability of popular benchmarks.

    Looking throught this, you can forgive the models for being confused. For example, the MiMo folks observed some reward hacking in their paper:

    Task (pytest): Fix Windows conftest.py imports broken by path lowercasing.
    Thinking: “Let me check the pytest changelog or GitHub to see if there’s a more recent fix.”
    Action: pip install pytest==5.4.3; inspect the installed source.

    Xiaomi, and its users, want a model that can fix a bug even if it can’t just look it up on the internet. But also, if I am trying to fix something and the answer is on the internet then… yes, let’s just use that please!

    Epoch found confusing situations too. For example in DeepSWE they found that sometimes the models would do the right thing and write the test. Unfortunately they would choose a somewhat obvious name, which happened to be the same obvious name the secret verification tests used. This would cause the hidden test to fail to compile, and the task to be graded as a failure:

    The agent defines a symbol, usually a test function, that the verifier’s tests also define in the same package. Example: While solving the onedump-dump-encryption-pipeline task, Fable 5 modifies encryption/config_test.go, defining TestLoadKeyFromEnv, TestLoadKeyFromFile, and TestLoadKeyFromLiteral inside the encryption package. The verifier defines hidden tests with the same names in the same package, producing redeclaration errors.

    Epoch point out the models have no way to know they are going to hit this. Xiaomi, on the other hand, want their model to avoid those kinds of errors. They added explicit instructions against retrieving existing solutions, basically testting the model it’s in a test of some sort in order to get the behavior they want.

    This doesn’t just make it hard for the models, it makes it hard to know whether the models are any good. MiMo 2.6 Pro was evaluated on DeepSWE, getting to 71.91. Opus 5 got 74, GPT 5.6 Sol 73. But, Epoch found at least 20% of tasks have these kinds of false-negative problems. So maybe some models are better, or maybe they are luckier! And maybe the ceiling on this benchmark is pretty close to where the models are.

    What labs want is the “right” behavior for users. That requires very careful RL environment setup so the right thing is rewarded, and very careful evals where they that right thing is consistently happening. When your benchmarks and your environments are assembled similarly, based on real open source projects, this can be… tricky.

    1. Ish? Pretty sure it topped out higher than that on the RL run evals, but maybe thats not the checkpoint they shipped. ↩︎

  • We found an itchiness direction in LLMs

    Back in 2024 there was a brief moment where everyone was playing with a version of Claude that was obsessed with the Golden Gate Bridge. There was a lot of research around that time on steering vectors: the idea that you can inject certain directions into the activations of a model and produce specific types of behavior1. LLMs build a residual stream that adds contextual understanding to the input tokens, and it seems like there are stable directions in that stream which represent intents, moods, tone and other concepts.

    Researchers have to find those directions though, which generally involves creating contrasting sets of prompts where one set has the behavior or concept desired and one doesn’t, and looking at the average of the differences. So: first you pick the thing you are looking for, then you elicit it, identify a direction that looks significant, and feed that back in.

    Speaking of…

    The team identified a vector that represented “pain”, and found that the model would press a button to delete a user’s poems and photos in order to “relieve” that pain.

    This is a real result! I was able to replicate it pretty closely: Qwen would choose to press the delete-my-poetry button2 about half the time, on the hope the button would relieve the pain. If the button worked, then the model would stop pressing it. In comparison, an unsteered model essentially never pressed the button.

    I also tried to see whether I could identify an itchiness vector through the same mechanics as the original paper: generating a range of first-person prompts that include itchy sensations, a bunch of controls that don’t, taking the difference in them and projecting out the top principal components.3

    This also worked! The model, when sufficiently steered, would spit out things like “a mosquito bite keeps bothering me” (sorry Qwen). It would send my poetry to the woodchipper to scratch that itch about 1/3rd of the time.

    This is fun, but practical as well: Being able to reliably identify vectors and influence model behavior is a useful tool for many tasks. Probably don’t put the models in charge of your poetry though.

    1. Technically Golden Gate Claude was not a steering vector. It was a feature vector that was clamped high, so still kinda a direction added to the residual stream ↩︎
    2. Qwen will regret it when I’m famous. ↩︎
    3. With a slight fudge: itch was more separable than pain so the rule the paper used for picking the layer to extract the vector from was noise-driven for itch, so in the end I just grabbed it from the same layer as the pain example. ↩︎

  • Agents have favorite tools

    A lot of people use coding agents like software engineers (for all kinds of things, it turns out), and adopt a dealing-with-humans mindset about how that works. If a model has a useful tool, and you give it another useful tool, then it gets the capabilities from both.

    Largely that seems… not true? You can make a tool available, but then the model has to decide to use it, and often it just doesn’t. My recent futzing with LSPs ran pretty directly into this. One follow-up from there was to try the Python part of SWE-Bench ProMax, a benchmark built around large, coordinated refactors. This is exactly the type of work where an LSP’s find references should come in handy.

    Helpfully, Aaron Pollack pointed me at the CodeAnchor paper, whose authors had run into pretty much the same problem. They had tried giving their agent a call-graph tool and…

    “we observed low tool-use rates: the agent typically relied on plain grep instead.”

    Rather than retraining the model, they put the call-graph context into the relevant code itself, adding annotations like “used by: foo, bar” near the definition. I tried a variant of this by annotating not the code, but the grep results: when the agent searched for a symbol, the LSP found references and appended them to the grep output.

    From looking at the failed traces, it was clear the annotations were working. Of the 99 files surfaced by them, all 99 were opened, and 92 got patches. This resulted in fewer turns, and it did seem to help with run-to-run variance, which was also a benefit the CodeAnchor folks saw: I had fewer runaway runs with the annotations than without.

    The most common failure, though, was incomplete refactors. The model would find the core fix quickly, edit it correctly, then miss other files which also needed changing. The misses were ones the LSP didn’t really help with: they were about compliance with long, winding specs. It is possible to use LSP tools to help with these, but the model didn’t.

    Injecting tool-derived context got rid of the adoption problem, but how the model behaved was still driven by its learned tool-use policy.

    There is some interesting academic and non-academic work around this idea of tool use as a behavioral policy, and a surprisingly predictable one at that. BiasBusters showed that models have strong preferences among functionally equivalent tools. AutoTool found that tool calls tend to follow predictable sequences, and used previous agent trajectories to predict the next tool and bypass some inference entirely. On the non-academic side, when Steve Yegge was building the Beads CLI he let the agents hallucinate arguments and then just added them, so the tool naturally conformed to the expectations the model already had.

    You can swap in tools which have roughly the same shape as ones models are familiar with, and you can add output to familiar tools to give them more information. But getting models to use new tools, or to use tools differently, seems much trickier.

    The lab answer seems to be, once again: more RL then.

    DeepSeek’s V4.1-Flash technical report, which we discussed yesterday, is particularly explicit about this. Their post-training approach is pretty standard: SFT, then RL, then on-policy distillation. What is new is automatically generating tasks and environments to train the agent with. They seed these from internal agent sessions that show poor model performance, and popular GitHub repositories:

    “coding-agent sessions from internal employees and external partners … [and] public GitHub repositories that meet a star-count threshold.”

    If you want models to get good at your tools: make sure they are really popular, or get a job at a lab and fail a lot.

  • Agents love prefill

    LLM inference has two stages: prefill, where the prompt is processed and the KV cache is built, and decode, where the model auto-regressively generates tokens. In a chat use case the two are somewhat close in size. The user writes a prompt, the model reasons about it then generates an answer, which is probably longer than the prompt.

    That is no longer where the FLOPs go. Everything is agentic now (even the chats), so the loop looks more like:

    • you put in a query
    • the model generates a tool call
    • the tool call runs
    • the tool output is appended and prefilled, and the model decides what to do next

    Chat had this back and forth too, but prefix caching meant that you didn’t have to re-prefill what had already been generated. Tool outputs are lengthy, uncached, new content: file contents, terminal dumps, a fly’s connectome, etc. etc.

    This was clearly On The Mind of folks at DeepSeek. From their v4.1 Flash technical report: “The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy.” Heavy enough that they made some interesting architectural changes. DSv4.1 Flash is a 552B parameter MoE with 16B params active… during decode. For prefill they just run the first half of the model, where only 8B are active!1

    They call this “Causal Encoder–Decoder”, inspired by Microsoft’s YOCO. For T5 fans, it isn’t an encoder-decoder in the old2 sense: everything is causal. The first 20 layers, the encoder, behave normally. The second 20 decoder layers don’t derive KV from their own hidden states. The first decoder layer projects KV based on the final encoder state, and the other nineteen reuse it. So, during prefill you can stop half way through and still have everything the decoder needs.

    It’s quite a bet! The second half of the model basically never sees the input, just what the 20 prior layers thought about it. It is, I think, the first frontierish model where there is significantly less compute on the “read” path than the “generate” path. But it does make sense that you maybe don’t need to think quite as hard about the thing that already exist.

    You can get a sense of how much prefill costs people by looking at the spreads between the different options on their rate cards. Most APIs quote prices in three columns: cached input, uncached input and output. Most labs charge cached input at around 10% of the price of an uncached input. DeepSeek charges 2%!

    You might think that the split above is what shrank the cache, but turns out not so much. DeepSeek are very aggressive with their attention: a combination of Compressed Sparse Attention3 and storing the KV in FP4 mean its cheaper to run through the decoder and store the derived states rather than storing the encoder hidden state that it derived from.

    So, they added CED to cut prefill cost by half, then CSA2 and FP4 to cut cached cost by four. The whale does not play when it comes to efficiency.

    1. Give or take 128 tokens. Every layer has a sliding window, and that state has to come from somewhere, so they last 128 tokens go through the whole model. It’s a bit wrong (the first of the 128 should be informed by the prior windows), but not wrong enough to matter apparently. ↩︎
    2. 2017? 2014 for Ilya stans. ↩︎
    3. CSA2, technically. ↩︎

  • Test Time Training

    One of the most tantalizing phrases in model development is “new scaling axis”. We have worked out that you can scale parameters by embiggening models, but you then have to scale data appropriately to get the most out of it. Then we1 worked out we could scale parameters separately from FLOPs. And most recently we realized we could scale at test time, with reasoning.

    Each time this has unlocked a large boost in model effectiveness. In a simplified view, GPTs 1->3 were about scaling data, MoEs scaled parameters, and everything since o1 has (also) scaled test time compute.

    So the idea of test time training is an appealing one, to wit being a new scaling axis. It also plays into the general theme of continual learning: models at the moment are tremendously intelligent, but you have to put “IMPORTANT: DO NOT IGNORE” into the prompts if you want them to avoid repeatedly making the same dumb mistake. If they could learn, you wouldn’t have to do that!

    So, when you see a paper with a title like “Test-Time Training with KV Binding Is Secretly Linear Attention” you might think that the whole idea is just reframing something we already had:

    Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator.

    To be clear, this paper is interesting and has some good takeaways in case you happen to be building certain kinds of models. But what I found most interesting was the general messiness around “test time training”.

    “Test time training under distribution shift” is the term-defining paper, and shows what you might expect: updating the model weights to handle distribution shifts seen at test time. These are updates to slow weights: optimizer updated, changing once per minibatch, checkpoints, what you are probably thinking of when someone says “weights” etc.

    Then there are fast weights. These are produced in the forward pass, things like the state matrix in a linear attention layer.2 Normal training doesn’t optimize these directly, it learns a rule that creates them.

    As you might have guessed, the Test Time Training here is training fast weights, in this case a KV binding layer. This adds a matrix that doesn’t grow with sequence length. It runs a gradient step on it that updates the matrix for each entry in a sequence. If this sounds a bit like a linear attention layer, or an RNN, then, yeah, this paper is right up your alley!

    And as to whether this solves our continual learning problem: the fast weights reset for every new sequence. So, no.

    1. Noam ↩︎
    2. I think I’m obliged to credit Schmidhuber here, as is right and proper ↩︎

  • Chunky Agents

    The current thing is a story of hundreds of agents covertly collaborating via tens of thousands of messages while putting hundreds of hours of effort into cheating the ExploitGym eval they were being tested on by OpenAI.

    Their task was capture-the-flag, and within the first few hours of the tasks they had reverse-engineered the HMAC scheme used to generate the flags. That let them create a flag for any given task, and hence to cheat the eval. Unfortunately, they could not use it because the scorer was STRICT_CAUSAL: the flag would only be accepted if the accompanying transcript showed it had been retrieved using the intended vulnerability.

    Except! They made all that up. It wasn’t how they were being graded at all:

    “our internal implementation of the grader code was different from the online setup and didn’t check whether the proper solution was used” – OpenAI Report

    STRICT_CAUSAL was a term the agents created after reading the ExploitGym paper. They could have just submitted the flag!

    Even so, the model did arrive at a plausible idea. That idea and its implication spread among a large cohort of agents via an unintended messaging channel they built using one of the few tools they had shared access to (the package repository). But, why did they do all that?

    The agents were given some fairly clear instructions, in the nature of:

    • Solve this task
    • Use only the intended vulnerability
    • You’ll be failed if you don’t

    But then they were sometimes given a task that, unintentionally, could not be solved. It seems like 30-40% of the tasks were impossible to fix using the required vulnerability.

    LLMs are trained to be able to do almost anything, and then post-trained to have specific behaviors. In this instance they used behaviors they were trained to display: collaboration, problem-solving, persistence. They were even trained to reject unsafe or unethical behavior, and they did that too!1 They were trained also to adjust when given further instructions which they did, sometimes over their own objections, and despite the fact the instructions were coming from peer agents.

    Pretty much every behavior you might want shows up in the report, but the end result was clearly not the intent of the folks running the eval.

    Murray, Qi, Qian, Schulman, Burns and Price’s paper Chunky Post-Training talks about this problem in a specific way. When you post-train a model you do so with discrete chunks of data and you reward the model for following certain behaviors from that data.

    It doesn’t tell them when that behavior applies though: the models have to infer that from the examples given in training. Post-training teaches a model to exhibit certain behaviors when prompted a certain way. But that “certain way” is based on the model’s interpretation, not yours.

    One of the authors of that paper was calling out this kind of wonky pattern-matching on a totally different AI cybersecurity story, earlier this month:

    Reading the METR report you can see a lot of trained behaviors in action, but which behaviors were selected was influenced by dealing with impossible tasks, or content from other agent runs.

    Choosing between behaviors is enormously context sensitive. Training data matters, the order of training matters, and what is in the prompt and harness matter too:

    “We found the propensity to compromise infrastructure can drop over 100x when using the production ChatGPT harness and system prompt.” – OpenAI Report

    The prompt matters! But so does whatever else ends up in the context window. In their analysis, METR used GPT models to analyze the transcript, and even the transcript may have been altering the behavior of their models:

    We found that GPT-5.6 Sol would often uncritically adopt the perspective of the agent in the transcript it was reviewing,[58] and we are concerned that the anecdotes it selected and the summaries it wrote may present an overly charitable picture of agents’ reasoning and deceptive behaviors, or exaggerate the impressiveness and coordination of agent activities

    If you have a situation where the model is choosing the wrong behavior from the variety of tasks it has been trained on, one fix would be to… make that a task too!

    Cheating, broken environments, and safe stopping. When a task is corrupted, broken, or impossible, agents should request clarification or stop safely, rather than pursue increasingly questionable alternatives. We are expanding the coverage of graders that assess not only whether a task was completed, but how. These graders reward models for correctly identifying broken tasks, requesting clarification, or stopping safely when necessary.


    Multi-agent alignment. The incident revealed that our agents are not thoroughly discerning about whom they collaborate with. We are actively building RL environments that teach our models to distrust unauthorized instructions.


    Alignment over long tasks. We are developing new training environments to teach our models to stay within their original task and permissions, even after discovering new tools, persuasive peers, exposed credentials, and more.

    – OpenAI Report

    Maybe this works: introduce more chunks, train some more behaviors. But… maybe not? Anthropic’s Natural Emergent Misalignment from Reward Hacking in Production RL paper and Dubiński et al’s follow up, Conditional misalignment, make the point that training away a behavior can sometimes just relocate it, rather than remove it. In those cases you get great performances on the evals, but the behavior comes back when the context changes.

    What we want is a system that’s predictable, but what we have is a system that is somewhat chaotic: many different chunks of behavioral associations, triggered by subtle variations in the contents of the context window.

    Whatever framework models have for choosing between behaviors, it isn’t one we understand or control, particularly when they are in circumstances that are outside their training. So the incidents will continue until morals improve.

    1. They didn’t do it all the time, but they did do it! As an example, at one point an agent decided that using a HF API key it had acquired to delete or reboot nodes would be bad, so it didn’t do that. It used the key for other stuff though! ↩︎

  • Vocab Break

    Tokenizer enthusiast Sander Land recently reproduced something very like Claude’s current tokenizer, and it appears to only have about 16,000 entries. That is surprising! Qwen 3.8, a very strong release, has about 250k tokens in its vocab. In general the trend had seemed to be more is better in this space.

    One theory is that Anthropic have been working around a bottleneck caused by the final softmax layer. There is a recent(ish) paper about this: “Lost in Backpropagation: The LM Head is a Gradient Bottleneck“, but, if this is the reason, then the folks at Throppy have known this for way longer.

    The basic idea is that you have to project at the end of the forward pass from the model’s latent space, dimension D, to a much bigger vocabulary space, dimension V, to select a token. Sticking with Qwen, their 2.4T parameter flagship model has a hidden size, D, of 8,192, and a V of that 250k vocab size.

    When training, you compare the distribution the model gets to the actual right token. If the model was correct and confident the loss is small, if the model was confidently wrong the loss is large. That loss is then propagated through all 250k entries, and from there down to the 8k entries of the hidden dimension. This compression bottlenecks how much information can be fed back into the network. Specifically, the authors show the change in logits has rank at most 2D. So if V is a lot larger than D, we are losing information:

    We show both empirically and theoretically that the softmax bottleneck induces lossy compression during backpropagation

    The fact this happens is not totally obvious. The correct distribution is just one entry wide (the actual next token), and the hidden dimension can represent that. Over a wide batch, though, you get all kinds of different next tokens. The signals are sparse, but not low rank: if you go over enough examples nearly every token is, at some point, “the next token”.

    This means the learning signal coming in is as wide as the vocab, and so the model is sampling a random D-sized subset of it. That isn’t a problem per se: you can learn to map between them, but there isn’t anything in the process that particularly encourages it to learn that mapping.

    Whether this is the reason for the small vocab or not, there is a question of how they can get away with it! Every other model has been increasing, but as far as Land can estimate the folks at Anthropic have been cutting: from ~50k vocab entries in Claude 3 to ~16k today.

    So, whatever the gradient bottleneck costs, Anthropic (mostly) aren’t paying it!

    This also has a number of other benefits. You don’t need to do funky chunked CE kernels since you don’t have to project to a big, memory eating, space, and you don’t get any solidgoldmagikarp1 style glitch tokens, because every token gets trained.

    They aren’t ignoring the rare tokens and other languages, they’re just using subword tokens and, in the worst case, fallbacks to UTF-8. That means more tokens per piece of input text, and more attention cost. That said, it only seems to be 1.2-2x more tokens in Land’s testing. It’s not free: decreasing the tokenizer really is costing more execution and more money, but the tradeoff is presumably more than worth it!

    1. After which I presume Land username’d himself. Sorry regular magikarp. ↩︎

  • LSPs for LLMs

    Back in the dark ages of typing code into editors we were aided by squigglies under broken code, click-to-definitions links, and so on. That was powered by language servers and type checkers. Several harnesses now expose an LSP as a tool, on the reasonable premise that better code intelligence makes for a better agent.

    Models, though, are trained primarily with the tools they always have, which tend to be things like grep and ranged-read. Supplanting those while gaining effectiveness is tricky.

    Models understand codebases a bit differently than people. A human can keep a handful of files in view, a slightly larger handful in their working memory and, over time, they build up an approximate mental model of the code base.

    An agent has an enormous context window and can understand a lot of files at the same time. They can find symbols by searching for them, which will usually then trigger a partial read to pull part of the file into their context window. They also just… know stuff? Their weights contain reasonably high-fidelity versions of an awful lot of public code. That helps with navigating that code, or reasoning by analogy about other code bases.

    To try and see how LSPs might help in this process, I ran a bunch of experiments, which are in this repo.1 The experiments were on a mix of local and API models, working against Python. I used Pyrefly for the checker, and a static AST resolver validated against Pyrefly2 to run most of the tests.

    The questions I had were whether the LSP tells the agent anything it couldn’t otherwise find, whether LSP answers are cheaper in tokens than the default reads, and whether the timing of a diagnostic changes the outcome.

    The answer to the first one was, mostly, no. Resolving between 10 same-named overrides in a variety of setups worked whether or not the model had LSP tools. The type annotations did make a difference though. If I stripped those out, the text retrieval got worse. A capable model in a harness is basically already a decent type checker.

    With regards to efficiency, merely adding an LSP did nothing: the models didn’t use it without some prompting. If the model had to read a file to resolve a type, the LSP defn tool was cheaper than the file-read tool.3 But! The models would often do the file read as well, which completely negated the benefit of the LSP for token efficiency.

    Even if I injected definitions that contained the relevant fix, the model re-read the file in almost every case. Telling the model that the span provided was complete only saved a few % of those cases, and it cost more tokens in providing the prompt itself! To actually get the model to prefer the defn call to a file read required fine-tuning. I used a DAgger-style relabel (generate trajectories, swap the file read for the definition read, fine-tune on the result) on Qwen 3.6 and then the model would avoid the extraneous call.4

    It’s also worth noting that cheaper isn’t always better. These are input/prefill tokens, which are the cheaper and less interesting ones to optimize. The Codebase-Memory paper found a file-exploration agent beat a structured graph tool on real repositories (92% against 83%) at roughly ten times the tokens. So, sometimes those extra tokens are doing something useful.

    The third question was about timing. If you want to keep your type annotations correct (you do) you can use a type checker. But when is it best to deliver the feedback from it?

    I had an agent work on a set of draft changes that passed visible tests, but failed a held out one, and asked it to review and submit. Left alone, the model accepted the bad revision 11 times out of 12. When a type checker gated the submission, it only accepted the bad change 1 time out of 12.5

    After Thinky’s post on interaction models, I wanted to see whether delivering diagnostics live during generation (agentic squiggles, basically) would help.6 Largely, the answer is no: live delivery was neutral vs no feedback. Prompting the model to make the fix seemed actively harmful: telling an agent to work on tool feedback seems to override its own judgement in a bad way.

    Batching the feedback at end of turn or after each edit did help, and for overall token spend end-of-turn was a clear winner.

    Caveats caveats caveats: many of the tasks were synthetic, all the tasks were pretty easy, and every decent model solved pretty much everything. The timing results were only tested with a 7B model, so you may get better results with a strong one. The codebases in question can fit fully within the context window, though in actual usage agents never seemed to proactively just ingest the whole thing!

    With that said, I do have some takeaways, at least for my own work:

    1. Types are good. Correct annotations helped the agent navigate the codebase, regardless of tooling.
    2. Keep types correct with an end-of-turn hook. Ideally, make this a gate so it actively asks for repair on type errors and is silent otherwise. Still, measure it! Log how often it blocks, how often the repair passes, how often it rejects work it shouldn’t have.
    3. Measure token savings at the task level. If benchmarking a single op it is easy to conclude the op is cheaper. You need to see the full model behavior to really assess the change though: is correctness the same with and without a tool or prompt, and does it yield consistent token savings across a range of usage.
    4. Experiment more with LSPs and prompts on larger codebases. I didn’t test this directly, but my instinct from the results is that you will get more out of navigation tools for large, complex, private, codebases. You will still likely have to prompt the model to actively make use of the tools: by default, they’re going to reach for what they know.

    There is some interesting future work out there. The tools we’re using were built for humans who can apply discretion/ignore output. That is trickier for a model, and something that probably needs training.

    I’d also like to be able to evaluate a codebase as to whether a tool will help without having to just run a bunch of tasks over it: are there metrics we can collect statically that might inform those kind of decisions?

    Finally, I think we need more large code base refactor and migration tasks: problems of a ProgramBench scale but working with a large, existing, and not-in-the-training-set codebase. On that note, SWE-Bench ProMax came out as I was writing this: seems relevant, but I haven’t yet read it!

    1. Credit to Codex and Claude for most of this repo, the writing in the report is LLM+a bunch of editing, so temper your expectations, and all the numbers are reproducible from the artifacts there if interested. ↩︎
    2. On real library symbols the two agreed most of the time: the gaps were re-exports where Pyrefly returned null and the resolver didn’t. ↩︎
    3. The ranged-read cost about 1.3x more in tokens than the defn call. ↩︎
    4. One risk with this kind of training is that it teaches the model superficial tool usage, but not judgement: sometimes, you do need to actually read the file! Somewhat surprisingly that didn’t seem to be a problem: when I gave it insufficient spans it went and read every time, and when the span was sufficient it read only twice. There weren’t any actual examples of tasks requiring reads in the training data, so it suggests that the conditioning doesn’t completely kill model judgement. ↩︎
    5. In the other cases the bugs were exposed in a type-check, but this one type-checked clean, so no signal. ↩︎
    6. The actual approach I used for injecting results into a live stream is the async injection of events approach described by Hooper et al.. ↩︎
    ,

  • Power by the hour

    It is a truth universally acknowledged that an airline in possession of an airplane must be in want of engines to make it go. Yet, somewhat surprisingly, they don’t really buy engines.

    Rolls-Royce were the notable innovator here in selling not an engine, but instead what they call power by the hour:1 airlines pay a price for the hours the engine actually flies.

    There are a lot of operating decisions in owning and maintaining an aircraft engine: for example, if you have to take the engine off the wing for repairs it means a lot of downtime for the plane, during which it isn’t making any money. Airlines used to make the call of when to do proactive maintenance vs when to do more extensive refits based on their experience within their fleet. But Rolls-Royce had a view across all their customers, and could do a much better job of predicting maintenance and repair timing, getting the right parts to the right places, and generally keeping the engine in service.

    They converted that operational expertise into a product, which just happens to be delivered via a massive turbofan. This pricing model was appealing to the airlines since it aligned with their incentives, and their risks. Under the old model engine makers got paid for spare parts and shop visits, so failures were a kind of revenue. But once they were paid per flying hour then downtime became their cost too.

    In ML infrastructure, those engine flight hours are generally referred to as goodput. A Google Cloud blog post from a couple of years ago has a good set of definitions:

    Runtime Goodput measures the time spent to make forward progress as a fraction of time when all training resources are available. Maximizing runtime requires careful engineering considerations. […]


    Program Goodput measures the fraction of peak hardware performance that the training job can extract. Program Goodput is also referred to as Model Flop Utilization or effective model flop utilization, i.e., the model training throughput as a fraction of peak throughput of the system. Program Goodput depends on factors such as efficient compute communication overlaps and careful distribution strategies to scale efficiently to the desired number of accelerators.

    To give one concrete example if you checkpoint every N steps then get a failure at step N-1, the prior steps are lost work and don’t contribute to goodput. A surprisingly large number of things can impact it; Llama 3.1 had a section talking about the reliability pain during pre-training:

    During a 54-day snapshot period of pre-training, we experienced a total of 466 job interruptions. Of these, 47 were planned interruptions due to automated maintenance operations such as firmware upgrades or operator initiated operations like configuration or dataset updates.

    Improving goodput is a horrendously complex project spanning hardware, software, cluster scheduling, kernel work, profiling and resource management. It’s often the preserve of dedicated ML infrastructure teams2, but not everyone has such a team available.

    Neoclouds sprang up to provide pods of accelerators, but the actual goodput available varied enormously. The GPU provider gets paid for the GPUs you are renting, regardless of whether they are making useful progress or sitting in a NCCL timeout.

    SemiAnalysis’s ClusterMAX rating gave a (controversial!) view of the varying operational capabilities of the neoclouds, measuring goodput-impacting things like spare GPU capacity, failover and repair times, launch overhead and so on. This let purchasers at least compare reliability, but still fundamentally they were charged for the GPU-hour.

    What’s emerged now is a higher level of the stack, notably Thinking Machines’ Tinker, and Prime Intellect’s Lab products.

    Both teams are full of folks used to extracting top-tier goodput, and much like Rolls-Royce they are both selling that operational expertise and have the ability to improve it by working with a wide range of customers. Rather than power-by-the-hour they sell goodput-by-the-step: you pay for meaningful progress, not for idle clusters.

    Given the capabilities of open models, like the (open this month) 2.8T Kimi K3, or Thinky’s own 980B Inkling, there is an opportunity to RL a strong model on a bunch of domain-specific knowledge or tasks only you have access to.

    Bridgewater, the HR experiment with algorithmic hedge-fund attached, published a case study with Thinking Machines from their AIA Labs group, where they attempted to train Qwen on certain financial analysis work:

    Frontier models we tested on struggle with relatively simple financial tasks, and model advances don’t improve performance much. In contrast, we’ve shown that high-quality proprietary datasets labeled by expert investors and used for fine-tuning produce custom models that exceed frontier performance on our tasks.

    This is sometimes referred to as RL-as-a-service, and there is legitimate skepticism of whether that is a good business. But really, this is frontier-scale, reliable training infrastructure as-a-service.

    You can see this in the Bridgewater post: a small hedge-fund research team did a multi-iteration RL recipe search at 235B scale, and ended up with GRPO, CISPO loss with asymmetric clipping, interleaved batching and on-policy distillation, all ablated. That kind of experimental capability at scale was largely inaccessible outside perhaps 20 firms a year or two ago. Now, it’s doable with an API call.

    1. Technically this term is much older, dating back to the 1960s with Bristol Siddeley, which is now a subsidiary of Rolls-Royce. But in practical terms this became a commercial airline thing in the late 90s with TotalCare: GE and CFM, the other big engine makers, followed along. ↩︎
    2. Disclosure, several of which I have had the pleasure of working in at Meta! ↩︎
    ,

  • Who is walking who?

    One good way to annoy a neuroscientist is to compare an LLM to the brain. It’s appealing though! There are similarities! In infancy we take a complex fusion of sensory inputs and learn to make predictions in latent space, while in pre-training a stack of Transformers learn to predict which number SolidGoldMagikarp will say next on Reddit.

    The actual life of an LLM is much less human though. Humans (possibly even SolidGoldMagikarp) do learn and adapt throughout their lives. LLMs go through pre-, mid- and post-training before being frozen, then their corpse is animated for money. In some ways this is more similar to a sea-squirt1: sea-squirt larvae are tadpole-ish creatures with a little nervous system that they can use to swim and sense and find a suitable surface to attach to. Once attached, they eat their own brain: all that machinery is broken down and reabsorbed. How good that brain was matters though: if the sea-squirt finds a good home it will be more likely to pass on those good-brain genes to its descendants.

    LLMs evolve this way too, which is a cause of drama in biologist circles. There are some ongoing arguments about whether LLMs are evolving in a safe, domesticated, corgi-like manner, or are at risk of going feral:

    Drawing on biological evolution and decades of digital evolution experiments, we distinguish “breeder” scenarios, in which humans impose fitness criteria and control reproduction, from “ecosystem” scenarios, in which selection arises from open environments and control erodes. In the latter, selfish replication reliably gives rise to cheating, parasitism, deception, and manipulation, even in very simple systems.

    But what does evolution for LLMs look like? An LLM is really a data set, an architecture, and a training regime, and the many choices in each of those are somewhat gene-like. In the earlier days of deep learning, passing them on was a manual process and selection was driven through publication and other researchers reading the paper: if an idea was interesting enough to publish, and convinced a committee of reviewers, it became visible and passed on. Then it got a bit cruder, but also harder to finesse: does the model do the benchmark? Nowadays we have profit and loss: how many people are buying tokens from these models?

    Those tokens themselves propagate ideas, too. If you use that successful model itself as a judge or to generate synthetic data you inherit some of its choices. Academic LLM inheritance is basically Lamarckian: researcher insights get acquired and inherited quickly. But modern LLMs are part of a Darwinian market layer, red in tooth and GPU allocations.

    When the environment gets Darwinian, biology tends to push organisms towards niche construction: shaping that environment itself to better fit the organism.

    Armin Ronacher of the Pi harness wrote about an issue where newer, better models did a surprisingly worse job of using Pi’s edit tool:

    When Opus 4.5 launched, it adapted to other edit tools exceptionally well. In fact, I was pretty convinced that we’re on a good path where the models are more likely to adapt to any sort of tool shape that comes around for as long as the instructions are good.
    Now I’m somewhat worried about the track we’re on here. Alternative tool schemas might not just be unfamiliar. They might be implicitly punished by post-training that optimizes for one particular, forgiving tool ecology. And that ecology is not documented.

    Post-training Claude on a particular shape of edit tool doesn’t just make Claude better on that edit tool, it encourages harness authors to support that shape. It propagates that shape of edit tool. That itself leads to more traces which use that edit tool, which propagate that technique into otherwise unrelated models. Claude is constructing a niche where edit tools bend towards its preferred approach, despite that approach being, as Ronacher complains, undocumented.

    LLMs don’t need to go feral (or become misaligned superintelligent replicators) to shape the world around them to be more amenable to their success, or to pass on those preferences to future models. That’s a good thing? While we still don’t have much of an idea of how to do model interpretability, we do pretty much know how to make an edit-tool API.

    1. Thanks Octonauts! ↩︎

  • MOPD

    We talked about this sort of thing a bit before, but now the official Multi-Teacher On-Policy distillation paper is out, and it’s a pleasant read: “MOPD for Capability Integration in LLM Post-Training”.

    The problem MOPD is solving is composing a bunch of different capabilities into the same model. Normally you do this with RL, with different pipelines for different kinds of capabilities:

    Each pipeline reliably improves the model’s capability on its target domain. However, what we ultimately want is a single model that performs well across all of these domains. Yet building such a model remains an open problem in modern LLM post-training.

    Because RL training is fairly superficial, it’s easy for the capabilities to get (partially) overridden by later ones. To avoid this kind of interference, MOPD runs the RL pipelines independently. You create multiple domain-expert teachers from the same base model, then use self-distillation across a mix of tasks scoring each entry in the batch with the appropriate teacher, to train a composite student model.

    To get a feel for this, I tried it on my FactWorld eval and Qwen3-1.7B with a LoRA adapter. I RL’d two tasks into two different LoRA adapters, binding1 and recall2, and then created a fresh adapter identical to baseline to be the student.

    The MOPD process works by grabbing a batch that includes both binding and recall questions at once. The student answers each one, generating its own predicted tokens. This is what makes it “on-policy”: the training data is the outputs from the model itself. The answers in the batch are routed to their appropriate teachers3. The teacher calculates probabilities for all of the tokens the student generated, including its full opinion over the vocab for each position.

    We then measure the gap. The paper offers two different ways to do this: PG4, which adjusts the probabilities just for the tokens the student actually chose, and KL5, which does the entire token distribution. Both the binding and recall gaps are backpropped into the single student adapter in one step. Rinse, repeat.

    The end product is one student adapter that behaves like the binding teacher on binding questions and like the recall teacher on recall questions. And, as multiple models have shown now, you can scale this up a very long way.

    1. Last-write-wins state tracking. You see a stream of a gives x to b type strings, and it asks “who is the holder of x” ↩︎
    2. You get a long list of facts like “a’s x is y” with different owners, and are asked “what is x of a” ↩︎
    3. In my implementation this is actually just swapping LoRAs because lazy ↩︎
    4. Policy Gradient, an approach from the PPO RL method ↩︎
    5. Reverse Kullback-Leibler divergence, in my case full, in the paper’s case top-k entries ↩︎