Blog

  • JEPA

    JEPA is an example of a predictive coding architecture. Predictive coding is the idea that the brain works by predicting the next sensory input and learns from the error between the prediction and the actual input. Hierarchies of this kind of prediction allow higher level elements to predict the outputs of lower-level elements, building up deeper and more complex semantics.

    The core idea in JEPA is to take two related things (say consecutive frames of a video) x and y, encode each of them into a latent space (an embedding), and then predict the embedding for y s(y) based on s(x). The encoders can be Transformer-based models — in practice models like I-JEPA have trained the x encoder and updated the weights via a moving average for the y (target) encoder.

    The learning is not based on how well the end-result predicts the target e.g. how close the pixels of the next frame are predicted. Instead, it’s based on how well the latent representation of the next frame is predicted.

    The advantage of working in the latent space for the prediction is the model can choose what level of detail it wants to capture, discarding some aspects and focusing on more foundational concepts. This helps build a more robust world-model, with the hope being that training in this way will then allow easier generalization to more tasks, with less data required

    Similarities

    This is somewhat similar to autoencoders. Autoencoders take an input, compress it in a latent space, then reconstruct the original from the latent space and propagate back the error. JEPA does a similar process across two different items with separate encoders, and only cares about error within the latent space.

    Contrastive models embed two different items into the same space and try to increase similarity between the embeddings for things known to be similar and make them dissimilar to other items. This is used in CLIP and other multimodal text-image encoders, where the text and the image embed to the same space so that a text caption and a matching image are close in embedding space. This requires a lot of pairwise comparisons, while JEPA is a more straightforward s(x)->s(y) prediction in training.

    Challenges

    Because JEPA models leave you with a latent they need to be paired with a generator for getting an observable/human viewable output, which is a per-domain challenge. This makes it harder to evaluate how well the model is learning, beyond measuring loss.

    Training stability can also be tricky — it is possible for the model to collapse and learn trivial representations to minimize prediction error. Even without complete collapse it can require some experimentation to ensure the model is learning a deep enough conceptual level. For example, I-JEPA, which worked in image space, found that using large enough masked patches was important to ensure the model captured sufficient detail.

  • Free-Threaded Python community

    As Thomas posted on the Python Discuss board there is a discord for discussing Free-Threaded/NoGil Python: https://discord.gg/rqgHCDqdRr

    Other than the helpful docs the Quansight folks maintain (py-free-threading) it’s been interesting to see some projects and tools pop up on there I hadn’t see. One being Zsolt’s py-free-threading which checks whether your project and deps have FT wheels for any non-pure python deps, which can be run as a 1-liner thanks to uv:

    uvx python-ft-deps

  • Colfax on Blackwell GEMMs

    CUTLASS Tutorial: Writing GEMM Kernels Using Tensor Memory For NVIDIA® Blackwell GPUs – Colfax Research

    Dives deep into TMEM into particular, and the trend over the last few Nvidia generations of special-casing GEMMS in hardware:

    Tensor Memory and UMMA do for MMA just what TMA did for copy, making it a single-threaded, asynchronous operation that does not consume registers. As a result, registers can primarily be used for other tasks like scheduling and fused epilogue operations.

    Edit: link no longer seems to be working! It was a great post though, so hopefully comes back! Edit edit: it did!

  • MegaScale-infer: disagg MoE inference

    https://arxiv.org/abs/2504.02263

    LLMs have most of their parameters in the FFN parts of the transformer layers — 50+bn params of the Llama 3 70b model, for example. The compute and memory requirements are a bit different between the FFN and attention parts of the model: attention requires a different KV cache for each request, so attention tends to be memory bound while the dense FFNs tend to be compute bound.

    Because of this it’s pretty common to split up tasks at inference time. The initial prefill stage (processing the initial prompt) populates the KV cache for the following autoregressive decoding. The decode can be more aggressively batched for getting better utilization. vLLM really helped popularize this idea!

    ByteDance extend this idea for mixture of expert models. In MoEs the compute intensity of the FFNs is limited by needing to load different experts, and having only a proportion of tokens going through a given expert. They extend the disagg idea to go from M “attention” GPUs to N (fewer!) expert GPUs, with a larger batch size for each of the expert calls. This gets better utilization on the matmuls and lowers overall cost of serving. The natural structure of transformer layers alternating attention and FFN lends itself well to a ping-pong pipelining approach that lets them hide the comms overhead.

    We present MegaScale-Infer, an efficient and cost-effective system designed for large-scale MoE serving. MegaScale-Infer disaggregates the attention and expert modules, assigning them to separate GPUs—a strategy we term disaggregated expert parallelism. Our approach offers two major benefits. First, it enables independent scaling of each module with customized model parallelism strategies. Specifically, attention modules are replicated using data parallelism, while FFN modules are scaled with expert parallelism. By consolidating requests from multiple attention replicas, the GPU utilization of each expert increases significantly as the batch size per attention replica grows. Second, it enables the deployment of attention and FFN modules on heterogeneous GPUs to fully leverage their different capabilities and achieve lower costs. For example, attention modules can be deployed on GPUs with more cost-effective memory capacity and bandwidth, while FFN modules can utilize GPUs with more affordable compute capability. As shown in Figure 1(c), FFN can easily become compute-intensive in MegaScale-Infer, while attention achieves higher GPU utilization per cost under heterogeneous deploymen

  • Pydantic Evals

    https://ai.pydantic.dev/evals/

    Ed Yang was recently recommending keeping your own benchmark of LLM evals, so you can test newer models on problems that they have struggled with in the past. I have recommended similar things to people, but there is some barrier to entry into knowing how to start. Ed references (and forks) Nicolas Carlini’s personal benchmark repo, but its nice to have some light(ish) weight options too.

    Pydantic Evals is a powerful evaluation framework designed to help you systematically test and evaluate the performance and accuracy of the systems you build, especially when working with LLMs.

    You can install the library with uv or pip:

    uv add pydantic-evals
    

    I tried it out with a strawberry test, calling openrouter with different models. I needed a custom eval as the default Contains is a bit rigid, but the approach seems nice!

    import os
    import asyncio
    from dataclasses import dataclass
    from pydantic_evals import Case, Dataset
    from pydantic_evals.evaluators import Evaluator, EvaluationReason, EvaluatorContext
    from pydantic_evals.evaluators.common import _truncated_repr
    from openai import OpenAI
    from typing import Any, Optional, cast
    @dataclass
    class FlexibleContains(Evaluator[object, object, object]):
        """
        Check if the output contains any one of the expected options.
        """
        value: Any
        case_sensitive: bool = False
        def evaluate(
            self, ctx: EvaluatorContext[object, object, object]
        ) -> EvaluationReason:
            failure_reason: Optional[str] = None
            # Normalize value into a list of options if it isn't already a list or tuple.
            options = self.value if isinstance(self.value, (list, tuple)) else [self.value]
            output_str = str(ctx.output)
            if not self.case_sensitive:
                output_str = output_str.lower()
            match_found = False
            for opt in options:
                opt_str = str(opt)
                if not self.case_sensitive:
                    opt_str = opt_str.lower()
                if opt_str in output_str:
                    match_found = True
                    break
            if not match_found:
                failure_reason = (
                    f"Output string {_truncated_repr(output_str, max_length=100)} does not contain "
                    f"any of expected strings: {[str(opt) for opt in options]}"
                )
            return EvaluationReason(value=match_found, reason=failure_reason)
    strawberry = Case(
        name="strawberry",
        inputs="How many rs are in strawberry?",
        evaluators=[FlexibleContains(value=["3", "three"])],
        metadata={"difficulty": "easy"},
    )
    dataset = Dataset(cases=[strawberry])
    MODELS = [
        "anthropic/claude-3.5-sonnet",
        "openai/gpt-4o",
        "meta-llama/llama-4-maverick:free",
        "meta-llama/llama-4-scout:free",
        "openrouter/optimus-alpha",  # secret model!
    ]
    def generate_completion(inputs: str, model: str) -> str:
        """Generate a completion using OpenRouter with specified model"""
        client = OpenAI(
            base_url="https://openrouter.ai/api/v1",
            api_key=os.getenv("OPENROUTER_API_KEY"),
        )
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": "You are a helpful AI assistant."},
                {"role": "user", "content": inputs},
            ],
            max_tokens=50,
            temperature=0.7,
        )
        return response.choices[0].message.content.strip()
    def evaluate_models():
        """Run evaluations across multiple models"""
        for model in MODELS:
            print(f"\nResults for model: {model}")
            print("=" * 50)
            # Wrap the synchronous generate_completion in an async function:
            async def model_specific_generate(inputs: str) -> str:
                loop = asyncio.get_running_loop()
                return await loop.run_in_executor(None, generate_completion, inputs, model)
            # Run evaluation for this model
            report = dataset.evaluate_sync(model_specific_generate)
            # Print results for this model
            report.print(include_input=True, include_output=True, include_durations=False)
    def main():
        evaluate_models()
    if __name__ == "__main__":
        main()
    

    To give a trimmed output:

    Results for model: openrouter/optimus-alpha
    ==================================================
                                          Evaluation Summary: model_specific_generate
    ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
    ┃ Case ID    ┃ Inputs                         ┃ Outputs                                                   ┃ Assertions ┃
    ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
    │ strawberry │ How many rs are in strawberry? │ The word **"strawberry"** contains **three** letter "r"s. │ ✔          │
    ├────────────┼────────────────────────────────┼───────────────────────────────────────────────────────────┼────────────┤
    │ Averages   │                                │                                                           │ 100.0% ✔   │
    └────────────┴────────────────────────────────┴───────────────────────────────────────────────────────────┴────────────┘
  • Bonanza Build System

    https://blogsystem5.substack.com/p/bazel-next-generation

    JMMV shares this thoughts on build beyond bazel, and highlights Ed Schouten (of BuildBarn) and his experiments with Bonanza. Bonanza maintains Bazel compatibility, but is remote execution first, does analysis on the remote workers with a distributed cache to minimize cold builds, and has starlark-only for rules (as with Buck2). Also like Buck2, its written in Rust.

    When I was at Lyft we went through a laborious build system migration, including on Android going from Gradle to Buck(1) to Bazel, which at one point involved a full shim that allowed single build files to work on either Buck or Bazel. The idea of being able to keep the same build definitions but swap out engines is pretty appealing.

    Julio actually calls for yet-another Bazel replacement as well, this one more focused on the small project/local build case: I can definitely see the appeal there!

    The time for these next-generation Bazel-compatible build systems is now. Google has spent the last 10 years Starlark-ifying Bazel, making the core execution engine replaceable. We are reaching a point where the vast majority of the build logic can be written in Starlark as Bonanza proves, and thus we should be able to have different build tools that implement the same build system for different use cases

  • Improving Recommendation Systems with LLMs

    https://eugeneyan.com/writing/recsys-llm/

    Eugene Yan has put together a really extensive survey of recent research exploring the use of LLMs in recommendation systems.

    Although early research in 2023—that applied LLMs to recommendations and search—often fell short, these recent efforts show more promise, especially since they’re backed by industry results. It suggests that there are tangible benefits from exploring the augmentation of recsys and search systems with LLMs, increasing performance while reducing cost and effort.

    Recommendation systems are enormously important to a large swathes of tech business, primarily for e-commerce, content and advertisement targeting. Traditional deep recommenders typically use a two-tower architecture: one tower for users and another for items, independently encoding features into embeddings that can be scored together to retrieve and rank items. Features in each tower include both sparse (usually categorical, e.g., item categories, user histories) and dense (often continuous, e.g., age, price).

    This design is popular because its effective and scalable: you can cache each tower’s embedding vectors and only pull in the ones you need for a given query (e.g. the batch of users you are getting recommendations for right now).

    Despite the effectiveness and scalability of this approach, traditional systems often struggle with a set of known issues, such as cold-start problems—predicting relevant content for new items or users—and typically don’t consider interaction recency without additional engineering.

    Yan categorizes recent research into four areas:

    1. LLM/Multimodal Architectures:
      • Directly embedding content understanding within the models. Content understanding has been used for a long time via separate models to generate additional metadata for content items to help both with cold start and accuracy.
      • Generative approaches, which reframe recommendation as predicting future user actions based on interaction sequences.
    2. LLM-Assisted Data Generation and Analysis:
      • Improving content understanding and generating richer metadata for items.
    3. Scaling Laws, Transfer, and Distillation:
      • Adapting LLMs to meet latency requirements of recommendations, through smaller models and efficient inference techniques. RecSys, particularly models for advertising, tend to have very low latency requirements.
    4. Unified Architectures for Search and Recommendations:
      • Consolidating search and recommendation tasks into unified models that enable returning items based on interaction histories and/or user queries simultaneously.

    There are a couple of common themes from reading the summaries:

    • Semantic Content Integration & Joint Tasks: Techniques like YouTube’s Semantic IDs and Kuaishou’s M3CSR generate content-based identifiers replacing traditional hashed IDs. The idea is to have inputs to the models represent the content in a way that carries meaning, rather than represent an identifier for the content.
    • Efficiency in Inference: Teacher-student distillation and efficient fine-tuning allow generating smaller, performant models for specific needs. For instance, Alibaba’s MLoRA trains a base model then LoRA fine-tunes for specific types of content, replacing a number of independently trained models.

    These two combine somewhat to enable a trend towards more foundation- model-like training in RecSys that tackle a variety of user personalization tasks with a unified view of users, content, and user/content interactions.

  • Dynamic Shapes in PyTorch

    Dynamic shapes are one of the more distinctive parts of torch.compile. Rather than specializing a graph to static shapes (which works in many cases!), PyTorch’s approach allows a single graph to work for a variety of sizes, so things like sequence length or batch size can vary. It does this by reasoning about shapes symbolically: instead of using fixed shape values, it uses placeholders and infers rules that constrain those shapes.

    Tracing & Symbolic Shapes

    PyTorch uses tracing in Python (via Dynamo) to capture the graph of operations. By default, it marks shapes as static during tracing. If a shape marked as static changes at runtime, it is marked as dynamic and treated symbolically in a recompilation. You can also proactively mark a dimension as dynamic to encourage symbolic treatment from the start:

    torch._dynamo.mark_dynamic(x, 0)  # Mark dim 0 as dynamic

    Under the hood, PyTorch uses SymPy to represent and manipulate symbolic shapes. Each dynamic shape is replaced with a SymInt and tracked by a central ShapeEnv.

    Every operation in PyTorch has a meta function — a lightweight implementation that computes metadata like shape changes without actually performing the computation. This lets PyTorch propagate symbolic shapes through the graph. For example, concatenating two tensors along dimension zero is represented symbolically as:

    s0 = s_x0 + s_y0

    To support branching logic, symbolic shapes carry a “hint” of their current concrete value. This allows specific branches of conditionals like if tensor.size(0) > 2 to be taken during tracing based on the hint. PyTorch adds a guard at this point to ensure that the resulting graph is only used if that branch is the correct one.

    Guards

    Guards are runtime checks inserted into the compiled graph to ensure the assumptions made during tracing still hold. For example, in the case of tensor.size(0) > 2, if the tensor is the result of concatenation, the guard will check that a.size(0) + b.size(0) > 2. If this fails, the code is retraced, and a graph for the new branch generated. Multiple graphs can be cached and selected at runtime based on guard validation.

    Guards don’t need to assert exact sizes; they can use symbolic constraints like x.size(0) > 2. This allows dimensions to vary within bounded ranges. The backend compiler (usually Inductor) can then compile code that operates over symbolic dimensions, as long as the variability is within the guarded constraints.

    For example, operations like broadcasting typically generalize well to symbolic shapes. In contrast, if an op specializes on a fixed shape (e.g., optimized path for 1D input), it may require conditional tracing and guards.

    What this means in practice is that most of the time compilation will follow this process:

    • Take a batch of data, assume all shapes are static, insert guards, and pass static sizes to the compiler
    • On the next batch see which guards have been violated, mark those dimensions as dynamic, add appropriate guards and pass symbolic dimensions to the compiler
    • Assuming no control flow, continue to reuse this dynamic graph for without recompilation

    Backed vs. Unbacked SymInts

    Most symbolic shapes are backed, meaning they have an associated concrete value at trace time. These are usually derived from inputs and show up in traces as s0, s1, etc.

    Unbacked SymInts lack a concrete value. These arise from data-dependent operations, e.g.:

    n = (x > 1).nonzero().size(0)

    Here, n depends on the data in x, so its size cannot be known at trace time. It will be represented as an unbacked SymInt like u0.

    If a control flow decision depends on an unbacked SymInt, tracing cannot proceed, resulting in a graph break or a GuardOnDataDependentSymNode error (when full_graph=True).

    However, you can guide the compiler with additional constraints, e.g.:

    torch._check(x.size(0) == y, lambda: f"size mismatch: {x.size(0)} != {y}")

    This lets PyTorch treat x.size(0) as equivalent to y throughout the graph. The check will be validated at runtime.

    There are other APIs to help mark unbacked SymInts as size-like to enable meta function compatibility (see Ed’s docs for more).

    Controlling Dynamic Shape Usage

    You can control dynamic behavior in torch.compile with the dynamic flag:

    • Not passed: default shape inference behavior
    • dynamic=false: force all shapes to be static
    • dynamic=True: treat all shapes as dynamic

    The default is usually best, but dynamic=True can help in testing.

    Use full_graph=True to attempt to generate a single, complete graph without breaks. This is often critical for performance, as graph breaks can drastically affect runtime and it’s easy to make innocuous looking code changes that can trigger additional breaks!

  • PyTorch and Python Free Threading

    https://trent.me/articles/pytorch-and-python-free-threading/

    Trent Nelson has written an extremely detailed breakdown of his experiments with running inference on GPT-2 on PyTorch and the GIL-free version of Python from 3.13 and 3.14.

    He implements parallel generation using multiple threads (on one GPU and later multiple devices), parallel model loading, and then some of the challenges with torch.compile (which doesn’t work great with nogil yet!)

    Hopefully this encourages more folks to experiment with free-threaded Python, or perhaps port their existing Python packages to play nicely when installed in a free-threaded Python environment. I personally can’t wait until free-threaded Python is the default! Although that’s probably at least five or so years out at this point.

    Free threaded python really changes the performance trade-offs around Python, and I expect it to be the default for ML work a lot sooner than that!

  • Understanding R1-Zero-Like training

    https://arxiv.org/abs/2503.20783

    Further support for elicitation and a very good example of why its worth starting with the evals.

    They started with evals for mathematical reasoning, and then tested base Deepseek, Qwen and Llama models with different templates to see how they did prior to any RL. In doing so they discovered that Qwen did best with no template, and that Deepseek v3 already would create “aha” moments (reasoning self reflection) without any further tuning. Some of the examples are amusing, particuarly the “awkward silence”:

    In Pascal’s Triangle, every row starts
    and ends with 1, …
    …
    This can be calculated as: awkward silence Wait, I’m overthinking. Let’s try again. The number of elements in the first n rows of Pascal’s Triangle…

    The second interesting takeaway for me was that the GRPO implementation (along with most PPO implementations) have a length bias: wrong but long answers are preferred over wrong but short ones. Their corrected, unbiased approach resulted in the same performance, but better token efficiency:

    We also revisit the GRPO’s optimization bias with the Llama base model. The right plot of Fig. 8 compares the model performance and response length trained with GRPO and Dr. GRPO [Ed: the unbiased version]. We can clearly see that GRPO can produce the “double-increase” phenomenon, potentially leading to a misperception that long-CoT can also emerge on Llama models after math pretraining. Unfortunately, the increase of length might be due to the optimization bias

  • Bank Conflicts in Shared Memory

    When data is in the global memory on a GPU it’s usually in row-major or column-major order. Loading from global memory is quite slow though, so for performance we want to move the data to shared memory for the threads in a warp to work on.

    To make that load from global memory performance we want memory reads to be coalesced, meaning we are reading contiguous chunk of memory at a time. Shared memory on the other hand is divided into banks, typically 32 banks which are 4 bytes wide. If multiple threads in the same warp try to write to different addresses in the same bank then the requests are processed sequentially, slowing things down while the threads wait on each other. Nsight and other profiling tools will helpfully point this out to you!

    For example, let’s say we’re loading a row major and column major tensor, and will be doing a multiplication between them (this is naive, to demonstrate the issue):

    __shared__ float Asub[TILE_DIM][TILE_DIM];
    __shared__ float Bsub[TILE_DIM][TILE_DIM];  // (No padding in this naive version)
    int lane = threadIdx.x;  // 0...31 (warp lane index)
     int tileRow = blockIdx.y * TILE_DIM;
     int tileCol = blockIdx.x * TILE_DIM;
    int globalRow = tileRow + lane;
    int globalCol = tileCol + lane;
    Asub[lane][0] = A[globalRow * N + tileCol + 0];
    Bsub[lane][0] = B[(tileRow + lane) + (tileCol + 0) * N];

    Now when we fill Bsub we will be writing everything to the same shared memory bank, significantly slowing things down. One easy fix is just to add padding:

    __shared__ float Asub[TILE_DIM][TILE_DIM];           // A tile (row-major, no conflict in our case)
    __shared__ float Bsub[TILE_DIM][TILE_DIM + PAD];     // B tile (extra column to prevent conflicts)
       

    With PAD as 1 (and TILE_DIM as 32) we have 32×33, or 132 bytes, offsetting the writes and ensuring that each thread gets its own bank.

    The downside is that this wastes shared memory, a scarce resource, so an alternative approach is swizzling: changing the layout such that consecutive thread accesses aren’t causing bank conflicts. That’s what Bert implemented to get performance in his recent GEMM walkthrough, but it’s easy to get it wrong.

    To make life easier than writing it in raw CUDA, Cutlass has a system called CuTE. Cute is a set of templates to express layout of data:

    auto tileLayout    = make_layout(make_shape(Int<32>{}, Int<32>{}), GenRowMajor{});
    auto swizzledLayout = composition(Swizzle<5, 0, 5>{}, tileLayout);

    Here you specify how the data is laid out in global memory with the shape and stride, then make_layout and the copy operation take care of translating from the row-major layout in global memory to the swizzled layout in shared memory.

    From a Triton perspective, Lei Zhang has a great post on memory access, and how it works in Triton, specifically the LinearLayout class that allows the language to similarly handle swizzling and layouts for you:

    Indeed the whole point of LLs is that they allow us to specify transposed and swizzled layouts as a “general case”. Instead of a layout class for registers in a thread, and another layout for registers in a thread but in MMAv2 order, and so on, all of these can be represented by different LLs. This gets rid of special cases and lets us write more general code.

    There’s a great colfax report on building GEMMS that covers shared memory bank conflicts, and Lei Mao has a post with a nice illustration. Axel Feldman also has a post about benchmarking different approaches and identifying bank conflicts, and some more efficient loading techniques.

  • Useful Reasoning Behaviors

    [2503.01307] Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs

    Very useful insight in this paper out of Stanford.

    Test-time inference has emerged as a powerful paradigm for enabling language models to “think” longer and more carefully about complex challenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improvement in language models on verifiable tasks, some models exhibit substantial gains while others quickly plateau.

    The authors were running a reasoning post-training process on both Qwen 2.5 3B and Llama 3.2 3B. They noticed that while both learned, Llama was consistently worse than Qwen, which feels odd as both models are strong. In looking at the reasoning approaches exhibited they observed 4 distinct reasoning strategies:

    • verification
    • backtracking
    • subgoal setting
    • backward chaining 

    They noticed that Qwen exhibited these behaviors more from the base model, and those behaviors were enhanced more in the RL process.

    While the larger Llama-3.1-70B showed generally increased activation of these behaviors compared to Llama-3.2-3B, this improvement was notably uneven — backtracking, in particular, remained limited even in the larger model.

    They then generated some custom reasoning traces that intentionally demonstrated all 4 behaviors, using Claude.

    We generate these datasets using Claude-3.5-Sonnet4, leveraging its ability to produce reasoning trajectories with precisely specified behavioral characteristics. While Claude does not always produce the correct answer (see Fig. 9), it consistently demonstrates the requested reasoning patterns, providing clean behavioral primitives for our analysis.

    They found that when using the SFT set before RL they closed most of the gap between Llama and Qwen. They also found that it isn’t even important that the reasoning traces are correct – demonstrating the behavior is more important than the reasoning itself at the SFT stage.

    Priming models with cognitive behaviors, by a small amount of finetuning, enabled significant performance gains even in models that initially lack these capabilities. Remarkably, this holds even when primed with incorrect solutions that exhibit the target behavioral patterns, suggesting that cognitive behaviors matter more than solution accuracy. 

    This fits with the elicitation idea — the SFT is training a style. By having increased activation of the reasoning styles the RL process is more able to explore these capabilities and reinforce extended reasoning generation.

    This also fits with my mental model that a base model’s capabilities are often pretty underexplored: a combination of targeted SFT + RL seems to be a very powerful elicitation tool!

  • Scarcity & Abundance

    https://alexdanco.com/2025/03/27/scarcity-and-abundance-in-2025/

    Alex Danco back with a rumination on a what is becoming abundant and that that makes scarce. The two takeaways I had were:

    a) How we think about code changes to a more malleabe, more immediate driver of value rather than some inherent store of value itself.

    b) The easiest place to deploy agents is in areas where you are both already doing the work, but also where that work is executed by somewhat lower-accountability folks (e.g. vendors, contractors, maybe very junior staff).

    It might be a bit too flippant to say, “We’re evolving from a mindset where the codebase is capital (the past few decades of software) and into a mindset where code is labor.” But this is a blog post, so it suits the medium. And it suits today’s energy: new projects and startups are writing a lot more code on the basis of “does this make me money now” (what Simon Wardley would’ve once called “worth-driven development”): the codebase is more like a workforce to be trained than like a factory line to be architected. You still want to put thought in it, but it’s a different kind of thoughtfulness. And you expect profit generation out of it a lot more aggressively than with patient capital deployment.

    The two points are connected. Most very large companies have a real but unspoken delineation between different types of code. Some code is absolutely bullet proof, very closely monitored, and staffed for proactive, ongoing maintenance and enhancement. And some code is… not that.

    We generally don’t explicitly call out the gradations, but we do put different degrees of gating around things. Getting better at both the identification and fencing of critical things (load bearing vs decorative, from a business point of view) will be necessary to unlock the pace of potential improvements via agentic systems, and to focus human attention on the areas where there is a very high inherent complexity/ambiguity.

  • Muon optimizer

    Last year Keller Jordan at OpenAI beat some of the existing NanoGPT speedrun records thanks to some optimizer improvements. Towards the end of the year the work was formalized as the Muon optimizer, and it’s making waves in a bunch of areas now

    Friendship ended with Adam, now Muon is my best friend.
    From Elie Bakouch’s great pretraining presentation

    Jeremy Berenstein has written up a great post on how Muon is derived:

    To handle individual layers, our idea is to normalize the weight updates in a clever way so that, given the structure of the inputs, the weight updates automatically induce a desirable effect on the outputs. As a community, we have invested so much effort into thinking about how to normalize the activations: think batch norm, layer norm, RMS norm, etc. Why not also consider how the weights and weight updates influence the activations?

    Keller also wrote a detailed blog post when introducing the optimizer, calling out some open questions (like does it scale to very large training).

    As the posts cover, the optimizer isn’t totally generally – it was designed for linear layers (and flattened convs), so you need to pair it up with Adam for most usage.

    You can install the library from Github: pip install git+https://github.com/KellerJordan/Muon

    from muon import Muon
    
    muon_params = [p for p in list(model.parameters()) if p.ndim > 2]
    muon_param_ids = {id(p) for p in muon_params}
    adamw_params = [p for p in model.parameters() if id(p) not in muon_param_ids]
    # Create the optimizer
    optimizers = [Muon(muon_params, lr=0.001),
    torch.optim.AdamW(adamw_params, lr=0.001)]

    And step both optimizers in the training loop:

    for opt in optimizers:
        opt.step()

    It’s great to have innovation in this area, particularly with this kind of fundamental reasoning around why it works!

  • What if your worry problem is really a planning problem?

    What if your worry problem is really a planning problem? | Psyche Ideas

    Article in Psyche by Paul B Sharp about whether worrying and anxiety is a misfiring planning process:

    My work, which I outlined more formally in a recent paper in Trends in Cognitive Sciences, builds on these earlier ideas and suggests that people suffering from chronic worry engage in biased and/or excessive planning. Biased planning involves inaccuracies in the planning process, which can lead us to imagine events in unrealistic ways that are incongruent with how they actually tend to transpire. Excessive planning involves planning at unhelpful times (such as when a threat is far in the future) or beyond the point when it’s useful. If you’ve ever found yourself catastrophising about the future, or tossing and turning in bed about a challenge that’s months way, then you will be able to relate to these descriptions.

    You can find the preprint of the paper online, which contains terms like state, actions, policy and other bits and pieces probably familiar to many reader!

    The prospective reward comprises the weighted average over possible actions in state s (policy), the new states, s’, those actions transition to, and the values of s’. For anxious individuals, the task construal p(s’|s,a), may be biased in ways that are systematic (select worst-case task transitions) or idiosyncratic (select transitions related to childhood fears).

    Sharp uses a computation approach to mental modelling. He has a new lab which covers the research projects, mainly focused on planning of all kinds.

    Our lab uses diverse methods, including computational models and neuroimaging, to understand human learning and planning. We draw from reinforcement learning and bayesian models of cognition to model these processes and their breakdown in psychopathology.