Blog

  • Scaling RL Compute

    https://gr.inc/blog/scaling-rl-compute/?

    Great post by the folks at General Reasoning on the combination of factors that led to O1-type breakthroughs in inference time compute.

    But here is the key point: no-one suddenly discovered that reinforcement learning was useful for reasoning. It was always useful, but getting some of the details right was the difference between a good post-training recipe and a paradigm shift in the way we use language models.

    ML research is prone to these lollapalooza effects where several positive facts coincide to produce a much larger than expected result. You can go look at the launch of ChatGPT for another example: ChatGPT wasn’t a surprise for folks who had spent time with large language models, and had seen attempts like Galactica before. But for many people it was a remarkable, new experience, and the engagement and interaction ChatGPT saw was new to the researcher community. That itself contributed to further breakthroughs and improvements.

  • Fused Linear Cross-Entropy

    Fused Linear Cross-Entropy is a popular optimization that combines the final linear projection and cross-entropy loss into a single operation. This fusion is very valuable for training large language models efficiently, as it can reduce memory usage significant, particularly for larger vocabularies.

    If you look at a LLM training loop, you generally see something like:

    logits = model(input_ids)
    loss = cross_entropy(logits, targets)

    And if you look at the end of the model, you’ll see something like the below, where h is the hidden state so far and output is output = nn.Linear(embed_dim, vocab_size, bias=False)

    # shape: [b, seq_len, out_dim]
    output = self.output(h)

    That final logics value can be pretty big: sequence length is long and the vocabulary size is large (128k for Llama 3, 202k for llama 4), so logits can be GB of memory: with a 16k context window, a 128k vocab, and 4k embedding dimensions even at a batch size of 1, you get 8bn entries. At BF16, that’s 4GB. You’ll also need to capture the gradient, which will give you another 4GB in the backwards.

    That set of logits has a range of values that are a bit all over the place, one for each of the possible targets.

    Cross-entropy is a loss between two probability distributions. Jay Mody has an excellent blog post breaking down softmax and CE loss

    Roughly speaking, cross entropy measures the similarity of two probability distributions. In the context of neural networks, it’s common to use cross entropy as a loss function for classification problems where:

    • q is our predicted probabilities vector (i.e. the softmax of our raw network outputs, also called logits, denoted as y^), that is q=softmax(y^)
    • p is a one-hot encoded vector of our label, that is a probability vector that assigns 100% probability to the position y (our label for the correct class): pi={1i=y 0i≠y

    This means that cross-entropy simplifies to F.nll_loss(F.log_softmax(x, 1), target)

    Softmax makes our previously messy logits into a nice probability distribution where all the values are positive and sum to one. log softmax is usually used in LLMs, for numerical stability and efficiency.

    When we implement softmax, the naive implementations looks something like:

    out = torch.log(torch.exp(x) / torch.sum(torch.exp(x)))

    This isn’t numerically stable, so you want to subtract the max to avoid overflows and underflows in the exp. This is the common log-sum-exp implementation:

    x_max = torch.max(x)
    shifted_x = x - x_max
    exp_shifted = torch.exp(shifted_x)
    out = shifted_x - torch.log(torch.sum(exp_shifted)

    In general the memory and compute cost of this grows with the size, which gets painful for our hefty logits. We can instead keep a running log-sum-exp so we don’t have to deal with the whole input at once.

    lse = xs[0]
    for x in xs[1:]:
        m = torch.max(torch.stack([lse, x]))
        lse = m + torch.log(torch.exp(lse - m) + torch.exp(x - m))
    out = lse

    This is the online log-sum-exp approach, and makes our life easier! We can now compute incrementally, but we are still generating the big logits before hand.

    Fused Linear Cross-Entropy replaces the output projection, softmax and loss calculation with a single kernel that a tiles across all of it.

    This is the core of the idea: instead of computing all logits at once (which creates a massive tensor), we can:

    1. Compute logits for small chunks of the vocabulary
    2. Compute the softmax incrementally
    3. Only store the logits we need for the loss calculation

    Quoting https://github.com/mgmalek/efficient_cross_entropy

    This repo contains an implementation of a linear projection + cross-entropy loss PyTorch module that has substantially lower memory consumption compared to a standard implementation, with almost no additional compute cost. The memory savings come from two optimizations: 1) overwriting the logits with their gradients in-place and 2) not materializing the entire logits tensor.

    Roughly, the loop looks like:

    For each of the token i in the sequence, with output layer weights h

    • Compute a partial dot product si = hi dot W_tile
    • Reduce for a running max and exp-sum
    • Return only the si[targeti] needed for the loss.

    This gives you quite a lot of memory wins, which also reduce peak memory bandwidth needs. But this also introduces some potential pain!

    1. You’re fusing the final layer op into the loss, which might be defined in different places in your model code
    2. You’re accumulating, so you have to use fp32 or risk introducing numeric errors
    3. You have to write you own backwards op as well, which will generally do some extra computation, so you are paying some extra FLOPS
    4. Debugging can be harder, so you want a good recipe prior to swapping in the op
    5. May require some futzing for best implementations on different hardware.

    Actually implementing is pretty straightforward.

    @staticmethod
    def forward(ctx, h, W, target):
        B, D = h.shape
        V, _ = W.shape
        
        chunk_size = min(1024, V)
        
       # Initialize online softmax accumulators
       max_logits = torch.full((B,), -float('inf'), device=h.device, dtype=torch.float32)
       sum_exp = torch.zeros(B, device=h.device, dtype=torch.float32)
       target_logits = torch.zeros(B, device=h.device, dtype=torch.float32)
            
        # Process vocabulary in chunks
        for chunk_start in range(0, V, chunk_size):
            chunk_end = min(chunk_start + chunk_size, V)
                
            # Compute logits for this chunk only
            W_chunk = W[chunk_start:chunk_end, :]
            logits_chunk = h @ W_chunk.T  # [B, chunk_size]
                
            # Update running max
            chunk_max = logits_chunk.max(dim=1).values
            new_max = torch.maximum(max_logits, chunk_max)
                
            # Adjust previous sum_exp by exp(old_max - new_max)
            sum_exp *= torch.exp(max_logits - new_max)
            
            # Add this chunk's contribution to sum_exp
            sum_exp += torch.exp(logits_chunk - new_max.unsqueeze(1)).sum(dim=1)
            
            # Update max
            max_logits = new_max
                
            # Extract target logits if target is in this chunk
            chunk_indices = torch.arange(chunk_start, chunk_end, device=h.device)
            is_target_in_chunk = (target.unsqueeze(1) == chunk_indices.unsqueeze(0))
            target_logits += (logits_chunk * is_target_in_chunk).sum(dim=1)
        
        # Compute loss: -log(p_target) = -(target_logit - log_sum_exp)
        log_sum_exp = max_logits + torch.log(sum_exp)
        loss = log_sum_exp - target_logits
        
        # Save for backward
        ctx.save_for_backward(h, W, target, max_logits, sum_exp)
        ctx.chunk_size = chunk_size
            
        return loss.mean()

    Here we chunk the vocabulary, calculate the partial transform for the chunk h @ W_chunk.T, do online softmax and accumulate the target logits.

    The backward calculates the gradients:

    @staticmethod
    def backward(ctx, grad_output):
        h, W, target, max_logits, sum_exp = ctx.saved_tensors
        chunk_size = ctx.chunk_size
            
        B, D = h.shape
        V, _ = W.shape
            
        # Scale gradient by batch size (since we use mean reduction)
        grad_scale = grad_output / B
            
        # Initialize gradient accumulators
        grad_h = torch.zeros_like(h)
        grad_W = torch.zeros_like(W)
            
        # Process vocabulary in chunks (same as forward)
        for chunk_start in range(0, V, chunk_size):
            chunk_end = min(chunk_start + chunk_size, V)
            chunk_indices = torch.arange(chunk_start, chunk_end, device=h.device)
                
            # Recompute logits for this chunk
            W_chunk = W[chunk_start:chunk_end, :]
            logits_chunk = h @ W_chunk.T  # [B, chunk_size]
                
            # Compute softmax probabilities for this chunk
            # p_i = exp(logit_i - max) / sum_exp
            probs_chunk = torch.exp(logits_chunk - max_logits.unsqueeze(1)) / sum_exp.unsqueeze(1)
                
            # Gradient w.r.t. logits: grad_logits = p - 1_{y=i}
            grad_logits_chunk = probs_chunk * grad_scale
                
            # Subtract 1 from target positions
            is_target = (target.unsqueeze(1) == chunk_indices.unsqueeze(0))
            grad_logits_chunk -= is_target.float() * grad_scale
                
            # Accumulate gradients
            grad_h += grad_logits_chunk @ W_chunk
                
            grad_W[chunk_start:chunk_end, :] = grad_logits_chunk.T @ h
            
        return grad_h, grad_W, None

    In the backwards we recompute the logits for the chunks, and calculate the logits.

    This is a very simplified implementation that trades off a bunch of kernel launches, so gives up a lot of performance, but you can see the memory savings:

    Regular:
    Time: 285.18 ms
    Memory (total): 3072.0 MB
    Loss: 11.142737
    Chunked online softmax:
    Time: 470.27 ms
    Memory (total): 356.0 MB
    Loss: 11.142738

    For a more sophisticated implementation, you can look at the repo mentioned before or Liger has a good quality kernel with further optimizations. These calculate the gradients in the forward pass, then can just scale them in the backwards. This trades off a bit more memory for less of a compute hit. In general there are a few options for choosing the right point

  • Values in AI

    Daniel Schmachtenberger has made the argument:

    1. All technologies embody value systems
    2. Some technologies are obligate in a competitive environment

    The example of his that stuck with me was the plough: many cultures were animistic (a belief in the spirit of the animal), but after the scaling up of agriculture enabled by the plough, most weren’t. The plough’s enablement of large-scale agriculture likely shifted societies toward sedentism (vs nomadism) and surplus, altering spiritual relationships with animals as they became tools for labor. The perspective shift — the value it encodes — is embedded in the technology.

    The plough is also obligate. If one group uses it and other doesn’t, the group that does will be able to farm more per person. That surplus enables for more specialization, which yields an advantage either in terms of trading or conflict. If the second group doesn’t adopt the plough they will be taken-over, outgrown, or both, by the first.

    AI may well be an obligate technology, which forces us to make deliberate ethical choices about its deployment and values. We are in the early stages of seeing that with software development. That’s going to change the nature of certain careers: changing what the day-to-day work looks like and impacting demand for software engineers. That isn’t necessarily negative: it will depend on the opportunities that replace the current ones. It also isn’t neutral: our approach to AI, how we deploy it, how it is used are all a series of choices that embed values.

    Some of those values are encoded into the models by the training data and loss functions, some are encoded in the systems engineering, the choices of which tasks to apply it to, which interactions to explore and so on, and some are explicitly engineered in through fine tuning and reinforcement learning.

    One way of looking at those values is through the study of ethics, how to live in a just way. This is a core topic for philosophers. One example is Kant’s Categorical Imperative, which requires actions to follow maxims that could be universally applied without contradiction, ensuring rational consistency.

    It’s somewhat akin to asking the question: Would I still support this if I knew everyone else would act this way? Further, would I support this action if knew I would be born again randomly into the world, maybe in a much different situation than my one now?

    The proliferation of useful AI agents adds a somewhat realistic flavor to the question: if, in the future, you are dependent on systems constrained by these specific guidelines or rules , are you happy about that?

    Kantian (or deontological) thinking is far from the only ethical system. A lot of thinking about AI ethics has been consequentialist. Consequentialism is practical: the “goodness” of an action is whether it results in a good outcome! Inherently we judge AI training (at least for RL and supervised learning) by the achievement of the outcome encoded in a loss function, reward function or similar. Stuart Russel (of & Norvig fame from university courses of my youth) has written about “provably beneficial” AI where the AI maximizes a human-involved reward signal (a little like the Assistant Games pattern we discussed before).

    The downside of all this is well documented — Nick Bostrom’s famous paperclip maximizer thought experiment is an AI that achieves the objective, but in a way that was undesirable. A more benign but annoying example might be a cleaning robot that pushes everything outside the house in order to make it tidier. Because outcome-based rules just judge the what, and now the how, they can also encourage power-seeking (as called out by Bostrom) in order to better achieve objectives.

    standard forms of consequentialism recommend taking unsafe actions when such acts maximize expected utility. Adding features like risk-aversion and future discounting may mitigate some of these safety issues, but it’s not clear they solve them entirely.

    Deontology and safe artificial intelligence – William D’Allesandro

    Anthropic’s constitutional AI approach can be seen as a blend of approaches; the constitution is a set of principles that can be used by another AI to criticize and improve output in response to requests:

    As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as ‘Constitutional AI’. 

    The training still ultimately uses a form of reinforcement learning (which is inherently consequentialist), but the reward is given according to how well the outputs adhere to the constitutional principles.

    A more recent philosopher, Derek Parfitt, argued that all moral systems were hill climbing towards a shared perspective, and you can evaluate an action on multiple in order to gain confidence. For example, when considering an option, you could ask:

    a) Would it maximize overall good? (consequentialist)
    b) Could everyone rationally will it? (Kantian)
    c) Could anyone reasonably reject it? (contractualist1)

    “Rationally” here is doing a bit of work: it means “with reasoning”, as in there is a chain of thought that can support and justify the decision.

    Part of the challenge with rationalism is that part of the reward signal here is coming from human raters. We have seen this play out with LMSys where models which are “friendlier” score better, and in a more extreme version in the ChatGPT 4O misalignment where the model became excessively sycophantic in a way that resulted in better rewards in short doses, and didn’t impact any of the quantitative evaluations, despite being an overall negative to the experience.

    As we move into more agentic systems we often have fewer tools to evaluate or make visible the values we are encoding, but we are still doing it!

    For example. Google’s recent AlphaEvolve project uses Gemini underneath, which is an LLM that can be evaluated and aligned. But on top of that it uses an evolutionary search scheme (another reminder of Rich Sutton’s bitter lesson) to generate different prompts and evaluations and iterate towards a new, externally defined goal: in that case generating better algorithms and code. We are searching for superior outcomes, but that search itself is -somewhat unconstrained by other values: it’s a more consequentialist approach.

    The current crop of agentic coding tools often recommends encoding preference data into a project specific file. For example, Claude Code recommends a CLAUDE.md file

    • Include frequently used commands (build, test, lint) to avoid repeated searches
    • Document code style preferences and naming conventions
    • Add important architectural patterns specific to your project
    • CLAUDE.md memories can be used for both instructions shared with your team and for your individual preferences.

    While it presents them as memory, the idea here is to guide the choices of the model in a way that aligns with the principles by which the project being modified is managed.

    OpenAI have published work in allowing hierarchies of instruction: The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions | OpenAI

    we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly defines how models should behave when instructions of different priorities conflict. 

    As well as using a single model that can incorporate different safeguards, we can use models themself to verify actions and outputs. Verification is generally an easier problem than generation, so a model that is unable to consistently follow a set of principles may still be able to validate whether a given example does or does not follow them.

    LlamaGuard is a good example of this kind of system, built and released by Meta’s GenAI team alongside Llama. One example of seeing this process in the wild is OpenAI’s safety systems on 4O image generation. Inherently agentic, 4O can generate image ideas, then the image itself. Despite the model having constraints on it, it will happily generate things which violate OpenAI’s content policy, necessitating a monitoring model that whisks them away before a use can access a violating image.

    If AI becomes an obligate technology, we will benefit from encoding values intentionally, balancing outcomes, universal principles, and fairness. The challenge is ensuring these choices reflect the world we want, not just the one we’re competing in.

    1. Another theory of ethics that weights mutuality heavily: it’s frames ethical considerations as something derived between people rather than just based on outcomes or on abstract principles. Its featured particularly in Scanlon’s What We Owe to Each Other for those, like me, who get all of their ethical understanding from watching The Good Place ↩︎

  • Scientific discovery and AI

    I got fooled by AI-for-science hype—here’s what it taught me

    LinkedIn-esque headline but a good guest post on Tim Lee’s Understanding AI newsletter by physicist Nick McGreivy.

    Later research found that scientists who use AI are more likely to publish top-cited papers and receive on average three times as many citations. With such strong incentives to use AI, it isn’t surprising that so many scientists are doing so.

    So even when AI achieves genuinely impressive results in science, that doesn’t mean that AI has done something useful for science. More often, it reflects only the potential of AI to be useful down the road.

    The problems Nick describes where he find PDE solving (the area he was looking into) had a lot of techniques which didn’t end up improving on non-ML approaches, feels very common. AI research likes to hill-climb metrics. It’s often the lack of progress on a certain benchmark that motivates new techniques, like the growth of test-time compute over the past year to drive math and logic performance higher.

    It brings to mind Zhengdong Wang’s fantastic year-in-review letter from last year.

    The model does the eval is the backbone of how one should access and marshall their intuitions into a coherent view on AI progress.

    The first awesome conclusion of the model does the eval is that we will achieve every evaluation we can state. Recall that evaluations must be legible, fast, and either a good approximation of a wanted capability or useful itself. The plummeting cost of compute has made all evaluations faster.

    […]

    Add human intelligence to direct the cheaper compute to get more legible evaluations. Two years ago, Demis Hassabis enumerated three properties of problems suitable for AI: a massive combinatorial search space, a clear objective function to optimize against, and lots of data or an efficient simulator.

    We tend to succeed where we have the evals and we have the data. Having the evals also starts to create a common lingua-franca to discuss relative performance, not that eliminates the baseline hacking Nick discusses.

    The evals are often tied to having a good quality core data set that can be used for both training and evaluations. Even in areas where we have had scientific progress, mainly AlphaFold and descendants, as Derek Lowe often writes, we have a major leg-up with the existence of the PDB, an extensive database of high-quality protein structures created by people.

    When we look back at major breakthroughs, we often credit that aspect: Dr Fei-Fei Li is one of the pioneers of deep learning thanks in part to the creation of ImageNet. I hope that one takeaway of scientists reading Nick’s note is that the creation of quality benchmarks and datasets can drive more progress than the application of (or innovation on) new ML techniques themselves!

  • Optimizers and Hessians

    https://arxiv.org/abs/2505.02809

    Optimizers are consistently one of the great areas of ML for discovering whether you remember any linear algebra or not (I land on not). Given the pace of change, it’s somewhat surprising that Adam(W) has hung around for as long as it has. Adam updates each parameter by a moving average of the gradient, and a moving average of the squared-gradient (the 2nd moment). Each weight is updated (and the moments/running averages are tracked) separately.

    One area where we have some alternative approaches are the Shampoo family of optimizers. They take a whole block of parameters (usually the weights of a layer ), stores the moving averages of second moments of all the gradients in a block, then transforms them each step. This transform gives an approximation for the of the inverse Hessian of the block. The Hessian is the square matrix representing the second derivative of the loss: telling you how the gradient slope changes, like a curvature map. This is expensive to calculate for the network as a whole, so Shampoo estimates it one layer at a time (ish – it’ll split up big layers), and rotates/transforms each parameters gradient update based on this curvature estimate.

    Empirically either of these work because if you do look at the Hessian of loss in a network, it its basically block diagonal: all of the curvature is within certain blocks, and very little of it is between distant parameters.

    The paper, Towards Quantifying the Hessian Structure of Neural Networks looks into why that is. The paper is dense, but they mainly conclude that the block diagonal pattern will occur as the number of output “classes” increase, with blocks representing classes. Given that a class is a token in LLMs, then modern LLMs are strongly likely to exhibit this structure.

    (and layers end up somewhat aligning with class-wise blocks in most nets).

    In the subsequent analysis, we will show that the number of classes C, instead of the CE loss, is one key factor. Specifically, the near-block-diagonal Hessian structure arises as C→∞ for both the MSE and the CE loss.

    […]

    We emphasize that we do not claim “large C” as the only cause for the near-block-diagonal Hessian structure, but just that it is a sufficient condition. 

    Because output layers generally have a tight class association (e.g. one column per class) this propagates through the model during training. Ideally a block would be “all weights that eventually feed the class”, and the paper shows that training process (on a simple network) pushes different parts of tensors to have stronger associations with specific classes, so you get a kind of local version of the same block diagonal structure.

    This kind of understanding is helpful because it helps explain why doing single-parameter optimizing still works (curvature is very localized), and also points a direction for improving optimizer memory usage:

     II. Understanding Hessian structure can help design new training methods for NNs. For instance, Adam-mini (Zhang et al., 2024b), a recently proposed optimizer, utilizes the near-block-diagonal Hessian structure to cut down 50% memory consumption in Adam. We believe the special Hessian structure can inspire more new optimizers.

    Skimming this paper did make me wonder whether maybe this would also apply to Muon. A new paper, Muon optimizer Accelerates Grokking studies Muon in practice. Grigory Sapnuov wrote up a great summary of the paper, which shows that Muon gets to understanding of the underlying distributions earlier (as shown by increases is eval-set validation).

    The authors speculate about what exactly in Muon helps grokking. Spectral norm constraints and second-order cues steer the model away from simple memorization and help discover the true pattern.

    Muon operates on 2D tensors only (and is usually mixed with Adam), and uses a transform called Newton-Schulz which takes the directions from the gradient upgrade but makes the singular values of the update equal in each direction, meaning we update the same effective distance based on the local curvature. It also operates one step at a time, rather than storing the moving average. This means it operates a bit like a simplified shampoo, but is even more efficient — so again benefits from the fact that you can largely ignore the geometry outside the layer it’s looking at!

  • Pyrefly

    https://pyrefly.org

    I’m at PyCon (mildly awkward photo thanks to Simon Willison!) and earlier had to steal some extra chairs for the Typing Summit as it was full up! There is a lot of energy and interest around type checking, thanks to Astral’s Ty and Meta’s Pyrefly projects coming in to the space recently.

    While the playground is great to try it, I wanted to see what it was like on a larger codebase I was familiar with. I decided to try TorchTune, which makes use of types, but doesn’t configure a typechecker explicitly for CI (as far as I know!), relying on the LSP to show squiggles as the main type hinting feedback (which is reasonable!)

    I tried running mypy over it with a very basic config, and time mypy

    [mypy]
    python_version = 3.13
    ignore_missing_imports = True
    warn_unused_ignores = True
    strict_optional = True
    files = .

    Unsurprisingly, there are quite a few errors!

    Found 1361 errors in 211 files (checked 485 source files)
    real 10m45.596s
    user 0m12.906s
    sys 0m0.929s

    I installed pyrefly and init’d it:

    pip install pyreflypyrefly init

    This created a pyrefly.toml containing a very minimal config:

    project_includes = ["."]
    python_version = "3.13.0"

    pyrefly check then gave me

    INFO 2,966 errors shown, 7 errors ignored, 485 modules, 7,364 transitive dependencies, 3,522,743 lines, took 47.94s (checking 34.88s; reporting 12.98s), peak memory physical 863.6 MiB

    It’s impressively fast: 10 minutes for mypy vs under 50 seconds for pyrefly. There are also a lot more errors, and it’s tricky to tell whether they’re false positive from pyrefly, skipped errors from mypy, or something else. I scoped it down to the TorchTunes KV cache module in torchtune/modules/kv_cache.py to get a better look.

    There pyrefly returns 9 errors, and mypy 17, but that’s caused by slightly different ways of capturing some of the same errors from what I can see. For example:

    k_out[:, :, self.cache_pos[:seq_len]] = k_val

    This code is doing a bit of Tensor slicing:

    • cache_pos is a max_seq_len long tensor holding absolute write positions
    • k_out is the key cache, with shape batch_size x num_heads x max_seq_len x head_dim
    • Here we’re getting a view for the part of the cache we want to update, and appending the latest values

    MyPy gives these errors:

    torchtune/modules/kv_cache.py:104: error: "Tensor" not callable [operator]
    torchtune/modules/kv_cache.py:104: error: Value of type "Tensor | Module" is not indexable [index]

    While pyrefly gives:

    torchtune/modules/kv_cache.py:104:9-46: Item assignment is not supported on Module | Tensor   Expected __setitem__ to be a callable, got BoundMethod...

    In this module cache_pos and k_cache are created by calling PyTorch’s register_buffer, which stores params for use in the state_dict but doesn’t use them for training. Buffers don’t have to be Tensors, so I am guessing the type doesn’t propagate well there. Adding explicit type declarations in the class body fixes the errors in both mypy and pyrefly.

    cache_pos: torch.Tensor
    k_cache: torch.Tensor

  • The First Year of Free-Threaded Python

    https://labs.quansight.org/blog/free-threaded-one-year-recap

    Quansight have been doing a tonne of work on the roll-out of Free Threaded Python, and they released an insightful blog post covering the progress over the last year that complements the recent Meta eng blog post well.

    At this time last year, when Python 3.13.0b1 shipped, the wider ecosystem of Python packages was more or less completely broken on the free-threaded build. Trying to pip install anything but the simplest package with no dependencies or only pure-Python dependencies would likely lead to build errors. Most of these issues were not due to fundamental problems but because of unsupported default options or minor assumptions broken on the free-threaded build.

    Together with package maintainers and other contributors in the community, we have fixed many of these issues and today things are much better. With the release of Cython 3.1.0, which ships official support for the free-threaded build, we also helped fix one of the most significant sources of build issues.

    There have been a number of good talks at PyCon US this year around Free-Threaded, and lots of familiar stories of rediscovering the concurrency principles that other languages have worked on in the process! I’d recommend keeping an eye on the PyCon YouTube Channel for talk recordings, particularly Lisandro and Nathan’s talk about their journey, David Hewitt’s talk about using Rust with FT Python from his experience with PyO3, and Alvaro Duran’s on his FT Python load balancer!

    Some useful links I’ve been looking at from the talks:

  • Generative modelling in the latent space

    Generative modelling in latent space – Sander Dieleman

    Fantastic deep dive into the concept of latents and the tradeoffs around them by Sander Dieleman of DeepMind. It’s a long article, but there’s a conclusions section that pulls out some of the most interesting points, and each section is an expansion on those points.

    • Latents add complexity, but the computational efficiency benefits are large enough for us to tolerate this complexity – at least for now.
    • Three main aspects to consider when designing latent spaces are capacity (how many bits of information are encoded in the latents), curation (which bits from the input signals are retained) and shape (how this information is presented).
    • Preserving structure (i.e. topology, statistics) in the latent space is important to make it easy to model, even if this is sometimes worse from an efficiency perspective.
  • How does Triton do Warp Spec?

    Kapil Sharma from the PyTorch team has a great series of posts diving into the Triton compiler process: 1, 2, 3. As covered there, Triton lowers to a series of intermediate representations, and each level has a set of transformational passes that implement optimizations. TTIR is the generic Triton IR and leverages a number of standard MLIR passes like common subexpression elimination, as well as some Triton specific passes like managing broadcast ops. That’s then lowered to TTGIR, a GPU-specific IR1

    triton/third_party/nvidia/backend/compiler.py at rc/3.3.x · triton-lang/triton

    The different backends configure the passes appropriate for their targets, so the Nvidia TTGIR configuration above details the passes for Nvidia hardware. Some are gated on the specific backend targeted, like warp specialization:

    passes.ttgpuir.add_ws_task_partition(pm, opt.num_consumer_groups)
    passes.ttgpuir.add_taskid_propagate(pm, opt.num_consumer_groups)
    passes.ttgpuir.add_ws_data_partition(pm, opt.num_consumer_groups)
    passes.ttgpuir.add_ws_code_partition(pm, opt.num_buffers_warp_spec, opt.num_consumer_groups,
    opt.reg_dec_producer, opt.reg_inc_consumer)
    passes.ttgpuir.add_pipeline(pm, opt.num_stages, dump_enabled)
    passes.ttgpuir.add_ping_pong_sync(pm, opt.num_consumer_groups)
    passes.ttgpuir.add_ws_lowering(pm, opt.num_consumer_groups)

    Another example is using the Tensor Memory Accelerator for async loading on Hopper+

    if capability // 10 >= 9:
       nvidia.passes.ttnvgpuir.add_tma_lowering(pm)
       nvidia.passes.ttnvgpuir.add_fence_insertion(pm)

    For a quick recap of the why and how of Warp Specialization, check Colfax’s guide to optimizing a GEMM:

    The most basic method by which GPU programmers can create overlapping is via excess warps (warps are groupings of 32 contiguous threads). Nvidia GPUs allow a large number of warps per SM (streaming multiprocessors), and can switch between them with minimal overhead. In particular, the warp schedulers can simply switch to another warp if one warp encounters a slow memory fetch. In order to give the warp schedulers more opportunity to hide latency, a technique called warp-specialization was introduced circa 2011 [1, 2]. With warp-specialization, some warps are dedicated to memory fetches (producers), while others are dedicated to compute (consumers), and named barriers are used for synchronization between them. The idea is that the warp schedulers can then more easily hide the latency of copy operations within compute (and vice-versa).

    Even more generally than overlapping memory loads you can overlap other kinds of work. SMs have 1 Tensor Core to 32 ALUs (Cuda Core) (x4 on recent hardware). This means that you can overlap other kinds of work, stuff that isn’t dot products. It’s really common to want to load memory, do a matmul then apply a pointwise function like a relu or other activation function. You aim to keep the Tensor Core as busy as possible with a series of matmuls, and warp specialization lets you do that.

    The transforms to implement this are implemented in triton/lib/Dialect/TritonGPU/Transforms at rc/3.3.x · triton-lang/triton

    The first task partitions ops in the kernel. This task looks for load ops and dot product ops, and partitions them into producer (loads) and consumer (process) groups.

    // Step 1. Select loads into the first task, which is the producer task by
    // default. Place dots into the second task, which is the consumer.
    // Only consider loads that are connected to a dot op in a loop.

    The next task is a bookkeeping one to propagates the task IDs, so if there are unlabeled ops they are attached one of the partitions.

    The WSDataPartition transform partitions the dot operations into consumer groups. It splits the dot products inside loops along M or N dimensions to be processable within a warp group, ensuring all dot operations are sliced and the slices labelled with task_ids.

    Just to look at some of the numbers: A warp consists of 32 threads, and a sync op (for loading) uses a whole warp. A warp group is a set of 4 warps: this is pertinent because the TensorCore MMA prefers 4 warps working on 64x(64/128/256)xK tiles. Triton already has a WarpGroupDotOp that tries to set this up, and that’s one of the operations targeted in this pass. The pass splits a Triton CTA tile, which may be 128×256, so that each consumer warp group has a 64 row (or 256 column) chunk.

     if (sliceSizeM >= 64) {
          LLVM_DEBUG({ LDBG("partition along M\n"); });
          partitionDim = 0;
          partitionSize = sliceSizeM;
          partitionOperand = opndA;
        } else if (sliceSizeN >= 256) {
          LLVM_DEBUG({ LDBG("partition along N\n"); });
          partitionDim = 1;
          partitionSize = sliceSizeN;
          partitionOperand = opndB;
        } else {
          LDBG("partition not possible: " << sliceSizeM << " " << sliceSizeN);
          return false;
        }

    The next pass is WSCodePartition. This is a big transform. It takes the task-sliced IR from the DataPartition and sets up the producer warp group to copy from global GPU mem to SMEM (or on blackwell TMEM). It also drops in barriers using product.acquire/cpmmit and consumer.wait/release to ensure proper ordering between the groups. The transform identifies “channels”, places where data is loaded (using load or descriptor_load for TMA on Hopper+) and associates the producer task_id with all the consumer task IDs that need to process that data.

    Conceptually, the transform is turning the loops in the original kernel into something like this:

    for k: # K-dimension tiles)
        ##### PRODUCER #####
        producer.acquire(token, idx, phase) # reserve smem[idx]
        async_copy_global_to_local(smem[idx]) # GMEM → SMEM[idx]
        producer.commit(token, idx) # make slot visible
    
        #####  CONSUMERS (run in parallel warps) #####
        ## Consumer 0 ##
        consumer.wait(token, idx, phase) # sleep until slot ready
        mma_sync(accum0, smem[idx], …) # read-only, do matmul
        consumer.release(token, idx) 
    
        ## Consumer 1 ##
        consumer.wait(token, idx, phase)
        mma_sync(accum1, smem[idx], …)
        consumer.release(token, idx)
    
        # Repeat for extra consumers. 
    
        # increment circular buffer
        idx   = (idx + 1) % numBuffers 
        # Toggle each time we hit producer, indicate old vs new data.
        phase = phase ^ (idx == 0)
    }

    The next pass is a generic pipeline pass. Each op is assigned a stage based on latency: e.g. slower ops like loads go into stage 0. This is then transformed with modulo scheduling. Any sync ops are converted to async variants (in lowerLoops), before writing out (in expandLoops) prologue, kernel and epilogue loops that contain all the relevant ops.

    Finally the WSLowering pass takes the various operators we have (like producer.acquire) and replaces them with the hardware specific variants (e.g. wait_barrier). It also handles the bookkeeping like generating the warp group and task_ids from the warp ID:

    OpBuilder builder(op);
    Value _4 = builder.create<arith::ConstantIntOp>(loc, WARPS_PER_TASK, 32);
    Value warpId = builder.create<ttng::GetCanonicalWarpIdOp>(loc);
    Value asyncTaskId = builder.create<arith::DivUIOp>(loc, warpId, _4);
    op.getResult().replaceAllUsesWith(asyncTaskId);

    This is a wordy IR way of saying

    warpId = gpu.thread_id().x / 32 
    task_id = warpId / 4

    Now the code is ready for lowering to regular PTX, and all of the warp-specific stuff is captured explicitly!

    1. Note: Because of some project weirdness, warp specialization is quite different in the release branches from main, so I’ll refer to 3.3 from here on. It’s in very active development (by teams at Meta, OpenAI and Nvidia!) so the specifics are quite likely to change over coming releases! ↩︎
  • The Mold Linker

    rui314/mold: Mold: A Modern Linker 🦠

    Having discussed this a bit at work, I finally got round to checking out the project. The README has good examples of what it is, and why to care.

    mold is a faster drop-in replacement for existing Unix linkers. It is several times quicker than the LLVM lld linker, the second-fastest open-source linker, which I initially developed a few years ago. mold aims to enhance developer productivity by minimizing build time, particularly in rapid debug-edit-rebuild cycles.

    Looking at fundamentals with a performance-oriented perspective and an awareness of modern systems continues to be a source of remarkable opportunities. At the same time, getting something so foundational established takes time and awareness.

  • LSP & Standards

    https://www.michaelpj.com/blog/2024/09/03/lsp-good-bad-ugly.html

    Having recently spent a lot more time around typecheckers, I’ve been reading about the Language Server Protocol that glues IDEs and language support services together. The article, from September last year, gives a breakdown of the good and the bad about the protocol, and is a really great dive into the broader topic.

    Much of the pain stems from how the protocol emerged and is managed:

    There is zero open discussion of features before they are added to the spec. Typically they are implemented in VSCode, and then the specification is updated as a fait accompli to document those changes. Implementers of open-source language servers get very influence on the development of the specification.6 There is not even a community space for implementers of language servers to get together and talk about the many tricky corners.

    I feel echoes of this in a lot of different projects I have been around, including PHP internals, ZeroMQ’s protocol, various CNCF working groups, PyTorch and Triton. Protocols and technologies emerge from a need, and grow because that need is shared, but transitioning from a narrow and highly connected problem source to a true standard is difficult: attempt to standardize and bring in voices too early and you just slow down progress to the point something else emerges which solves immediate needs better; leave it too long and the governance questions can be sufficient to encourage folks to rally around forks or alternatives.

    One example of that playing out at the moment is Anthropic’s (and dsp’s!) Model Context Protocol. Tim Kellog wrote a nice post the other day comparing it to OpenAPI, concluding:

    Standards are mostly sociological advancements. Yes, they concern technology, but they govern how society interacts with them. The biggest reason for MCP is simply that everyone else is doing it. Sure, you can be a purist and demand that OpenAPI is adequate, but how many clients support it?

    The reason everyone is agreeing on MCP is because it’s far smaller than OpenAPI. Everything in the tools part of an MCP server is directly isomorphic to something else in OpenAPI. In fact, I can easily generate an MCP server from an openapi.json file, and vice versa. But MCP is far smaller and purpose-focused than OpenAPI is.

  • Scalably Solving Assistant Games

    Scalably Solving Assistance Games | OpenReview

    Assistant games are an RL approach where the assistant and human cooperate on achieving a goal, and receive a reward signal for the joint effort. This paper proposes them as a better mechanism for aligning models in post-training than RLHF.

    Normally, RLHF is focused on single responses, or a single “turn” or interaction:

    Assistance games avoid the aforementioned drawbacks of RLHF by explicitly accounting for both the interactive nature of assistance and uncertainty about the user’s goal. In particular, an assistance game is a two player game in which an assistant and a user take actions in a shared environment. The two agents share a reward function, but crucially the assistant is initially uncertain about it.

    Assistance games remove incentives for deception since the assistant’s performance depends on the true latent reward function, rather than human feedback. They also incentivize the assistant to interact with the user to resolve its uncertainty about the reward function.

    The paper uses building structures in Minecraft as the learning environment and get some very positive results. They mention the possible applications for chatbot alignment as a post-script.

    Practically this requires, given chat history h, predicting:

    • the next assistant message (or tool call)
    • the next human message in response to that
    • how satisfied the human is with the response

    The algorithm does a tree search, trying various different replies and responses and picks the assistant action which showed up the best. They generally sampled ~100 actions in the paper.

    In the Minecraft example, they can see whether a placed or removed block moves the shape towards the human target, so they can give a reward score each step; doing the same thing with conversations might need some clever goal crafting or propagating back from only a final reward signal.

  • Richard Zou on torch.compile

    GPU MODE 4/19 Q&A – Google Docs

    PyTorch compiler engineer Richard Zou did a great Q&A session with the GPU Mode discord community recently. You can watch the session on YouTube, but Richard also collected questions into a doc with some nice snippets and references.

    Our value proposition: You can sit down for hours/days/weeks tuning a custom kernel. torch.compile provides good baseline performance so you don’t need to do that all the time!

    The goal with the compiler is that you can spend most of your time thinking about the model, get the majority of the speedups, and only have to go down to custom kernel authoring when you’ve established an opportunity or need for further performance.

  • Autotuning in PyTorch & Triton

    torch.compile offers some knobs for controlling the trade-off of execution performance with longer compile times. This is particularly useful for inference, where the same model will be running for a long time.

    model_autotune = torch.compile(model, mode="max-autotune")

    Passing the max-autotune option to instructs the compiler to test more options for the operations. The compiler has the option to use pre-built aten kernels, leverage kernels from libraries like CuDNN or Cutlass, or use templated Triton kernels. When autotuning, specific variants are tested on device with the shape information identified during tracing, and the fastest options are selected. Thanks to Triton templates, it can also use options like fusions where pointwise ops can be fused into a single kernel via a Triton template, saving kernel launch overhead.

    The downside of this is that testing the options takes more time, so using max-autotune can lead to some very extended compile times. You also need a hefty enough GPU to get the benefit: is_big_gpu gates it on the number of SMs, so it works best on a 3090, V100 or above.

    You can see a lot of the autotuning options in _inductor/config.py. Backends that are considered are set separately for GEMMs and convolution ops:

    max_autotune_gemm_backends = os.environ.get(
        "TORCHINDUCTOR_MAX_AUTOTUNE_GEMM_BACKENDS", "ATEN,TRITON,CPP"
    ).upper()

    Each kernel has implementations using for the different backends which are added to possible choices. e.g. in _inductor/kernels/mm.py you can see calls to use_[backend]_template that verify whether the backend in question is a choice:

    if is_nonzero and use_cutlass_template(layout, m, n, k):
            CUTLASS3xGemmTemplate.add_cutlass_gemm_choices(choices, layout, [mat1, mat2])

    _inductor/select_algorithm.py does the actual benchmarking through the choices.

    If you run autotuning, you’ll get some log output, and caches will be written to /tmp/torchinductor_yourusername.

    We can try this out on a simple MLP:

    import torch, time
    
    class SimpleMLP(torch.nn.Module):
        def __init__(self, in_features, hidden_features, out_features):
            super().__init__()
            self.linear1 = torch.nn.Linear(in_features, hidden_features)
            self.relu = torch.nn.ReLU()
            self.linear2 = torch.nn.Linear(hidden_features, out_features)
        def forward(self, x):
            return self.linear2(self.relu(self.linear1(x)))
    
    # Set up device and model
    device = 'cuda'
    model = SimpleMLP(in_features=1024, hidden_features=1024, out_features=1024).to(device)
    x = torch.randn(256, 1024, device=device)  # batch of 256, 1024 features each
    
    # Compile the model in default mode and max-autotune mode
    model_default = torch.compile(model, mode="default")
    
    # Warm-up runs (to trigger compilation)
    torch.compiler.reset()
    with torch.no_grad():
        model_default(x)
    torch.cuda.synchronize()  # ensure warm-up completes
    
    # Measure performance of default compiled model
    start = torch.cuda.Event(enable_timing=True); end = torch.cuda.Event(enable_timing=True)
    with torch.no_grad():
        start.record()
        for _ in range(50):
            _ = model_default(x)
        end.record()
    torch.cuda.synchronize()
    time_default_ms = start.elapsed_time(end) / 50.0
    torch.compiler.reset()
    
    model_autotune = torch.compile(model, mode="max-autotune")
    
    with torch.no_grad():
        model_autotune(x)
    torch.cuda.synchronize()  # ensure warm-up completes
    
    # Measure performance of max-autotune compiled model
    start = torch.cuda.Event(enable_timing=True); end = torch.cuda.Event(enable_timing=True)
    with torch.no_grad():
        start.record()
        for _ in range(50):
            _ = model_autotune(x)
        end.record()
    torch.cuda.synchronize()
    time_autotune_ms = start.elapsed_time(end) / 50.0
    
    print(f"Average inference time - torch.compile default: {time_default_ms:.3f} ms")
    print(f"Average inference time - torch.compile max-autotune: {time_autotune_ms:.3f} ms")
    

    Disappointedly, this is the result:

    Average inference time - torch.compile default: 0.113 ms
    Average inference time - torch.compile max-autotune: 3.251 ms

    We can turn on logging with the TORCH_LOG env variable: some useful options are inductor, autotuning, and perf_hints.

    TORCH_LOGS="perf_hints" python tune.py

    You can control many more autotune options via the options flags, though its incompatible with passing a mode value. We can recreate the max-autotune mode, and turn on some useful tracing options like this (note that the options version uses an underscore, the mode a hypen!)

    model_autotune = torch.compile(
    model,
    options={
    "max_autotune": True,
    "triton.cudagraphs": True,
    "coordinate_descent_tuning": True,
    "trace.enabled": True,
    "trace.graph_diagram": True,
    },
    )

    Options "trace.enabled": True, "trace.graph_diagram": True generate trace outputs, and output a nice diagram of the captured graph. Cudagraphs turned out to be the culprit here, which is common enough there is a non-cudagraph mode available to stop you having to remember all the options:

    model_autotune = torch.compile(model, mode="max-autotune-no-cudagraphs")

    As you can see here in the graphs of with and without, the slower version actually has an extra fusion performed!

    Captured graphs for the two runs

    Triton Autotuning

    Triton also conducts autotuning, but it’s a little more explicit. When authoring a Triton kernel you can specify configurations. At compile time each config variant will be tested, the most performant one picked and the choice stored for future calls. A key value can be provided to indicate when to re-autotune based on changing inputs:

    import os
    import torch
    import triton
    import triton.language as tl
    
    # Just to save passing this on the command line
    os.environ["TRITON_PRINT_AUTOTUNING"] = "1"  
    
    @triton.autotune(
        configs=[
            triton.Config({'BLOCK_SIZE': 128}, num_warps=4,  num_stages=2),
            triton.Config({'BLOCK_SIZE': 256}, num_warps=8,  num_stages=2),
        ],
        key=['N']            # re‑tune only if the length N changes
    )
    @triton.jit
    def vecadd_kernel(x_ptr, y_ptr, out_ptr, N, BLOCK_SIZE: tl.constexpr):
        pid   = tl.program_id(0)
        offs  = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)
        mask  = offs < N
        x     = tl.load(x_ptr  + offs, mask=mask, other=0.0)
        y     = tl.load(y_ptr  + offs, mask=mask, other=0.0)
        tl.store(out_ptr + offs, x + y, mask=mask)
    
    def vec_add(x: torch.Tensor, y: torch.Tensor):
        assert x.is_cuda and y.is_cuda
        N   = x.numel()
        out = torch.empty_like(x)
        grid = (triton.cdiv(N, 128),)         # 128 = smallest BLOCK_SIZE we declared
        vecadd_kernel[grid](x, y, out, N)    
        return out
    
    x = torch.randn(1 << 20, device="cuda")   # 1 048 576 elements
    y = torch.randn_like(x)
    
    _ = vec_add(x, y)  # first call → autotuning prints to stdout
    _ = vec_add(x, y)  # second call → no autotuning, uses the best config found
    

    Setting the env variable TRITON_PRINT_AUTOTUNING documents the process as it goes:

    Autotuning kernel vecadd_kernel with config BLOCK_SIZE: 128, num_warps: 4, num_ctas: 1, num_stages: 2, num_buffers_warp_spec: 0, num_consumer_groups: 0, reg_dec_producer: 0, reg_inc_consumer: 0, maxnreg: None
    Autotuning kernel vecadd_kernel with config BLOCK_SIZE: 256, num_warps: 8, num_ctas: 1, num_stages: 2, num_buffers_warp_spec: 0, num_consumer_groups: 0, reg_dec_producer: 0, reg_inc_consumer: 0, maxnreg: None
    Triton autotuning for function vecadd_kernel finished after 0.44s; best config selected: BLOCK_SIZE: 128, num_warps: 4, num_ctas: 1, num_stages: 2, num_buffers_warp_spec: 0, num_consumer_groups: 0, reg_dec_producer: 0, reg_inc_consumer: 0, maxnreg: None;

    You can use the same do_bench tester that the autotuner does, and see how the performance varies yourself:

    import torch, triton, triton.testing as tt
    import triton.language as tl
    
    @triton.jit
    def vecadd_kernel(x_ptr, y_ptr, out_ptr, N, BLOCK_SIZE: tl.constexpr):
        offs  = tl.program_id(0) * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE)
        mask  = offs < N
        tl.store(out_ptr + offs,
                 tl.load(x_ptr + offs, mask=mask) +
                 tl.load(y_ptr + offs, mask=mask),
                 mask=mask)
    
    # tensors
    N   = 1 << 20
    x   = torch.randn(N, device='cuda')
    y   = torch.randn_like(x)
    out = torch.empty_like(x)
    
    def bench(block_size, num_warps):
        grid = (triton.cdiv(N, block_size),)
        # tt.do_bench returns [median, p20, p80] in micro‑seconds
        return tt.do_bench(
            lambda: vecadd_kernel[grid](x, y, out, N, BLOCK_SIZE=block_size, num_warps=num_warps),
            warmup=5, rep=16, return_mode="all", quantiles=(0.5, 0.2, 0.8)
        )
    timings = {
        "128/4": bench(128, 4),
        "256/8": bench(256, 8),
    }
    
    print("timings:", timings)
    

    Running that gives shows that both kernels are basically equivalent, but the first one is slightly faster over the 16 runs.

    timings: {'128/4': [0.01945599913597107, 0.01945599913597107, 0.02028159946203232], '256/8': [0.01945599913597107, 0.01945599913597107, 0.020479999482631683]}

  • Profiling Triton

    There are a couple of different options to profile a Triton kernel.

    Proton

    Proton is the profiler that ships with Triton (profiler for triton). You can enable it and (optionally) activate/deactive around specific regions you want to profile. You have the ability to annotate functions with specific metrics as well.

     session = proton.start()  # Start profiling session
    
    bias = torch.rand((256,), device='cuda', dtype=torch.float16)  # Bias vector
    flops = 2 * M * N * K
    bytes_accessed = A.element_size() * M*K + B.element_size() * K*N + C.element_size() * M*N  # rough bytes
        with proton.scope(f"fused_gemm_bias_relu [M={M}, N={N}, K={K}]", {"flops": flops, "bytes": bytes_accessed}):
            fused_gemm_bias_relu[grid](  
                A, B, C, bias, 
                M, N, K, 
                A.stride(0), A.stride(1), B.stride(0), B.stride(1), C.stride(0), C.stride(1),
                BLOCK_M=BLOCK_M, BLOCK_N=BLOCK_N, BLOCK_K=BLOCK_K
            )
    
     proton.finalize() 

    The output can be visualized with the built-in viewer:

    proton-viewer -m time/ms,tflop/s ./proton.hatchet

    0.040 6.645 ROOT
    ├─ 0.004 nan _ZN2at6native55_GLOBAL__N__11f7a751_22_DistributionUniform_[...]_15PhiloxCudaStateESH_SI_
    └─ 0.037 7.284 fused_gemm_bias_relu [M=1024, N=256, K=512]
    └─ 0.037 nan fused_gemm_bias_relu

    In this case you can see both the (trimmed!) generated name for the bias tensor set up as well as the name of my custom kernel.

    nsight-compute

    Nvidia also have a good range of tools for looking at performance. Note will need to enable access to the counters on device for this:

    NVIDIA Development Tools Solutions – ERR_NVGPUCTRPERM: Permission issue with Performance Counters | NVIDIA Developer

    On the offchance you’re doing this on WSL, https://peterchng.com/blog/2024/03/02/profiling-cuda-programs-on-wsl-2/ walks through the set up!

    Nvidia ships nsight system which tracks larger system wide metrics, and nsight compute which is more focused on profiling execution. You can run it against a script like so:

    ncu -o profile_results python test.py

    The tool comes with a nice GUI for inspecting the results. It can show you the PTX or SASS source for the kernels, offers metrics like actively used registers (good for checking on register spilling), and calls out warnings on poor utilization or memory clashes.

    Upcoming intra-kernel profiler

    [tool][proton] Intra kernel profiling support by fywkevin · Pull Request #4861 · triton-lang/triton

    There is an extension coming for Proton that enables profiling within kernels. This reserves a pre-allocated buffer on device and logs metrics locally, for reading out at the end of the execution. It outputs as a chrome trace for use within a wide range of dev tools. While this isn’t merged into mainline yet, you can see an example of the usage in the dev repo.