Blog

  • Thunderkittens /GPUs go brr

    Just got round to reading the intro post to the (now improved) thunderkittens kernel DSL.

    https://hazyresearch.stanford.edu/blog/2024-05-12-tk

    Many good nuggets on kernel writing in general and the hopper in particular.

    But to us a “register” is a 16×16 tile of data. We think AI wants this — after all this time, it’s still just matrix multiplies, reductions, and reshapes. And we think the hardware wants this, too — small matrix multiplies are just begging for hardware support beyond just the systolic mma.

    In fact, more broadly we believe we should really reorient our ideas of AI around what maps well onto the hardware. How big should a recurrent state be? As big can fit onto an SM. How dense should the compute be? No less so than what the hardware demands. 

  • Generalization progress in RLHF

    https://swtheking.notion.site/4e0cbb325aaf458da710f0b36dbb239c?v=c9231e8c988b4d66a1d2dc34df4cf7b5

    Breakdown of where generalization capabilities come from in RLHF – even more pertinent with the use of this for reasoning capabilities. Also includes one of the best concise overviews of RLHF I’ve read.

    Fine-tuning with RL: Given a reward model, we employ RL to fine-tune the policy of a LLM. First, the policy is a language model that takes in a prompt and returns a sequence of text (or just probability distributions over text). The action space of this policy is all the tokens corresponding to the vocabulary of the language model and the observation space is the distribution of possible input token sequences, which is also quite large given previous uses of RL (the dimension is approximately the size of vocabulary ^ length of the input token sequence). The reward function is a combination of the preference model and a constraint on policy shift. Finally, the update rule is the parameter update of the policy from PPO that maximizes the reward metrics in the current batch of data [3].

  • Ways to use torch.export

    Ways to use torch.export

    Part 2 of the series, covers export, AOTInductor for getting runnable models on the server and ExecuTorch for runnable models on edge device (phones, wearables etc). There are a number of good examples from real world experience in there of how to use these tools as well. As usual I learned about something I didn’t know existed, in this case it was intermediate tensor logging in AOTI:

    AOTInductor has an option to add dumps of intermediate tensor values in the compiled C++ code. This is good for determining, e.g., the first time where a NaN shows up, in case you are suspecting a miscompilation.

  • TLParse

    https://github.com/ezyang/tlparse – or pip install tlparse

    Ed Yang’s Torch Logs Parser gets used a lot within Meta, where it has a bunch of extra integrations to make it even more helpful. Its still useful everywhere else too when working with torch.compile, particularly if approaching a more complex model that generates a lot of log output or trying to get a feel for performance issues.

    Basic usage is just:

    TORCH_TRACE=/tmp/my_traced_log python module.py
    tlparse /tmp/my_traced_log/filename.log -o tl_out/ --overwrite

    The result breaks down the log into a number of easier to consumer sections for different times the analysis restarted, graph breaks etc., and gives you chromium perf trace files for looking at performance.

    A screenshot of the build products from tlparse
  • TIL: TunableOp in PyTorch

    I wasn’t aware of this particular autotuning lever! There is a breakdown of TunableOp on the AMD blog from back in July:

    https://rocm.blogs.amd.com/artificial-intelligence/pytorch-tunableop/README.html

    Instead of using the default GEMMs, TunableOp will search for the best GEMMs for your specific environment. It does so by first querying the underlying BLAS library for a list of all solutions for a given GEMM, benchmarking each of them, and then selecting the fastest. TunableOp then writes the solutions to disk, which can then be used on subsequent runs. 

    Though the infrastructure is generic, this is effectively an AMD-specific tuning tool right now, as mentioned in the original docs.

    Currently only a TunableGemm for ROCm is implemented. Note that CUDA builds of PyTorch will function correctly when using TunableOp but the only solution available to CUDA builds is the ‘Default’ implementation i.e. the original cuBLAS default, now called through TunableOp.

  • nonzero_static in PyTorch

    Natalia Gimelshein added cuda support for nonzero_static to PyTorch the other day — its a feature Jax has had for a while that lets you avoid a bit of data dependent annoyingness.

    I most often see nonzero pop up in logs: its underlies a lot of boolean mask operations and torch.where. A downside of torch.nonzero is that we don’t know the size of the returned tensor. The shape is data dependent, which causes pain for the compiler, and when running on an accelerator requires a device to host sync. This can significantly slow down otherwise fast operations.

    nonzero_static overcomes this by allowing you to supply a shape for the output — if the actual result is smaller it is padded, if larger the result is truncated. The PR linked above enables that for CUDA. You can’t seamlessly use it with bool masks or torch.where, but you can easily replace the call with one to nonzero_static.

    To see the difference, imagine we have a big tensor where we have some sense of the output shape, for example searching for a one-hot index like this:

    import torch
    
    # Set device
    device = torch.device("cuda:0")
    
    # Generate a large tensor of 0s with exactly one 1 at a specific index on the GPU
    size = 100_000_000
    large_tensor = torch.zeros(size, device=device, dtype=torch.long)
    large_tensor[size // 2] = 1  # Place a single 1 in the middle of the tensor
    
    # Record GPU events to ensure asynchronicity
    start_event = torch.cuda.Event(enable_timing=True)
    end_event = torch.cuda.Event(enable_timing=True)
    
    start_event.record()
    ones_indices = torch.nonzero(large_tensor)
    end_event.record()
    torch.cuda.synchronize()
    
    elapsed_time_ms = start_event.elapsed_time(end_event)
    print(f"torch.nonzero() execution time: {elapsed_time_ms:.2f} ms")
    print("Indices of 1s:", ones_indices.cpu())  

    That gives us (on my laptop)

    torch.nonzero() execution time: 177.58 ms
    Indices of 1s: tensor([[50000000]])

    If we modify it to use nonzero_static:

    import torch
    
    # Set device
    device = torch.device("cuda:0")
    
    # Generate a large tensor of 0s with exactly one 1 at a specific index on the GPU
    size = 100_000_000
    large_tensor = torch.zeros(size, device=device, dtype=torch.long)
    large_tensor[size // 2] = 1  # Place a single 1 in the middle of the tensor
    
    # Record GPU events to ensure asynchronicity
    start_event = torch.cuda.Event(enable_timing=True)
    end_event = torch.cuda.Event(enable_timing=True)
    
    start_event.record()
    ones_indices = torch.nonzero_static(large_tensor, size=1)
    end_event.record()
    torch.cuda.synchronize()
    
    elapsed_time_ms = start_event.elapsed_time(end_event)
    print(f"torch.nonzero_static() execution time: {elapsed_time_ms:.2f} ms")
    print("Indices of 1s:", ones_indices.cpu())  
    
    torch.nonzero_static() execution time: 78.53 ms
    Indices of 1s: tensor([[50000000]])

    A very nice improvement!

  • Diff Risk Score Model

    Just heard this was accepted to a conference, but had completely missed the preprint being released. I briefly supported a some of the folks involved in this work at Meta, and am very glad to see it being shared publicly:

    Moving Faster and Reducing Risk: Using LLMs in Release Deployment

    The project predates Llama, and the paper goes through the history of using a couple of different BERT-style models and then (code)llama, modified for classification/scoring, for evaluating the riskiness of a change.

    The goal of this was less to evaluate every change, but to allow low-risk changes to go through even during code freezes in business-critical periods.

    Its also a good example of practical fine-tuning for a somewhat unusual use case!

    During inference, we can query the model to generate the label token. To get the risk score rather than just the label, we extract the probability of the label tokens “0” and “1” from the next token distribution. If the model is aligned well, all other tokens in the model’s vocabulary should have close to zero probability of being generated.

    Congratulations to Rui, Vijay, Nachi, Chandra and everyone on the team!

  • Chinese Tech Term Glossary

    Very interesting list, via Jeff Ding at ChinAI.

    from Chinese primary sources on technology and security, with expert translations and annotations by CSET’s translation team. It’s created and maintained by Ben Murphy with support from the Emerging Technology Observatory.

    https://docs.google.com/spreadsheets/d/15MS8Qp9U-KOaoQF0R_e7lVxF3UBMyZFpC6-cohhnsD0/htmlview#gid=0

  • ChatGPT Canvas can make API requests now, but it’s complicated

    https://simonwillison.net/2024/Dec/10/chatgpt-canvas/

    This stuff is hard to use. The more features and capabilities we pile onto these systems the harder it becomes to obtain true mastery of them and really understand what they can do and how best to put them into practice.

  • Things that can go wrong with priority levels

    I read this post years back as we did a big migration to Bazel for Android and iOS at Lyft, but was reminded of it from a recent HN comment by jmmv, the author:

    And then, almost by chance, I stumbled upon this QoS system, did a few spot-checks, and found that the VPN daemon was misconfigured to run at a low class. Consequently, Bazel starved the VPN for those users and made everything misbehave. Adjusting that one parameter fixed everything.

    https://jmmv.dev/2019/03/macos-threads-qos-and-bazel.html

  • Way to use torch.compile

    Ways to use torch.compile

    The dream which we sold with torch.compile is that you could slap it on the top of your model and get a speed up. This turns out to… not quite be true? But the fact remains that if you’re willing to put in some work, there is almost always performance waiting at the end of the road for you. Some tips

  • Engineering Culture at Meta

    A question I’ve been asked recently is how Meta compares to other places I’ve worked, or what makes it different.

    From my conversations and observations, those who disliked working at Meta often cited chaos, short-term focus, and internal politics, while those who liked it called out  autonomy, speed, and the feeling they could work on important projects. To explain that disparity, I refer to three values or themes that shape the work culture.  

    Individuals are responsible for doing impactful work: “impact” is an important concept at Meta, and having a level-appropriate collection of impactful work at performance review time is important for every engineer. If you find yourself in a situation where your impact feels limited, you are generally responsible for exploring ways to address it, or make a change. 

    This means that ICs (individual contributors) at Meta are willing to cross team boundaries to find important work, and will also gravitate towards highly visible projects. They care about how their work is regarded and how it fits in to the wider organization. Internal mobility is fairly easy, so folks will leave teams if they can’t find the right kind of work. Its also reflected in the growth expectations: when I worked at Google and Lyft they also had expectations that IC3s would become IC4s, and IC4s become IC5s (though Google later removed this part), but the timelines were somewhat soft. At Meta, they are firm, and expectations ramp at defined intervals as you approach the boundaries. 

    Practically, this means ICs should expect to identify and collaborate on projects that align with organizational goals and take initiative to push them forward. Managers and leadership provide support, but success heavily depends on individuals ability and desire to chart a path, adapt as needed, and ensure their contributions are visible. In general, there’s a strong bias towards getting things done, getting things out,  “rough consensus and running code”.

    Dave Anderson has written about how much more helpful he found teams at Meta, vs Amazon where there was a lot of horse trading for collaboration. Part of that is driven, I think, by this responsibility for impact. Having another team’s thanks, or enabling results for them, allows you to claim some credit for their impact with relatively little effort. Conversely, intentionally blocking another team can be seen as gatekeeping, which is frowned upon.

    No gatekeeping: Some version of “Move Fast” has been in Meta’s official values for a long time, and the company still operates at a good clip. Part of that is aided by generally making it easy to go make changes wherever they need to be made. One example of this I use with Google folks is OWNERS files. Google and Meta are both monorepo based, but at Google you have sets of services with clear owners, and touching code in another team’s service requires their full blessing. Meta also operates a monorepo, but there is much more fluidity — folks can land changes anywhere they need to. In part this is because the original Meta product, Facebook itself, is a monolith, but there is a deeper cultural aspect here.

    For example, even very senior engineers will very rarely say “no” to something. Instead of outright rejection, feedback is often framed as suggestions or concerns to consider. This encourages risk-taking and innovation, but it also places a significant responsibility on engineers to weigh feedback carefully, address risks , and seek out champions from stakeholders. This is one source of the disconnect between folks who see Meta as a place where it’s ok to take risks and folks who don’t: if you take a risk, it fails, and at PSC (performance review) time someone affirms they called out the problems that occurred and you didn’t take appropriate measures, you will be dinged. I have seen people interpret this kind of feedback as nits or suggestions, rather than weighing it heavily and convincing others that the risk is well managed ahead of time.

    In general, folks are expected to be helpful, to provide guidance to others, and not to put up walls, so it can be a tricky balance when outside teams or other engineers come in and ride roughshod over a team’s plans or projects. As an engineer, escalating misalignments on goals/priorities to management is usually well supported, and as a manager, putting engineers together to get to technical solutions across teams is expected. Enacting hard blocks where one team can’t achieve their goals because another was in the way is less so.

    The heavy dependence on individuals and relationships, particularly for cross-team projects, is another key theme:

    It’s a social company: Somewhat unsurprisingly for a company that started around a social network, Meta is a pretty social company. The internal social network is a firehose of information, and there are deep networks of connections across the company between senior engineers, managers and executives.

    Its important for engineers to talk about their work in order to find folks interested in it, build connections and relationships with them, and have a good sense not just of their org but the universe of organizations that they operate within. A fairly common failure pattern is to build a good relationship with one side of an org and ignore another, developing a sizable blind spot that later comes back to be a problem.

    The official org chart is the secondary and lagging structure at the company. The more important structure is the informal network of relationships that where many things get done, and decisions get made.

    This dynamic can sometimes feel political, which is why some describe Meta this way. While there are large-company politics (it is a large, influential company), for most people, it’s less about traditional power politics and more about navigating cliques and informal networks of folks who have worked together on multiple projects and have mutual trust and respect. The company leans a lot on strong, senior engineers to drive projects to success, and those folks may not report into the org that is officially doing the work, or may be at a lower or higher level in the org chart than you might expect. They will often work by going directly to the people they know to unblock issues, drive important changes, or get alignment on a controversial decisions.

    For big enough changes, org structural changes do follow, but they usually lag rather than lead the work itself.

    What are the downsides and upsides?

    As folks who have had a bad time can attest, Meta can be chaotic. There can be parallel implementations, people can swarm on important projects to the detriment of those trying to work on them, and less impactful projects can end up unowned and passed around. It can feel like information overload, with more being published than you can possibly follow. At the same time, there’s can be an information drought when truly important conversations happen in small, exclusive groups. For example, I’ve seen feedback about a project be shared openly between a small collection of engineers and leaders, without ever clearly reaching the team responsible for developing it.

    The flip side is that this combination of values allows Meta to pivot surprisingly fast. Changes that would have required months at some companies can be kicked off in a day, particularly by very senior, well-connected leaders. Senior ICs can take problem descriptions, and quickly form an idea of who might have thoughts on it, and pull them in. Soon you have a loose group (often later structured into a “v-team” or virtual team) that can quickly align and drive change. The lack of gatekeeping, both technically and culturally, reduces the corporate immune reaction to large changes. The incentive to individual impact encourages folks to jump onboard important things without having to work out if that means a team change, or what their long-term situation might be.

  • A very lazy mental model for parallelisms

    There is a great deal of technical complexity in distributed data parallel, model parallel, tensor parallel, fully-sharded-data-parallel, ZeRO 1/2/3, context parallel etc. To fit ideas into a picture I tend to segment in-practice parallelism into three buckets.

    1. Data parallel
    2. Pipeline parallelism
    3. Everything else

    I will tackle these out of order:

    Data parallelism

    Data parallel is a very convenient training concept because you take multiple copies of the model and give different data to them, do a forward-backward pass, then all-reduce gradients between them i.e., aggregate gradients across all devices so they operate as if they had trained on the unified set of data. This lets you train more quickly by using more devices!

    This all-reduce operation can usually be effectively overlapped with other computations, minimizing the overhead of data parallelism, and the copies of the model can be largely anything: a model running on a single GPU, or a cluster of devices running many parallelisms of their own.

    Because you get to run a whole batch of the computation through the model before needing comms, its relatively network efficient as well, which makes it plausible to do distributed data parallel over more normal network links (than the ones I’m about to mention). 

    Everything else

    Almost everything about these parallelism methods are annoying: either the communication collectives are difficult to overlap with other operations, or they must occur more frequently, leading to increased sensitivity to latency.

    This means you want faster links, which means you need very tight networking like, canonically, NVLink, which is Nvidia’s high-speed interconnect between GPUs. This tends to mean you have tensor parallelism, context parallelism etc. implemented within these fast domains, which traditionally is one host (8 GPUs), but with Blackwell can be quite a lot more (up to 36 or 72). There are a lot of options for efficiently packing the compute and scheduling work between devices in there based on the characteristics of the model, and they form this big set of potential parallelisms.

    Pipeline parallelism 

    When your model unit is too big for the available “fast” networking domain, you try and divide the model into multiple sequential stages (like layers) and process them in parallel, overlapping the computation of one stage with the communication of the next.

    This is painful for all the reasons pipelining in anything is painful but is additionally painful in that you have to write your training code with, effectively, a bunch of if statements for whichever stage it happens to be in. 

    There is some leakage between these buckets (e.g. the first and last stages of the pipeline tend to vary a lot due to dealing with embeddings for input and output), but it’s a reasonable first approximation to treat them as discrete.

    (Or FSDP everything) 

    This does presume quite a lot of scale, as the cost of getting everything working well together is non-trivial. FSDP (preferably FSDP2 in PyTorch at least) directly mixes up between the DP and Everything Else buckets, and works pretty well for most folks up to a decently large (100s+) number of GPUs. So roughly: 

    • Smallish model, lots of data: Distributed data parallel.
    • Medium sized model, medium sized number of GPUs: FSDP2
    • Anything expensive: units of <parallelisms>, chunked into pipelines if the model is too big, copied into multiple DP copies to speed up training.  
  • TIL: When does PyTorch upgrade Python versions

    The policy is in the RELEASE documentation:

    PyTorch supports all minor versions of CPython that are not EOL: https://devguide.python.org/versions/

    This is a little more consistent than how it was handled in the past, with annual upgrades and deprecations to match the cpython release schedule.

  • A Meticulous Guide to Advances in Deep Learning Efficiency over the Years

    https://alexzhang13.github.io/blog/2024/efficient-dl/

    A long and wide selection of topics around deep learning, with links and citations for everything.

    This post offers a comprehensive and chronological guide to advances in deep learning from the perspective of efficiency: things like clusters, individual hardware, deep learning libraries, compilers — even architectural changes. This post is not a survey paper, and is intended to provide the reader with broader intuition about this field

    I’d say it achieves that goal!