Tag: ml

  • HuatuoGPT-o1, domain specific reasoning

    https://arxiv.org/abs/2412.18925

    A good paper on creating domain specific (medical in this case) reasoning LLM. I had been somewhat vague on how creating a reasoning model actually worked, and this paper felt very clear (as to one recipe at least). As I read it:

    Start with a base model you are fine tuning for reasoning (in their case qwen 2.5) and a strong general model for verification and data generation (in this case GPT-4o)

    1. Gather a bunch of problems with ground-truth answers. In this case they had medical problems. Take about half of them for fine tuning data.
    2. Prompt the general model to think step-by-step for each problem to get some initial reasoning.
    3. Use the general model to evaluate whether each answer reached is correct vs ground truth.
    4. If the reasoning gets to the wrong answer (which is likely!) try a search strategy randomly – e.g. backtracking (start from an earlier step), critique-and-correct the existing line of reasoning, explore a new path distinct from the one given, or verify the current reasoning. Again, this is done by prompting the general model.
    5. Each problem gets three tries to get to a correct answer before giving up. At the end of this process we have a series of reasoning traces that get to the correct answer. Pair these with the problems and used for the next steps, but first rewrite them into a chain of thought incorporating “hmms” and other smooth transitions between thoughts, via prompting the general model.
    6. Fine tune the base model on the problem traces.
    7. Do RL (PPO) on the fine tuned model with the other half of the problems, rewarding when the model is correct with reasoning (verified by the general model), giving a small reward for a incorrect answer but with reasoning, and giving 0 for no reasoning provided (regardless of answer). Also constrain with KL divergence from the fine tuned model, as is standard in RLHF etc.

    This seems like a pretty reproducible recipe and the results seem strong. They include the prompts they use in the appendix, helpfully, and some good ablations/practical notes.

    1. Effectiveness of Complex CoTs: We further examined the impact of different types of Chain-ofThought (CoT) reasoning. The results show that direct learning of response (yˆ) performs the worst,
      while simple CoT (y0, e0) offers only little benefit. In contrast, Complex CoT (yˆ0, eˆ) significantly
      improves performance by an average of 4.3 points. This demonstrates the importance of teaching
      models to refine their answers with reflection

    One other interesting note on sourcing data was that they used GPT-4o to filter a number of multiple choice questions to generate the set of problems and ground truth. They used it to evaluate whether questions were complex enough to require reasoning, and whether they had a single clear and unambiguous answer. I am guessing it is a lot easier to get multiple choice question banks than other kinds, so this is a clever approach.

  • Thunderkittens /GPUs go brr

    Just got round to reading the intro post to the (now improved) thunderkittens kernel DSL.

    https://hazyresearch.stanford.edu/blog/2024-05-12-tk

    Many good nuggets on kernel writing in general and the hopper in particular.

    But to us a “register” is a 16×16 tile of data. We think AI wants this — after all this time, it’s still just matrix multiplies, reductions, and reshapes. And we think the hardware wants this, too — small matrix multiplies are just begging for hardware support beyond just the systolic mma.

    In fact, more broadly we believe we should really reorient our ideas of AI around what maps well onto the hardware. How big should a recurrent state be? As big can fit onto an SM. How dense should the compute be? No less so than what the hardware demands. 

  • Diff Risk Score Model

    Just heard this was accepted to a conference, but had completely missed the preprint being released. I briefly supported a some of the folks involved in this work at Meta, and am very glad to see it being shared publicly:

    Moving Faster and Reducing Risk: Using LLMs in Release Deployment

    The project predates Llama, and the paper goes through the history of using a couple of different BERT-style models and then (code)llama, modified for classification/scoring, for evaluating the riskiness of a change.

    The goal of this was less to evaluate every change, but to allow low-risk changes to go through even during code freezes in business-critical periods.

    Its also a good example of practical fine-tuning for a somewhat unusual use case!

    During inference, we can query the model to generate the label token. To get the risk score rather than just the label, we extract the probability of the label tokens “0” and “1” from the next token distribution. If the model is aligned well, all other tokens in the model’s vocabulary should have close to zero probability of being generated.

    Congratulations to Rui, Vijay, Nachi, Chandra and everyone on the team!

  • A very lazy mental model for parallelisms

    There is a great deal of technical complexity in distributed data parallel, model parallel, tensor parallel, fully-sharded-data-parallel, ZeRO 1/2/3, context parallel etc. To fit ideas into a picture I tend to segment in-practice parallelism into three buckets.

    1. Data parallel
    2. Pipeline parallelism
    3. Everything else

    I will tackle these out of order:

    Data parallelism

    Data parallel is a very convenient training concept because you take multiple copies of the model and give different data to them, do a forward-backward pass, then all-reduce gradients between them i.e., aggregate gradients across all devices so they operate as if they had trained on the unified set of data. This lets you train more quickly by using more devices!

    This all-reduce operation can usually be effectively overlapped with other computations, minimizing the overhead of data parallelism, and the copies of the model can be largely anything: a model running on a single GPU, or a cluster of devices running many parallelisms of their own.

    Because you get to run a whole batch of the computation through the model before needing comms, its relatively network efficient as well, which makes it plausible to do distributed data parallel over more normal network links (than the ones I’m about to mention). 

    Everything else

    Almost everything about these parallelism methods are annoying: either the communication collectives are difficult to overlap with other operations, or they must occur more frequently, leading to increased sensitivity to latency.

    This means you want faster links, which means you need very tight networking like, canonically, NVLink, which is Nvidia’s high-speed interconnect between GPUs. This tends to mean you have tensor parallelism, context parallelism etc. implemented within these fast domains, which traditionally is one host (8 GPUs), but with Blackwell can be quite a lot more (up to 36 or 72). There are a lot of options for efficiently packing the compute and scheduling work between devices in there based on the characteristics of the model, and they form this big set of potential parallelisms.

    Pipeline parallelism 

    When your model unit is too big for the available “fast” networking domain, you try and divide the model into multiple sequential stages (like layers) and process them in parallel, overlapping the computation of one stage with the communication of the next.

    This is painful for all the reasons pipelining in anything is painful but is additionally painful in that you have to write your training code with, effectively, a bunch of if statements for whichever stage it happens to be in. 

    There is some leakage between these buckets (e.g. the first and last stages of the pipeline tend to vary a lot due to dealing with embeddings for input and output), but it’s a reasonable first approximation to treat them as discrete.

    (Or FSDP everything) 

    This does presume quite a lot of scale, as the cost of getting everything working well together is non-trivial. FSDP (preferably FSDP2 in PyTorch at least) directly mixes up between the DP and Everything Else buckets, and works pretty well for most folks up to a decently large (100s+) number of GPUs. So roughly: 

    • Smallish model, lots of data: Distributed data parallel.
    • Medium sized model, medium sized number of GPUs: FSDP2
    • Anything expensive: units of <parallelisms>, chunked into pipelines if the model is too big, copied into multiple DP copies to speed up training.  
  • A Meticulous Guide to Advances in Deep Learning Efficiency over the Years

    https://alexzhang13.github.io/blog/2024/efficient-dl/

    A long and wide selection of topics around deep learning, with links and citations for everything.

    This post offers a comprehensive and chronological guide to advances in deep learning from the perspective of efficiency: things like clusters, individual hardware, deep learning libraries, compilers — even architectural changes. This post is not a survey paper, and is intended to provide the reader with broader intuition about this field

    I’d say it achieves that goal!

  • Training Language Models to Self-Correct via Reinforcement Learning

    Training Language Models to Self-Correct via Reinforcement Learning – Aviral Kumar, Vincent Zhuang et al at Deepmind

    Observes that taking traces (any method) of LLM output, reflection and corrected output and using that for SFT tends to result in the model learning a minimal edit strategy that does either no self correction or minimal edits to the original answer.

    We find that such approaches fall prey to either: (1) distribution shift, where the trained model is able to correct errors made by the base model that generated the data, but these gains do not transfer to self-correction under the learned model’s own mistakes; or (2) behavior collapse, where the learning progress simply learns to produce the best first-attempt response followed by superficial or no modifications in the second attempt.

    To avoid that, they use a 2 stage RL process where they constrain the first to the same distribution as the base model, then maximize the reward on the second.

  • On what neural net architectures actually make a difference

    https://nonint.com/2024/03/03/learned-structures/

    As time has passed, I’ve internally converged on the understanding that there are only a few types of architectural tweaks that actually meaningfully impact performance across model scales. These tweaks seem to fall into one of two categories: modifications that improve numerical stability during training, and modifications that enhance the expressiveness of a model in learnable ways.

    I think this is a super interesting post, as the mental model it gives of architectures is helpful. The idea that the model learns “in stages” is very intuitive, and is easy to see in some architectures (like convnets!)