Observes that taking traces (any method) of LLM output, reflection and corrected output and using that for SFT tends to result in the model learning a minimal edit strategy that does either no self correction or minimal edits to the original answer.
We find that such approaches fall prey to either: (1) distribution shift, where the trained model is able to correct errors made by the base model that generated the data, but these gains do not transfer to self-correction under the learned model’s own mistakes; or (2) behavior collapse, where the learning progress simply learns to produce the best first-attempt response followed by superficial or no modifications in the second attempt.
To avoid that, they use a 2 stage RL process where they constrain the first to the same distribution as the base model, then maximize the reward on the second.
I’ve been following the development of the higher order ops in PyTorch nightlies for a little bit, and got a chance to try out while_loop. The best examples right now are in the tests, but as another, here’s a mandlebrot example:
import torch
from torch._higher_order_ops.while_loop import while_loop
import matplotlib.pyplot as plt
def mandelbrot_step(z, c):
"""Performs one iteration of the Mandelbrot sequence."""
return z**2 + c
def mandelbrot(c, max_iter, threshold):
"""Compute Mandelbrot set membership for a grid of complex numbers."""
def cond_fn(z, iter_count, mask):
return torch.any(mask & (iter_count < max_iter))
def body_fn(z, iter_count, mask):
z_next = mandelbrot_step(z, c)
diverged = torch.abs(z_next) > threshold
mask_next = mask & ~diverged
iter_count_next = iter_count + mask_next
return z_next, iter_count_next, mask_next
# Initialize variables
z0 = torch.zeros_like(c)
iter_count = torch.zeros(c.shape, dtype=torch.int32)
mask = torch.ones(c.shape, dtype=torch.bool) # All points start as candidates
final_state = while_loop(cond_fn, body_fn, (z0, iter_count, mask))
_, iterations, _ = final_state
return iterations
# Define the grid of complex numbers
x = torch.linspace(-2.0, 1.0, 500)
y = torch.linspace(-1.5, 1.5, 500)
xx, yy = torch.meshgrid(x, y)
complex_grid = xx + 1j * yy
# Compute the Mandelbrot set
max_iter = 100
threshold = 2.0
mandelbrot_set = mandelbrot(complex_grid, max_iter, threshold)
# Plot the Mandelbrot set
plt.figure(figsize=(10, 10))
plt.imshow(mandelbrot_set, extent=(-2, 1, -1.5, 1.5), cmap="inferno")
plt.colorbar(label="Iteration count")
plt.title("Mandelbrot Set")
plt.xlabel("Real")
plt.ylabel("Imaginary")
plt.show()
In general, the only non-obvious thing about while_loop is that the cond_fn is returning a tensor, not a bool, so make sure you are getting your types right, and that the shapes must be consistent from loop to loop. If you need more accumulating type behavior, look at scan!
Off-topic framing but a very relatable feeling and process – how various chemists hit their “eureka” moments and discoveries.
Some people deliberately make time for activities that seem to lead to more chances for insight (long walks, showers, etc.) Others have been content to let these things hit them more randomly, although you do wonder of some of then have wondered about trying to recreate these experiences deliberately. One common experience is the feeling that some part of one’s brain has apparently been working on a problem outside conscious awareness and that it’s somehow reporting back with its results. But others describe what sounds very close to random chance – they just happened to read or see something that triggered the key insight.
The error message you get actually tells you the fix, but I found it non-intuitive to what I was doing enough I was hesitant to actually just try the config:
The general issue is capturing shapes on scalars isn’t turned on by default due to various issues, but for your case it may actually work. It is also interesting to see where TorchVision hit this, and worked around with torch.where instead.
I think I did know this at some point, but I was reminded today that unlike the (sometime) black box that is CUDA https://github.com/ROCm/ROCm is actually available on Github, which is operationally much nicer!
I also recall learning that NCCL, which is open, https://github.com/NVIDIA/nccl is in part because some of it was funded by the Lawrence Berkeley National Laboratory!
Whenever I am trying to do something fun (read: poorly considered) with torch.compile I find myself googling for the same handful of references, most of which are in the PyTorch Drive folder, so putting them here for my own reference.
If you are doing regular compile usage you shouldn’t need these, but they are helpful when debugging more complex shape issues or investigating performance issues or graph breaks.
Every time I have to set up a clean system I manage to mess up cuda somehow, so leaving this as an aide memoir. In general, there is a default nvidia-cuda-toolkit package that ships with Ubuntu-based systems, and you should ignore that. The right options are either:
conda install cudatoolkit
This gives you the same thing in a conda env, but is somewhat limited in terms of the range of versions available
nvidia-container-toolkitt
Lets you run different cuda versions in docker containers, which can work for having completely isolated environments.
The conda option generally works for me, but do try and install this first as you will end up in a painful package resolution process if you forget and have installed PyTorch first.
A very good passing comment from Nikita at work. When folks train in eager mode they are generally able to leverage the full set of aten ops, with their attendant implementations in different backends.
Taking the trained model and exporting using TorchScript was largely guaranteed the same results, assuming it TorchScripted cleanly, as it’s just using libtorch underneath. The downside of this is shipping with a lot of stuff you probably don’t need, which is not great for LiteInterpreter on mobile in particular, plus of course the downside of the all the TorchScript weirdness.
Torch.export offers a much cleaner flow, but on the flip side does a lot more processing of the model. The aten ops are decomposed into a simpler set of ~200 ops for the export IR, which might then be further processed for specific hardware. While you have a lot of control of this process, it means there are a lot of places to potentially introduce subtle differences between training time and inference, further training or whatever you are doing next.
The important thing here is not to treat the two technologies as straight substitutes but as different paths to the same goal, that require different processes around them.
The repo from Tristan Rice and Chirag Pandya’s poster at the PyTorch conference has been continually being updated with refinements and improvements.
torchft implements a lighthouse server that coordinates across the different replica groups and then a per replica group manager and fault tolerance library that can be used in a standard PyTorch training loop. This allows for membership changes at the training step granularity which can greatly improve efficiency by avoiding stop the world training on errors.
There are lots of clever techniques in here. They center around the idea of having replica groups which serve as the failure domain, rather than the whole training job. In a loose sense, this means that when there is a failure you simply drop the replica group its in and carry on with the rest to the next batch, adding the replica group back in when its recovered.
To make that possible, there’s custom comms that allow for error handling, health monitoring of individual processes, and fast checkpointing that allows recovered workers to be quickly added back to replica sets.
The alignment stage of LLM post-training requires lots of human-generated data, and the quality is bounded on how good that data is. Lots of techniques here, with citations.
Questions that can support better understanding, to inform better strategy and design.
Daniel Schmachtenberger speaks compellingly on the value systems implied and expressed by any piece of technology. He has a list of questions to help think more broadly around something when introducing it.
There have been a few, good, attempts at Transformer based recommendation systems, but this one from Jiaqi Zhai and the folks inside the modern recommendation systems team at Meta is particularly strong.
In this work, we treat user actions as a new modality in generative modeling. Our key insights are, a) core ranking and retrieval tasks in industrial-scale recommenders can be cast as generative modeling problems given an appropriate new feature space; b) this paradigm enables us to systematically leverage redundancies in features, training, and inference to improve efficiency. Due to our new formulation, we deployed models that are three orders of magnitude more computationally complex than prior state-of-the-art, while improving topline metrics by 12.4%
The model uses sequences of user history rather than traditional embedding towers, and is a lot more compute intensive than most recommendation systems. To make all that work, the accompanying Github repo has some custom Triton kernels: https://github.com/facebookresearch/generative-recommenders
As time has passed, I’ve internally converged on the understanding that there are only a few types of architectural tweaks that actually meaningfully impact performance across model scales. These tweaks seem to fall into one of two categories: modifications that improve numerical stability during training, and modifications that enhance the expressiveness of a model in learnable ways.
I think this is a super interesting post, as the mental model it gives of architectures is helpful. The idea that the model learns “in stages” is very intuitive, and is easy to see in some architectures (like convnets!)