Tokenization has always struck me as one of the odder aspects of natural language deep learning. Despite the extensive end-to-end learning processes we typically use, tokenization initially involves creating a dictionary of optimal sub-word segments from your dataset. One of the appealing concepts in the Byte Latent Transformers paper is the potential to learn tokenization dynamically, recognizing that tokenizers solve deeper problems than merely providing a fixed vocabulary.
This paper addresses tokenization from a theoretical perspective by modeling sequences using kth-order Markov processes, where the likelihood of each token depends on the preceding sequence, as in natural language. The parameter k corresponds to the model’s context window size. Key findings include:
Training without tokenization leads models to effectively behave as unigram predictors, significantly limiting performance.
Using a well-designed tokenizer (e.g., Byte Pair Encoding – BPE) enables models to achieve nearly optimal performance in capturing sequence dependencies.
Increasing the tokenizer’s dictionary size improves the model’s performance, moving it closer to the ideal probability distribution.
Tokenizers which do a good job at learning patterns in the data and assigning these frequent patterns as tokens in the dictionary are compatible with an i.i.d. model over tokens.
This insight suggests that despite the complexity of natural language’, a good tokenizer converts sequences into something approximating an independent and identically distributed (i.i.d.) format, which brings the modeling tasks for transformers closer to the one they can solve.
While the paper does not explicitly explore the Byte Latent approach, I wonder if its entropy-driven dynamic token allocation might similarly achieve this i.i.d. simplification. In BLT the entropy model, trained separately, could be dynamically transform inputs into a distribution that is more palatable for transformers.
Fused Linear Cross-Entropy is a popular optimization that combines the final linear projection and cross-entropy loss into a single operation. This fusion is very valuable for training large language models efficiently, as it can reduce memory usage significant, particularly for larger vocabularies.
If you look at a LLM training loop, you generally see something like:
logits = model(input_ids)
loss = cross_entropy(logits, targets)
And if you look at the end of the model, you’ll see something like the below, where h is the hidden state so far and output is output = nn.Linear(embed_dim, vocab_size, bias=False)
That final logics value can be pretty big: sequence length is long and the vocabulary size is large (128k for Llama 3, 202k for llama 4), so logits can be GB of memory: with a 16k context window, a 128k vocab, and 4k embedding dimensions even at a batch size of 1, you get 8bn entries. At BF16, that’s 4GB. You’ll also need to capture the gradient, which will give you another 4GB in the backwards.
That set of logits has a range of values that are a bit all over the place, one for each of the possible targets.
Roughly speaking, cross entropy measures the similarity of two probability distributions. In the context of neural networks, it’s common to use cross entropy as a loss function for classification problems where:
q is our predicted probabilities vector (i.e. the softmax of our raw network outputs, also called logits, denoted as y^), that is q=softmax(y^)
p is a one-hot encoded vector of our label, that is a probability vector that assigns 100% probability to the position y (our label for the correct class): pi={1i=y 0i≠y
This means that cross-entropy simplifies to F.nll_loss(F.log_softmax(x, 1), target)
Softmax makes our previously messy logits into a nice probability distribution where all the values are positive and sum to one. log softmax is usually used in LLMs, for numerical stability and efficiency.
When we implement softmax, the naive implementations looks something like:
out = torch.log(torch.exp(x) / torch.sum(torch.exp(x)))
This isn’t numerically stable, so you want to subtract the max to avoid overflows and underflows in the exp. This is the common log-sum-exp implementation:
x_max = torch.max(x)
shifted_x = x - x_max
exp_shifted = torch.exp(shifted_x)
out = shifted_x - torch.log(torch.sum(exp_shifted)
In general the memory and compute cost of this grows with the size, which gets painful for our hefty logits. We can instead keep a running log-sum-exp so we don’t have to deal with the whole input at once.
lse = xs[0]
for x in xs[1:]:
m = torch.max(torch.stack([lse, x]))
lse = m + torch.log(torch.exp(lse - m) + torch.exp(x - m))
out = lse
This is the online log-sum-exp approach, and makes our life easier! We can now compute incrementally, but we are still generating the big logits before hand.
Fused Linear Cross-Entropy replaces the output projection, softmax and loss calculation with a single kernel that a tiles across all of it.
This is the core of the idea: instead of computing all logits at once (which creates a massive tensor), we can:
Compute logits for small chunks of the vocabulary
Compute the softmax incrementally
Only store the logits we need for the loss calculation
This repo contains an implementation of a linear projection + cross-entropy loss PyTorch module that has substantially lower memory consumption compared to a standard implementation, with almost no additional compute cost. The memory savings come from two optimizations: 1) overwriting the logits with their gradients in-place and 2) not materializing the entire logits tensor.
Roughly, the loop looks like:
For each of the token i in the sequence, with output layer weights h
Compute a partial dot product si = hi dot W_tile
Reduce for a running max and exp-sum
Return only the si[targeti] needed for the loss.
This gives you quite a lot of memory wins, which also reduce peak memory bandwidth needs. But this also introduces some potential pain!
You’re fusing the final layer op into the loss, which might be defined in different places in your model code
You’re accumulating, so you have to use fp32 or risk introducing numeric errors
You have to write you own backwards op as well, which will generally do some extra computation, so you are paying some extra FLOPS
Debugging can be harder, so you want a good recipe prior to swapping in the op
May require some futzing for best implementations on different hardware.
Actually implementing is pretty straightforward.
@staticmethod
def forward(ctx, h, W, target):
B, D = h.shape
V, _ = W.shape
chunk_size = min(1024, V)
# Initialize online softmax accumulators
max_logits = torch.full((B,), -float('inf'), device=h.device, dtype=torch.float32)
sum_exp = torch.zeros(B, device=h.device, dtype=torch.float32)
target_logits = torch.zeros(B, device=h.device, dtype=torch.float32)
# Process vocabulary in chunks
for chunk_start in range(0, V, chunk_size):
chunk_end = min(chunk_start + chunk_size, V)
# Compute logits for this chunk only
W_chunk = W[chunk_start:chunk_end, :]
logits_chunk = h @ W_chunk.T # [B, chunk_size]
# Update running max
chunk_max = logits_chunk.max(dim=1).values
new_max = torch.maximum(max_logits, chunk_max)
# Adjust previous sum_exp by exp(old_max - new_max)
sum_exp *= torch.exp(max_logits - new_max)
# Add this chunk's contribution to sum_exp
sum_exp += torch.exp(logits_chunk - new_max.unsqueeze(1)).sum(dim=1)
# Update max
max_logits = new_max
# Extract target logits if target is in this chunk
chunk_indices = torch.arange(chunk_start, chunk_end, device=h.device)
is_target_in_chunk = (target.unsqueeze(1) == chunk_indices.unsqueeze(0))
target_logits += (logits_chunk * is_target_in_chunk).sum(dim=1)
# Compute loss: -log(p_target) = -(target_logit - log_sum_exp)
log_sum_exp = max_logits + torch.log(sum_exp)
loss = log_sum_exp - target_logits
# Save for backward
ctx.save_for_backward(h, W, target, max_logits, sum_exp)
ctx.chunk_size = chunk_size
return loss.mean()
Here we chunk the vocabulary, calculate the partial transform for the chunk h @ W_chunk.T, do online softmax and accumulate the target logits.
The backward calculates the gradients:
@staticmethod
def backward(ctx, grad_output):
h, W, target, max_logits, sum_exp = ctx.saved_tensors
chunk_size = ctx.chunk_size
B, D = h.shape
V, _ = W.shape
# Scale gradient by batch size (since we use mean reduction)
grad_scale = grad_output / B
# Initialize gradient accumulators
grad_h = torch.zeros_like(h)
grad_W = torch.zeros_like(W)
# Process vocabulary in chunks (same as forward)
for chunk_start in range(0, V, chunk_size):
chunk_end = min(chunk_start + chunk_size, V)
chunk_indices = torch.arange(chunk_start, chunk_end, device=h.device)
# Recompute logits for this chunk
W_chunk = W[chunk_start:chunk_end, :]
logits_chunk = h @ W_chunk.T # [B, chunk_size]
# Compute softmax probabilities for this chunk
# p_i = exp(logit_i - max) / sum_exp
probs_chunk = torch.exp(logits_chunk - max_logits.unsqueeze(1)) / sum_exp.unsqueeze(1)
# Gradient w.r.t. logits: grad_logits = p - 1_{y=i}
grad_logits_chunk = probs_chunk * grad_scale
# Subtract 1 from target positions
is_target = (target.unsqueeze(1) == chunk_indices.unsqueeze(0))
grad_logits_chunk -= is_target.float() * grad_scale
# Accumulate gradients
grad_h += grad_logits_chunk @ W_chunk
grad_W[chunk_start:chunk_end, :] = grad_logits_chunk.T @ h
return grad_h, grad_W, None
In the backwards we recompute the logits for the chunks, and calculate the logits.
This is a very simplified implementation that trades off a bunch of kernel launches, so gives up a lot of performance, but you can see the memory savings:
For a more sophisticated implementation, you can look at the repo mentioned before or Liger has a good quality kernel with further optimizations. These calculate the gradients in the forward pass, then can just scale them in the backwards. This trades off a bit more memory for less of a compute hit. In general there are a few options for choosing the right point
LLMs have most of their parameters in the FFN parts of the transformer layers — 50+bn params of the Llama 3 70b model, for example. The compute and memory requirements are a bit different between the FFN and attention parts of the model: attention requires a different KV cache for each request, so attention tends to be memory bound while the dense FFNs tend to be compute bound.
Because of this it’s pretty common to split up tasks at inference time. The initial prefill stage (processing the initial prompt) populates the KV cache for the following autoregressive decoding. The decode can be more aggressively batched for getting better utilization. vLLM really helped popularize this idea!
ByteDance extend this idea for mixture of expert models. In MoEs the compute intensity of the FFNs is limited by needing to load different experts, and having only a proportion of tokens going through a given expert. They extend the disagg idea to go from M “attention” GPUs to N (fewer!) expert GPUs, with a larger batch size for each of the expert calls. This gets better utilization on the matmuls and lowers overall cost of serving. The natural structure of transformer layers alternating attention and FFN lends itself well to a ping-pong pipelining approach that lets them hide the comms overhead.
We present MegaScale-Infer, an efficient and cost-effective system designed for large-scale MoE serving. MegaScale-Infer disaggregates the attention and expert modules, assigning them to separate GPUs—a strategy we term disaggregated expert parallelism. Our approach offers two major benefits. First, it enables independent scaling of each module with customized model parallelism strategies. Specifically, attention modules are replicated using data parallelism, while FFN modules are scaled with expert parallelism. By consolidating requests from multiple attention replicas, the GPU utilization of each expert increases significantly as the batch size per attention replica grows. Second, it enables the deployment of attention and FFN modules on heterogeneous GPUs to fully leverage their different capabilities and achieve lower costs. For example, attention modules can be deployed on GPUs with more cost-effective memory capacity and bandwidth, while FFN modules can utilize GPUs with more affordable compute capability. As shown in Figure 1(c), FFN can easily become compute-intensive in MegaScale-Infer, while attention achieves higher GPU utilization per cost under heterogeneous deploymen
Very useful insight in this paper out of Stanford.
Test-time inference has emerged as a powerful paradigm for enabling language models to “think” longer and more carefully about complex challenges, much like skilled human experts. While reinforcement learning (RL) can drive self-improvement in language models on verifiable tasks, some models exhibit substantial gains while others quickly plateau.
The authors were running a reasoning post-training process on both Qwen 2.5 3B and Llama 3.2 3B. They noticed that while both learned, Llama was consistently worse than Qwen, which feels odd as both models are strong. In looking at the reasoning approaches exhibited they observed 4 distinct reasoning strategies:
verification
backtracking
subgoal setting
backward chaining
They noticed that Qwen exhibited these behaviors more from the base model, and those behaviors were enhanced more in the RL process.
While the larger Llama-3.1-70B showed generally increased activation of these behaviors compared to Llama-3.2-3B, this improvement was notably uneven — backtracking, in particular, remained limited even in the larger model.
They then generated some custom reasoning traces that intentionally demonstrated all 4 behaviors, using Claude.
We generate these datasets using Claude-3.5-Sonnet4, leveraging its ability to produce reasoning trajectories with precisely specified behavioral characteristics. While Claude does not always produce the correct answer (see Fig. 9), it consistently demonstrates the requested reasoning patterns, providing clean behavioral primitives for our analysis.
They found that when using the SFT set before RL they closed most of the gap between Llama and Qwen. They also found that it isn’t even important that the reasoning traces are correct – demonstrating the behavior is more important than the reasoning itself at the SFT stage.
Priming models with cognitive behaviors, by a small amount of finetuning, enabled significant performance gains even in models that initially lack these capabilities. Remarkably, this holds even when primed with incorrect solutions that exhibit the target behavioral patterns, suggesting that cognitive behaviors matter more than solution accuracy.
This fits with the elicitation idea — the SFT is training a style. By having increased activation of the reasoning styles the RL process is more able to explore these capabilities and reinforce extended reasoning generation.
This also fits with my mental model that a base model’s capabilities are often pretty underexplored: a combination of targeted SFT + RL seems to be a very powerful elicitation tool!
Nathan Lambert of the Allen Institute writes about their (very strong) Olmo 2 32B release, and the just released Gemma 3 model from Google. One of the many interesting points:
Comparing Gemma 3 27B to OLMo 32B, the pretraining evaluations for both are super similar, but Gemma 3 scores are way better after post-training. The ceiling on post-training expectations has been shifting extremely fast among open models.
Given that Google have about the best crawling infrastructure in the world, and that Al2 have published the complete pretraining dataset used for Olmo, I think this is slightly surprising. You can see the benchmarks in the blog and technical report: for example, Gemma 3 27B gets 78.8 on winogrande from pretraining (a little below Gemma2 as it happens) while Olmo2 32B get 78.7.
The vibes have definitely shifted to post-training for where model differentiation is coming from, opening the question of what exactly is happening there. Nathan also posted about that recently, linking to this post by Mohit Ragavendra of Scale and Georgia tech:
The post looks at The Superficial Alignment Hypothesis, which is (largely) that post-training is just about preference tuning for behaviors the base model can already do
[…]
It initially seems like “Less Is More” in the sense that the LIMA model response was highly preferred by the GPT-4 evaluator for Math prompts (in-line with the work’s original claim). However, these model responses were also largely incorrect – the accuracy of models fine-tuned specifically for Math was substantially better, with the same data budget. If we went by subjective win-rate comparisons, we would have picked a model that was significantly worse.
In the post (and the two linked papers) Mohit breaks down how post-training actually helps. Starting with SFT, the work shows that mimicking style happens quickly, with relatively few samples.
with just a hundred finetuning examples, the model’s formatting mistakes were virtually solved – the model was perfect at mimicking the expected style. However, the model took a lot more supervised finetuning data to get better at reasoning – the substance of the task.
They find though that, largely, more-is-more when it comes to SFT, but that there is a power-law style scaling curve: big gains initially followed by slower, marginal gains. Adding in RL doesn’t change the fundamental curve, but it does shift it, leading more efficiently to the model gaining the reasoning capabilities they were training towards:
Preference data offers a weaker signal compared to supervised finetuning data. So, running DPO directly on the base model on reasoning tasks, is asking the model to learn a completely different response style from its reference model, with a weaker signal, while penalizing for being different from the reference model. Small amount of SFT on the base model teaches it the reasoning style and PFT can use the reward signal to focus on reasoning within the required response space.
I did wonder when reading this whether the results would look different with an online process (like PPO), rather than an offline. Luckily, Mohit links to another recent paper on this topic:
We prove that under idealized assumptions, online and offline PFT techniques should return policies of equivalent quality
but also
we observe that despite the lack of information-theoretic separation, online PFT out-performs offline PFT across different sampling distributions, labelers, and model sizes. Furthermore, it appears to be “easier” to learn a global RM than it is to learn a local RM, leading to higher validation likelihood.
The result here seems to be that the reward model is simply easier to model, and it helps “translate” the problem of the distribution.
This all feels like a continuum: at some level the superficial alignment hypothesis is directionally correct but its not that “superficial”: the base models have a lot of capabilities that are hard to elicit, and fine tuning/post training can juice them effectively, while adding some learning of its own (as more data is better!)
The best way of performing that elicitation turns out to be solving different problems at different levels: SFT for format, then RL for the deeper capability, and having a reward model effectively simplifies the learning process again.
Very clear paper on how RL and SFT combine to elicit reasoning capabilities, with some practical takeaways:
SFT on long chains of thoughts are much better than short
Reward shaping is important for getting stable scaling
Reasoning approaches (like backtracking) are probably present in the base model, if it’s big enough
Based on these observations, we hypothesize that RL primarily guides the model to recombine skills it already internalized during pre-training towards new behaviors to improve performance on complex problem-solving tasks.
This paper, from back in December last year, presents an interesting approach to handling raw byte sequences in LLMs without relying on tokenization.
Vocab sizes for tokenizers have gone up over the last couple of years with attendant gains in usefulness, but this remains a particularly hand-tuned number in the training process. BLT proposes a method that processes raw UTF-8 byte sequences directly, leveraging a dynamic patching mechanism to group bytes into variable-length patches based on entropy.
Higher-entropy regions receive more attention and shorter patches, while lower-entropy regions can be processed more efficiently.
There are conceptually three levels of processing:
Local Encoder: A small transformer stack encodes raw byte sequences into higher level representations, which are then structured into patches.
Latent Global Transformer: A standard large transformer model operating on patch-level representations
Local Decoder: The encoded patches are decoded back into byte sequences, using a cross-attention mechanism to reconstruct text.
In the paper they show they can achieve parity in pretraining with a traditional tokenized approach in llama for similar parameter count, while being more robust and offering some inference time performance gains. The patching approach allows for allocating compute where needed most.
Retrofitting existing models
One of the ideas I found most interesting is starting with a traditionally pretrained model. The paper discusses using the main transformer layers from Llama and training the byte latent approach successfully.
I gave the approach a go with a simplified local encoder, entropy and patching approach, and took the transformer layers from Qwen 2.5 3B, a strong model that could still be trained locally (no corporate resources were harmed, etc).
The basic approach was replacing the tokenizer, adding a small transformer and patch pooling based on a local entropy measure to generate patches, then cross-attending in some of the Qwen layers. Its training a new encoder while leveraging Qwen for the backbone of the global transformer and adding new cross-attention params to make it also the decoder, with the embedding layers at each end chopped off – so a significant domain shift. For inference I leverage the same patch generation process to try and generate effective tokens.
You can find my Torchtune recipe on GitHub, running through the Alpaca dataset. Thus far I’m still training so while loss is improving, I have no idea whether it will turn into something useful. The fact that there is something trainable is fun though, and I have hopes that this kind of technique will lead to some breakthroughs in tokenizer-free models in the future!
Every paper in this series has been required reading in (very) large language model training. The basic theme is that model training requires gang-semantics, where a large cluster of accelerators need to do coordinated work together in order to make progress, which gets progressively more expensive to enable and harder to do reliably as the number of devices in the cluster increases.
The prior papers explored ways of splitting up the training into an inner loop where the model trained fairly traditionally, and an outer optimization loop that aggregated the differences and updated based on them – the outer optimizer works on the deltas between parameter values at the sync point. The outer optimizer still runs on the same cluster as all the inner loops, but it means that only at the “outer” sync point do you need to do synchronization between all the devices. This loosens the coupling between devices and allows introducing failure domains.
This paper addresses the challenge that when you do synchronize you still have to send data for all the parameters, which requires a lot of bandwidth and can block forward progress. Streaming DiLoCo divides the model layers into different shards and syncs those at different times (in practicality, ever 5 inner optimizer steps), lowering the peak bandwidth required. They take shards in a strided fashion rather than sequentially to mildly improve stability and performance.
To further reduce bandwidth, the communication between devices for the outer loop is done in 4-bit floating point! They still do the accumulations/optimization in 32 bit, but they didn’t see any performance loss when using the lower bit rate for comms. All of these comms are overlapped with the inner loop training, which helps minimize stalls.
GRPO (Group Relative Policy Optimization) is an RL technique originally proposed in the DeepSeekMath paper. Instead of using a full-blown value network like PPO does, GRPO samples a group of completions for a given prompt and then computes a relative (normalized) reward for each output. The rewards are “verifiable” because they come from checking the final answer against ground truth and confirming. E.g. does the response follow the expected format (i.e. a <think>…</think> block for reasoning and an <answer>…</answer> block for the solution) and is the answer accurate against a predetermined fact. Not every problem fits this model, but there are a bunch that do, including math reasoning with the GSM8K dataset of grade-school math word problems. These look like this:
“Julie is reading a 120-page book. Yesterday, she was able to read 12 pages and today, she read twice as many pages as yesterday. If she wants to read half of the remaining pages tomorrow, how many pages should she read?”
How Does the Training Work?
Sampling Completions: For each prompt, the model generates a group of candidate completions. These are produced in inference mode (gradients aren’t collected) using a KV cache for speed (or a dedicated inference engine like VLLM)
Verifiable Reward Calculation: Each completion is scored between 0 and 1—rewarding outputs that follow the prescribed format and yield the correct answer.
Forward Pass for Gradients: Both the “policy” (the model being tuned) and a reference (typically the base, unmodified model) are used for a forward pass with the prompt and completions to compute per-token logits and log-probabilities.
Loss and Backwards: The loss is then calculated as a combination of the (group-averaged) reward and a KL divergence term between the tuned model and the baseline, to constrain learning to similar responses. This loss is backpropagated through the policy model based on the earlier forward pass.
Getting it going in TorchTune
Over last weekend I hacked up a quick and dirty version of the training loop in the TorchTune, and over a couple of bus rides to Menlo Park cleaned it up into something that could work as a more general recipe(PR). Most of the work goes into the recipe and getting the dataset shaped properly to generate completions. This version—tested on a smaller model (the 1B Llama 3.2 variant, with LoRA)—showed some promising improvements in approach but I didn’t get to the point of having something converge enough to be confident in the overall recipe. In the DeepSeek R1 paper they had discussed trying a smaller model, but found 3B was the lowest they were able to get results on with some of their fine-tuning approaches.
Luckily for everyone, at around the same time Ariel Kwiatkowski also put together a version that included distributed device support, making it easier to experiment on bigger models. This PR is more modular, and I’m excited to see it refined and landed so the recipe is widely available!
There’s a growing energy around tools like torchtune, and it’s exciting to see how easy it is to “hack on” these ideas. It’s also great to see the techniques show up in other libraries, like HuggingFace’s TRL, which is being used as part of the OpenR1 replication effort!
This weekend I was reading the Tulu v3 paper (link), which offers a deep dive into building robust post-training setups. This is an very good resource for anyone aiming to build a really robust fine-tuning workflows. It covers critical elements like data set selection, synthetic data generation (with example prompts!), strategies for SFT and preference tuning, and various things they struggled with.
One struggle was an issue with gradient accumulation they ran into where the loss was worse than without it on. The community at large also hit this, and fixed it, thanks to an excellent blog post by Unsloth (link).
The bug
Gradient accumulation is a technique used to simulate larger batch sizes by accumulating gradients over several smaller batches before performing a backward pass. This approach is particularly useful for managing memory constraints during training, so comes up a lot when post-training on a biggish model with more limited hardware.
The problem arises when dealing with sequences of varying lengths within these mini-batches. In standard practice, the loss is calculated and normalized by the number of non-padded (i.e., valid) tokens in each sequence. However, when accumulating gradients across multiple mini-batches, each with different sequence lengths, the naive summation of gradients can lead to an incorrect total loss calculation.
The discrepancy occurs because the cross-entropy loss function normalizes by the number of valid tokens, and this normalization factor can vary between mini-batches. When these normalized losses are accumulated without proper adjustment, the final loss does not match what would have been obtained using a single large batch. This results in a higher observed loss during training when using gradient accumulation compared to full batch training.
Daniel and co at unsloth addressed this issue by developing a methodology that ensures the accumulated gradients are correctly scaled, accounting for the varying sequence lengths across mini-batches. This fix aligns the gradient accumulation process more closely with the theoretical foundations of full batch training, leading to more accurate loss calculations and improved training performance.
Fixes and Workarounds
Recent updates in both Hugging Face Transformers (pull request) and TorchTune (pull request) offer fixes. And, at least in Evan’s case, a little bit snark:
In honor of the day the ML community first discovered the fact that (x1 / n1) + (x2 / n2) != (x1 + x2) / (n1 + n2)
I really like seeing these small but practical problems pop up, and seeing the community rally around to fix them. I missed this when it happened in October, so glad to look back at it now!
DeepSeek dropped another quality release with their R1 series. The models are an exploration of how to improve improving reasoning capabilities in LLMs. They’ve released a crop of models, which, despite some quirks in its licensing, are extremely helpful. The paper continues the trend of being clear and open.
Core Approaches for Developing Reasoning Capabilities
DeepSeek’s approach to reasoning in LLMs is built on three distinct approaches
1. Reinforcement Learning (RL) on a Base Model (DeepSeek-R1-Zero)
DeepSeek-R1-Zero applies RL directly to the base model without relying on any supervised data.
Training Approach:
Leveraged Group Relative Policy Optimization (GRPO), earlier work by the same folks, a simplified version of PPO that avoids the need for a large critic model by basing optimization on group-level scores.
Effectively it generates multiple completions per prompt, scoring them using a combination of reward models and reward functions:
Accuracy rewards: Validating correctness for problems like coding or math with deterministic solutions.
Format rewards: Encouraging structured reasoning traces using tags.
Rewards were averaged across completions, with constraints applied using KL divergence to maintain proximity to the base model.
Emergent Behaviors:
Spontaneous reflection and backtracking during training.
Majority voting across completions further boosted the model’s reasoning accuracy.
Over time, the model naturally extended its “thinking” process, solving increasingly complex tasks.
I saw a bit of skepticism about the “Aha” moments (the model demonstrating backtracking), with suggestions that perhaps they had some O1 data in the training mix. GitHub – Jiayi-Pan/TinyZero came out today that reproduced the recipe from R1-Zero and saw the same behavior, based on Qwen-2.5 3B. While that doesn’t totally rule it out, I think it does likely indicate that this method is a pretty sound way of eliciting reasoning.
2. Supervised Fine-Tuning (SFT) with Chain of Thought (CoT) + RL
Building upon the groundwork of R1-Zero, DeepSeek-R1 is a supervised finetune on DeepSeek V3 with reasoning trace/cha of thought data.
Data Preparation:
Generated ~600k CoT examples by iteratively fine-tuning on outputs from converging models (e.g. run it, get some good traces, add that to training set, rinse and repeat)
Supplemented the dataset with ~200k non-CoT examples to ensure the model learns to use CoT selectively and appropriately (they used data from the preexisting Deepseek v3 fine tuning set)
Training Process:
Fine tune on the 800k
Applied RL exactly as with R1-Zero to refine reasoning capabilities, using diverse prompts (including some non-Chain-of-Thought ones) and classic preference alignment techniques (e.g. don’t chain of thought “hello”) to optimize response quality and coherence.
Outcome:
This model seems both good at reasoning, and generally pretty strong. Its not everything everywhere, but this feels like a plausible recipe towards general purpose models, though they note it is a bit worse at tool use etc. than base V3.
3. Distillation to Small Dense Models
In a really interesting extension, DeepSeek distilled the reasoning knowledge of R1 into compact versions based on models like Qwen and Llama.
Process:
Fine-tuned smaller models with the 800k training examples used before, but didn’t do any fine tuning.
Distillation did not involve direct logic comparison, likely due to the challenges posed by token set differences.
RL stages were not applied to distilled models, but researchers suggested that doing so could enhance their performance further.
Challenges and Unsuccessful Attempts
I really like they called out some things that didn’t work:
Process Reward Model (PRM):
Attempted to reward reasoning steps individually rather than focusing on end results.
Faced challenges like reward hacking and inconsistencies in defining intermediate reasoning steps.
Monte Carlo Tree Search (MCTS):
Explored breaking problems into smaller parts and systematically searching for solutions.
Encountered exponential search space complexity, local optima issues, and difficulties in training reliable value models to guide search steps.
Additional Observations
Few-shot Prompting:
Observed that few-shot prompting degraded R1’s performance, a behavior also noted by folks working with OpenAI’s o1 series.
Software Engineering challenges:
Slow evaluations limited the application of large-scale RL to software engineering tasks.
They highlighted the need for methods like rejection sampling (have “not chosen” examples) or asynchronous evaluations to address these inefficiencies.
It is extremely cool to see that pure RL can push a model to improve reasoning capabilities. The approach of developing traces, fine tuning, and mixing in capabilities feels like a very practical approach as well. I am really looking forward to people exploring further!
Vita 1.5 is a decent MLLM with speech and vision capabilities and comes with a good paper that gives a clear recipe for how to take a solid LLM (in this case Qwen 2.5) and add audio and vision to enable it to do end-to-end speech interaction (you can see a video of it in practice!)
It can handle images and speech audio as input, and can handle video by sampling frames, treating them as sequential images, which allows it to manage video inputs without a dedicated video processing module. The interesting part of the paper is the training recipe they provide. They used generally available datasets, and some ones they sourced- 11k hours of speech-transcription data and 3k hours of text-speech data. The recipe goes in multiple stages of fine-tuning:
First, train the image understanding and question answering capability:
Take 20% of image-text caption data, pass images through a pretrained visual encoder, a vision adaptor that connects the encoder to the LLM, and send the text to the LLM directly. Freeze the encoder and the LLM, and just train the adapter.
Take all the image-text caption data and train the LLM, adapter and vision encoder.
Mix in 20% of the caption data and all of the image question/answer data and train all three components again
Next train the audio input:
Take speech-transcription data and train a speech encoder, a task guidance token, and freeze the LLM
Then they take a bit of the caption and QA data and train the speech encoder, speech adaptor, vision encoder, vision adapter, and the state head (that helps the model keep track of which modality its working in) and LLM at the same time. Half the time they replace the text in the caption or QA with the speech equivalent generated from a TTS system to exercise the speech system.
Finally, they train the audio output
Use the text speech data to train the codec encoder to speech tokens then from the decoder back to speech.
Encode the text with the LLM embeddings, run it through two speech decoders.
The DeepSeek V3 paper is very clearly written and goes into the infrastructure they used to get a very strong model on a relatively small amount of compute.
One example that stood out was the work they did to get efficient utilization on their expert routing.
In detail, we employ the warp specialization technique (Bauer et al., 2014) and partition 20 SMs into 10 communication channels. During the dispatching process, (1) IB sending, (2) IB-to-NVLink forwarding, and (3) NVLink receiving are handled by respective warps. The number of warps allocated to each communication task is dynamically adjusted according to the actual workload across all SMs. Similarly, during the combining process, (1) NVLink sending, (2) NVLink-to-IB forwarding and accumulation, and (3) IB receiving and accumulation are also handled by dynamically adjusted warps.
The name of the game in this kind of efficiency work is overlapping comms, and they are doing it at multiple levels: overlapping Infiniband (backend network) calls with NVLink (much faster) calls, using warp specialization to enable concurrency and overlap within a GPU etc. They have some feedback for hardware manufacturers on this:
We aspire to see future vendors developing hardware that offloads these communication tasks from the valuable computation unit SM, serving as a GPU co-processor or a network co-processor like NVIDIA SHARP Graham et al. (2016). Furthermore, to reduce application programming complexity, we aim for this hardware to unify the IB (scale-out) and NVLink (scale-up) networks from the perspective of the computation units.
Reminds me a little of the specialized comms cores in the Tenstorrent hardware!
They also train in FP8 – getting stable training with FP8 is non-trivial, so having a list of what components they included/excluded is very helpful for other folks that might be interested in similar training:
For this reason, after careful investigations, we maintain the original precision (e.g., BF16 or FP32) for the following components: the embedding module, the output head, MoE gating modules, normalization operators, and attention operators.
Though, there is some work to do to get the same performance:
One key modification in our method is the introduction of per-group scaling factors along the inner dimension of GEMM operations. This functionality is not directly supported in the standard FP8 GEMM.
A good paper on creating domain specific (medical in this case) reasoning LLM. I had been somewhat vague on how creating a reasoning model actually worked, and this paper felt very clear (as to one recipe at least). As I read it:
Start with a base model you are fine tuning for reasoning (in their case qwen 2.5) and a strong general model for verification and data generation (in this case GPT-4o)
Gather a bunch of problems with ground-truth answers. In this case they had medical problems. Take about half of them for fine tuning data.
Prompt the general model to think step-by-step for each problem to get some initial reasoning.
Use the general model to evaluate whether each answer reached is correct vs ground truth.
If the reasoning gets to the wrong answer (which is likely!) try a search strategy randomly – e.g. backtracking (start from an earlier step), critique-and-correct the existing line of reasoning, explore a new path distinct from the one given, or verify the current reasoning. Again, this is done by prompting the general model.
Each problem gets three tries to get to a correct answer before giving up. At the end of this process we have a series of reasoning traces that get to the correct answer. Pair these with the problems and used for the next steps, but first rewrite them into a chain of thought incorporating “hmms” and other smooth transitions between thoughts, via prompting the general model.
Fine tune the base model on the problem traces.
Do RL (PPO) on the fine tuned model with the other half of the problems, rewarding when the model is correct with reasoning (verified by the general model), giving a small reward for a incorrect answer but with reasoning, and giving 0 for no reasoning provided (regardless of answer). Also constrain with KL divergence from the fine tuned model, as is standard in RLHF etc.
This seems like a pretty reproducible recipe and the results seem strong. They include the prompts they use in the appendix, helpfully, and some good ablations/practical notes.
Effectiveness of Complex CoTs: We further examined the impact of different types of Chain-ofThought (CoT) reasoning. The results show that direct learning of response (yˆ) performs the worst, while simple CoT (y0, e0) offers only little benefit. In contrast, Complex CoT (yˆ0, eˆ) significantly improves performance by an average of 4.3 points. This demonstrates the importance of teaching models to refine their answers with reflection
One other interesting note on sourcing data was that they used GPT-4o to filter a number of multiple choice questions to generate the set of problems and ground truth. They used it to evaluate whether questions were complex enough to require reasoning, and whether they had a single clear and unambiguous answer. I am guessing it is a lot easier to get multiple choice question banks than other kinds, so this is a clever approach.
Breakdown of where generalization capabilities come from in RLHF – even more pertinent with the use of this for reasoning capabilities. Also includes one of the best concise overviews of RLHF I’ve read.
Fine-tuning with RL: Given a reward model, we employ RL to fine-tune the policy of a LLM. First, the policy is a language model that takes in a prompt and returns a sequence of text (or just probability distributions over text). The action space of this policy is all the tokens corresponding to the vocabulary of the language model and the observation space is the distribution of possible input token sequences, which is also quite large given previous uses of RL (the dimension is approximately the size of vocabulary ^ length of the input token sequence). The reward function is a combination of the preference model and a constraint on policy shift. Finally, the update rule is the parameter update of the policy from PPO that maximizes the reward metrics in the current batch of data [3].