This week I’ve been at the Triton and PyTorch conferences in San Jose, and yesterday I had a chance to browse around the PyTorch poster presentations.
There’s a lot of talk about agentic development, of course, and about how to scale it. Much of that comes down to verification. But what are we verifying against? The theme from many of the posters was: we are verifying the code against… some other code. Which is odd! We could just check whether our models are working properly, but models are surprisingly forgiving: they will happily train straight through a bug.
Folks from ByteDance presented OpGuard which compares two training runs op-by-op and reports the first op where bits differ. They found race in PyTorch’s embedding backwards that had been there for 7 years. The bug hardly affected the loss (about 1e-5!)
LinkedIn presented their people search, which scores member matched via embedding on GPU in FP8. Compared to their FP16 reference about a third of the top 10 results shifted, but the LLM scored relevance didn’t move a bit. This forgiveness is a nice feature most of the time, but makes validation difficult. LinkedIN ended up keeping the FP16 copy in DRAM and reranks the FP8 candidates with it. Metric wise it isn’t it was just… there first.
These reference implementations are becoming a kind of an extant spec, that will be then processed by agents under some set of constraints around exactly how close an optimized version has to be.
Guilherme Leobas at OpenTeams (building on an idea from Richard Zou) has been rebuilding Dynamo’s (the PyTorch compile front end) object model to mirror CPython’s directly, using Python’s own test suite as the reference. The HuggingFace folks have been enabling on-device models for WebGPU, where they match against a CPU reference for correctness.
Nvidia presented a “Day-0 RL” poster that was basically a checklist for using something as a reference: use identical weights and inputs, check trainer and rollout log-probs agree, and monitor more than just the reward score. As if to demonstrate why, AMD presented on getting DeepSeek V4 Flash RL on MI355X. They had SGLang doing rollouts and Megatron doing training, and ran into disagreements. It wasn’t the usual suspects either, SGLang scored experts with a sigmoid where Meatron used sqrt softplus1, and there were some scale and clamp config differences. But… which is “right”? They ended up using an independent HuggingFace version as the reference, to break ties.
There were some hallway track discussion about formal methods for kernels and models. Maybe by next year we won’t have to test to see whether two things are equivalent, we’ll be able to prove it! Still have to decide which is “right” though.
- This is DeepSeek’s actual score function. It probably didn’t change the expert picked, and I am guessing that SGLang reused an existing module because it already worked! ↩︎