The model can probably write the code

The current vibes in software engineering are a mix of crushing despair at years of accumulated personal skills being displaced by the CEO prompting some stuff, and crushing despair at years of corporate investment in an existing codebase that isn’t vibe-y enough. People worry whether the models will be effective in their programming language of choice, not on some general benchmarks.

One angle to approach that is to ask how well the language is covered by the distribution of the training data¹. An interesting paper the other day gave a pretty clear idea of how to check: 1-shot some prompts against the base model and see if they ever get it right. Getting access to base models is not always possible, but you can certainly call the post-trained models with roughly the same idea: no tools, no iterations, just generate this program.

To try this, I² wrote up 20 project-euler like³ puzzles of varying difficulties and had a few different models YOLO solutions in several languages. These ranged from common ones like Python to fairly rare ones like Zig and Hack.

After validating all the solutions, we can calculate some stats using pass@k: in k trials, how often did the model solve the problem. Here’s some stats for pass@1: what % of the time can you expect the model to one-shot the solution:

Lang	GPT-4.1 Mini	Gemini 3 Flash	OLMo 3.1	Kimi K2.5	GLM-5
Python	.93	.99	.72	.97	.98
Type Script	.94	1.00	.43	.95	.95
Go	.95	.91	.46	.86	.86
Rust	.89	.94	.43	.95	.95
Kotlin	.90	.99	.29	.91	.93
OCaml	.76	.86	.08	.94	.90
Zig	.14	.55	.00	.79	.88
Hack	.43	.76	.05	.47	.68

And here is the same thing for pass@128: what is the chance it is right at least once in 128 samples:

Lang	GPT-4.1 Mini	Gemini 3 Flash	OLMo 3.1	Kimi K2.5	GLM-5
Python	1.00	1.00	.95	1.00	1.00
Type Script	1.00	1.00	.90	1.00	1.00
Go	1.00	1.00	.85	1.00	1.00
Rust	.95	1.00	.88	1.00	1.00
Kotlin	1.00	1.00	.59	1.00	1.00
OCaml	.98	1.00	.38	1.00	1.00
Zig	.49	1.00	.05	1.00	1.00
Hack	.99	1.00	.46	1.00	1.00

To make that a bit more visual, here is a per-language chart for GPT-4.1-mini:

Given enough chances GPT 4.1-mini solves all the problems, in almost all the languages. Of course, we don’t actually know what GPT 4 was trained on, but we do know what OlMo 3.1 was trained on, thanks to the wonderful folks at AI2. That means we can see how much code-specific data for each language there was⁴:

Language	Code Corpus (GB)	Est. Tokens (B)	Category
Python	60.40	17.3	High-resource
TypeScript	26.52	7.6	High-resource
Go	23.78	6.8	High-resource
Rust	9.11	2.6	Medium-resource
Kotlin	5.68	1.6	Medium-resource
OCaml	1.03	0.29	Low-resource
Zig	0.18	0.05	Low-resource
Hack	0.00	0.00	Very-low-resource

There is a pretty decent correlation between the presence of training data and the pass@k rates. But, importantly, its not 1: despite Hack having no StarCoder data and Zig negligible, the model clearly does know at least something about them. Given enough chances it has a decent chance at coming up with the correct answer for Hack, and a non-zero one for Zig:

We have seen for human language that models learn a language substrate, enabling them to perform strongly even on tasks they haven’t seen such as translating between unseen language pairs. I suspect something similar happens with code: despite the language differences there is a logical programming substrate, and the model doesn’t need much exposure to the language in order to generalize to it.

Once you start giving the model multiple attempts, it gets into the right region quickly for the high-resource languages: with GPT-4.1 mini, Python, TypeScript, Go and Kotlin saturate at k=10. The less-common languages continue to rise: the model can write valid OCaml or Zig or Hack but need more attempts to stumble into the right region.

Thinking models flatten the curve substantially. Kimi K2.5 and GLM 5 both use high effort by default⁵, and that appears to give them multiple bites at the apple from internally exploring and self-correcting. By k=10 the models saturate all problems on all languages, though at the cost of a remarkable number of tokens⁶!

It’s also instructive to see the ways in the which the models get it wrong. There were four patterns that showed up:

1. Ecosystem: One problem involved a sum of very large digits. GPT-4.1 Mini regularly used num::BigUint. This is a crate, not a standard language feature, and in an agentic loop would probably be a very valid choice but doesn’t strictly work. In contrast, GLM-5, a thinking model, implements digit-by-digit multiplication from scratch with Vec<u32>.
2. API confusion: The model knows roughly what the code should look like, but chooses the wrong API. For example, OlMo generated while ... do ... in mixing OCaml’s while...do...done loop with Haskell’s do notation and OCaml’s let...in binding.
3. Surface-form invention: The model has a sense of how things stylistically look in the language, but doesn’t know the real API. GLM occasionally writes Zig with invented functions: std.mem.Allocator.alloc(usize, limit) (Allocator is a type, not a callable) or @intCast(usize, limit), which actually was valid syntax in earlier versions.
4. Systematic convention gaps: Models would regularly put in <?hh for the hack samples, which broke in modern Hack.

My takeaway from this is that models learn to code, not just to reproduce syntax. That means you can almost certainly post-train or prompt your way out of most programming language problems with any frontier model: while some models were still pretty poor at Zig even with a lot of tries, Gemini most certainly was not. I doubt the folks at GDM spent a whole lot of time on Zig evals⁷.

A well pre-trained model has broad capabilities in programming, and it’s mostly a case of eliciting them rather than having to teach them.

1. I’m going to take as a given that models are good at generalizing within the distribution of their training data, and poor at generalizing outside it. This is not settled! Reasonable people can disagree! But, it’s a decent starting point. ↩︎
2. Claude. ↩︎
3. Not actually project Euler. I confirmed that the models never respond with an actual Euler puzzle answer in the incorrect ones, so I’m fairly (this is not good science) sure it wasn’t memorization. ↩︎
4. OLMo’s full training corpus (Dolma v1.7) includes a massive web crawl in addition to code-specific data from StarCoder, so the 0.00 GB for Hack means “absent from code specific training ” not “absent from all training data”. Hack documentation and other content are almost certainly present in the web crawl portion. ↩︎
5. Gemini also reasons, but the 2.5 Flash model was doing minimal reasoning when answering.
   ↩︎
6. Somehow averaging over 3k per sample for GLM, I say while ruefully staring at my OpenRouter bill. ↩︎
7. By posting this on the internet I am guaranteed to be corrected, at length, by a Googler ↩︎