Code-repair training data where every example proves itself by running.
Most training data is graded by another model's opinion. Mine is graded by execution: the broken program has to fail a hidden checker, the fixed program has to pass the same checker, both runs verified before an example is allowed to exist. No AI judges anywhere in the correctness path — and every result on this site ships with the receipts to back it.
Support the research
This is one person paying for GPUs and API calls out of pocket. If you want more of this kind of work — verified data, published spreads, results you can read in full — you can fund it directly. Donations aren't earmarked: they fund whatever experiment is next on the blog, and every result is published whichever way it goes, wins and losses alike.
What your support buys is more published results — another training run, another frontier comparison, another batch of execution-verified data — each one shipped with the receipts to check it yourself.
The proof, not the pitch
- Five faults where frontier models miss — on these specific tasks, graded through the same hidden checker, GPT-4o and Claude Sonnet 4/4.5 did not produce a passing fix; the fine-tuned 7B did. Every number published, including the table where the frontier models win overall.
- I retrained my own headline four times — the full run-to-run spread (22.9–40.0%), published voluntarily. Numbers are floors here, not ceilings.
How it's made (the short honest version)
Faults are constructed backwards from working code, so the fix is known and provable. Every checker gets attacked with wrong-but-plausible code before its task can ship — if a bad fix can pass, the task is cut. On multi-file faults, fixing only one file provably still fails, so partial credit is impossible by construction. The generation pipeline itself is the one thing I don't publish; the proofs are the product, and you can run those.
Code repair is where this starts. The method — construct the fault backwards from a known-good answer, then prove the fix by execution — isn't specific to code; it works anywhere correctness can be run rather than judged. That's the direction. Code repair is just the first domain with the receipts to show it.