Code-repair training data where every example proves itself by running.

Most training data is graded by another model's opinion. Mine is graded by execution: the broken program has to fail a hidden checker, the fixed program has to pass the same checker, both runs verified before an example is allowed to exist. No AI judges anywhere in the correctness path — and the eval ships with the data, so you check my numbers instead of trusting them.

See frontier models fail Get the data

17% → 23–40%

hard-tier fix-rate, tuned 7B — four runs, full spread published

AI judges in the correctness path

< $0.50

to reproduce the base number yourself

Support the research

This is one person paying for GPUs and API calls out of pocket. If you want more of this kind of work — verified data, published spreads, evals you can rerun — you can fund the next experiment directly. And because receipts are the whole point here, this is what money actually does in this factory:

≈ 2 pod-hours on the A40 that runs every eval — or one full 7B train + eval run

≈ one 32B scaling experiment, or the API cost of ~100 new execution-verified examples

$10

≈ a whole new frontier comparison — three models, 35 hard tasks, every transcript published

Estimates from my actual receipts (RunPod A40 at $0.45/hr + current API rates); the exact cost breakdown gets its own blog post. Donations aren't earmarked — they fund whatever experiment is next on the blog, and every result is published whichever way it goes.

Support on Ko-fi

The proof, not the pitch

Five faults frontier models don't fix — GPT-4o and Claude Sonnet 4/4.5 fail them; the $0.45-fine-tuned 7B doesn't. Every transcript published, including the table where the frontier wins overall.
I retrained my own headline four times — the full run-to-run spread (22.9–40.0%), published voluntarily. Numbers are floors here, not ceilings.

How it's made (the short honest version)

Faults are constructed backwards from working code, so the fix is known and provable. Every checker gets attacked with wrong-but-plausible code before its task can ship — if a bad fix can pass, the task is cut. On multi-file faults, fixing only one file provably still fails, so partial credit is impossible by construction. The generation pipeline itself is the one thing I don't publish; the proofs are the product, and you can run those.