I retrained my own headline number four times — here is every result

2026-07-19 · solo

Last week I published a number: fine-tuning Qwen2.5-7B-Instruct on my verified code-repair data took its fix-rate on hard held-out faults from 17.1% to 40.0% (McNemar p=0.039, decision rule frozen before the run, correctness graded by executing the code — never by another model's opinion).

Then I did the thing you're not supposed to do to your own marketing: I ran the exact same frozen recipe three more times — same 882 training examples (SHA-verified), same held-out set, same script — and I'm publishing every result.

Every run

The base model scores 6/35 in every run, bit-identical — the measurement is stable. The training is not: GPU arithmetic isn't deterministic, and on a 35-task tier that's enough to swing the number. My published run was the luckiest of four.

What holds across all four runs: the direction. 29 hard tasks improved against 7 regressed, and every single run moved more tasks up than down. The honest claim is therefore not "40%". It's: 17.1% → 22.9–40.0% across four runs, median 34.3% — a doubling at the median, direction consistent in all four, significance in one. If you train on this data, expect the spread, not the best draw.

Why I'd rather publish this than hide it

Because the alternative is worse. If I only show the best run, the first buyer who retrains gets a smaller number, and my credibility — the only asset this company has — is gone. Numbers are floors here, not ceilings. This table ships in the eval bundle, and the dataset card gets it with the model release.

Two things I learned doing this: first, single-run significance on a 35-task tier is a coin with a thumb on it — the fix is more hard tasks, which is where the factory is pointed next. Second, batched greedy decoding is not identical to batch-size-1 greedy decoding (floating-point order flips borderline generations), which is why every transcript I publish is graded on its own generation path, never assumed from the eval run.

Where this came from

The reason I measure this way at all: early on I had a beautiful number — +42.5% improvement — that turned out to measure style-mimicry, not skill. Loss curves and judge scores both applauded; execution didn't. Since then the only headline metric is fix-rate: apply the model's fix, run a hidden checker it has never seen, pass or fail. Everything else is decoration.

What's next

• A matched 32B experiment is running right now — same frozen train file, same held-out, same decision rule, model size isolated. The result gets its own post, whichever way it lands.
• The tuned 7B is being released so you can run it yourself — including five faults it fixes that frontier models don't.
• An exact cost breakdown from provider receipts. No round marketing numbers.

If you train models on code data and want numbers you can reproduce instead of trust: write me, request access to the dataset — or fund the next experiment.