Five faults frontier models don't fix (and a $0.45 fine-tune does)
On 2026-07-19 I ran GPT-4o, Claude Sonnet 4 and Claude Sonnet 4.5 against the 35 hard tasks of my held-out set — the same prompt, temperature 0, and the same hidden checkers that grade my own models. Full disclosure first:
The frontier models win overall. GPT-4o fixed 19/35, Sonnet 4 fixed 19/35, Sonnet 4.5 fixed 22/35. My tuned 7B fixed 12/35. This page is not "a 7B beats GPT-4o." It's the five specific faults where the big models fail and the small tuned one doesn't — with every transcript published so you can check me.
| fault | gpt-4o | sonnet-4 | sonnet-4.5 | tuned 7B |
|---|---|---|---|---|
202f4a13 template re-render | fail | fail | fail | pass |
73db72e9 revenue report crash | fail | fail | pass | pass |
99b8b877 UTC-naive timestamps | pass | fail | fail | pass |
d2e3f44f CRLF breaks CSV→JSON | fail | pass | pass | pass |
f62d646f idempotency violated | pass | fail | pass | pass |
The star is 202f4a13: all three frontier models fail it — and
they don't even agree on which file the fault is in. GPT-4o rewrites
renderer.py; both Sonnets rewrite parser.py. All
three fixes look plausible. All three fail the checker. The tuned 7B fixes the
right file.
The five, in one line each
202f4a13 — the template engine remembers the previous render
Rendering "Hello, {{ name }}!" with "Alice" works; rendering
again with "Bob" still prints "Alice". State leaks between renders across two
files. Multi-file: fixing one file provably still fails.
73db72e9 — monthly revenue report crashes with AttributeError
CSV loading is fine; the report module blows up on a type mismatch between what the loader stores and what the report expects.
99b8b877 — timestamps lose their timezone
Events come back with naive datetimes; the contract says UTC-aware. The fault sits in a helper other code silently depends on.
d2e3f44f — Windows line endings poison the output
CSV→JSON conversion fails on CRLF input with a stray carriage return inside a value.
f62d646f — "idempotent" import inserts duplicates
Running the same import twice should insert zero new rows the second time. It inserts three.
Check me
The full bundle — each fault's issue report, broken code, the (now-public) checker, all frontier transcripts verbatim, and my model's passing fix — ships alongside the model release. Grading rule, for every model equally: the proposed fix is applied to the broken tree and the hidden checker runs; pass means the checker genuinely completes. Publishing these five retires them from my held-out set — a public checker is no longer hidden, so they will never be part of a future headline number.