It Was the Harness, Not the Model — 90% of It

Tue, 22 Sep 2026 00:00:01 -0800

I ran five different coding agents at the same local model, on the same task, with the same frozen test suite — and then I counted why they failed. The answer wasn’t subtle. About 90% of the failures were harness problems. Only about 10% were the model. And here’s the part that should change how you spend your next dollar: throwing a bigger, less-quantized model at it fixed none of them.

Quantization on Greg Herlein

It Was the Harness, Not the Model — 90% of It