I ran five different coding agents at the same local model, on the same task, with the same frozen test suite — and then I counted why they failed. The answer wasn’t subtle. About 90% of the failures were harness problems. Only about 10% were the model. And here’s the part that should change how you spend your next dollar: throwing a bigger, less-quantized model at it fixed none of them.
Here’s a result that surprised me: the reasoning model wrote more correct code than the plain one — and still lost. It nailed all 32 test cases, then sat there and refused to declare itself done. Burned the whole time budget re-checking work that was already right. “Correct, but won’t stop” turns out to be a real failure mode, and it’s exactly the wrong one for an autonomous agent loop.
A room full of CTOs and architects met in a Swiss village this summer and, without quite meaning to, gave a name to the thing I’ve been quietly building for months. They call it harness engineering. I’ve been calling it “the tooling is the new model.” Same animal. And the part they flagged as unsolved, I think I’ve got a first, opinionated stab at.