Harness on Greg Herlein

It Was the Harness, Not the Model — 90% of It

Tue, 22 Sep 2026 00:00:01 -0800

I ran five different coding agents at the same local model, on the same task, with the same frozen test suite — and then I counted why they failed. The answer wasn’t subtle. About 90% of the failures were harness problems. Only about 10% were the model. And here’s the part that should change how you spend your next dollar: throwing a bigger, less-quantized model at it fixed none of them.

Right, But It Wouldn't Stop: Turning Thinking Off in the Agent Loop

Wed, 09 Sep 2026 00:00:01 -0800

Here’s a result that surprised me: the reasoning model wrote more correct code than the plain one — and still lost. It nailed all 32 test cases, then sat there and refused to declare itself done. Burned the whole time budget re-checking work that was already right. “Correct, but won’t stop” turns out to be a real failure mode, and it’s exactly the wrong one for an autonomous agent loop.

Harness Engineering Has a Name Now

Mon, 07 Sep 2026 00:00:01 -0800

A room full of CTOs and architects met in a Swiss village this summer and, without quite meaning to, gave a name to the thing I’ve been quietly building for months. They call it harness engineering. I’ve been calling it “the tooling is the new model.” Same animal. And the part they flagged as unsolved, I think I’ve got a first, opinionated stab at.