Harness Engineering Has a Name Now

A room full of CTOs and architects met in a Swiss village this summer and, without quite meaning to, gave a name to the thing I’ve been quietly building for months. They call it harness engineering. I’ve been calling it “the tooling is the new model.” Same animal. And the part they flagged as unsolved, I think I’ve got a first, opinionated stab at.

What the Report Actually Said

In late June, Thoughtworks and Martin Fowler ran a second Future of Software Engineering retreat in Engelberg, Switzerland — an unconference of senior technologists, CTOs, CEOs, architects. (The full report is a PDF.) Forty sessions, participant-led, sometimes argumentative. The value wasn’t consensus, it was the range and seriousness of the thinking. And two of their headline findings stopped me cold, because I’d already written both of them.

The first:

“As models commoditize, multiple sessions converged on the same claim: the scaffolding around a model — context management, deterministic guardrails, skills, self-improving feedback loops — is what actually differentiates good agentic engineering from bad.”

They even reported the punchline as measured fact: a smaller model with a good harness was said to outperform a larger model with a weak one.

The second finding is the one that’s actually interesting, because it’s an admission of ignorance:

“Governance of shared harnesses/skills remains an unsolved organizational problem. Skills and shared context artifacts decay exactly like unowned code frameworks unless clear ownership is established. However, centralizing into a dedicated ‘harness team’ risks recreating the old ops team anti-pattern.”

So: the harness is the differentiator, a good harness beats a bigger brain, and nobody in a room full of very smart people knows how to own the harness without recreating the exact centralized-ops mess we spent a decade escaping.

They asserted the first part. I want to show you my work on it — because I ran the experiment. And then I want to take a real swing at the second part, because I’ve been building a tool that only exists to solve it.

I’ve Been Calling This “The Tooling Is the New Model”

Back in March I wrote that the tooling is the new model. The argument was simple and, at the time, a little contrarian: the frontier models are already good enough for most real work, the gap between the top few is shrinking every month, and so the thing that actually determines whether your agent ships good software isn’t which model you picked. It’s everything wrapped around the model.

That “everything wrapped around” is the harness. Let me define it plainly, because the word is doing a lot of work:

• Context management — what goes into the prompt, in what order, and what gets left out. This is most of the game.
• Tools — what the agent can actually do, and how those calls are shaped and validated.
• Skills — reusable, portable instructions that teach an agent how to do a specific thing well.
• Guardrails and loops — deterministic checks, permission boundaries, and the feedback loop that lets an agent fail, notice, and try again.

None of that is the model. All of it is engineering. And it turns out it’s the engineering — the part that separates an agent that quietly corrupts your codebase from one you’d actually trust. The retreat gave it a name. Good. Named things get taken seriously.

I Measured It, So You Don’t Have To Take a Room of CTOs’ Word For It

Here’s where the engineering instinct kicks in. “A good harness beats a bigger model” is a great line. It’s also exactly the kind of claim that deserves data, not vibes.

So a few weeks ago I ran the experiment. I wrote about it in AI Agents are a Context Management Strategy in a Trenchcoat, and the setup was deliberately clinical: one host, one model, one frozen task, one frozen test suite the agent never sees. The only free variable was the agent itself — the harness. Same Qwen3.8-27B weights behind both contenders, same sampling parameters, same everything underneath. If one won, it couldn’t be because it was calling a smarter brain. It was the same brain.

The matchup was a heavyweight (oh-my-pi — rich tool set, subagents, planning modes) against a featherweight (hax — a minimalist agent in C, small tool set, small system prompt). And the featherweight won, and not by a little:

hax reached a finished solution 1.82x faster than omp — median 45.7 minutes vs. 83.4 minutes. The Mann-Whitney p is 0.00033, and the distributions barely touch. It also used half the tokens.¹

Correctness was a tie — both hit 32/32 — and I’ll repeat the caveat I made then, because honesty matters more than a clean story: that tie is a ceiling effect, not proof they’re equally good. The task was too easy, the needle pegged. If you want to separate agents on correctness, the spec has to get meaner. That’s a post I still owe you.

But the time result is real and it’s the point here: with the model nailed down as a constant, the harness moved the outcome by nearly 2x and halved the cost. That’s the retreat’s headline finding, reproduced on a desktop box in my office. The harness isn’t a nice-to-have wrapped around the smart part. On a fixed model, the harness is the variable that matters. And — importantly, for this task — leaner won. More machinery is not the same as a better harness.

Let me be crystal clear about the scope of that claim, because it’s easy to over-read. The goal I optimized for was narrow and specific: a relatively easy coding task, delivered to a satisfactory, tested result, in the shortest wall-clock time. That’s it. That’s the whole target. I am not claiming leaner always wins, or that hax is a better agent than omp, or that fewer tools is universally correct. Other coding tasks — big refactors across a sprawling codebase, gnarly debugging, anything that genuinely needs planning and subagents — could easily flip this result on its head, and I’d expect the heavyweight to start earning its weight as the problem gets harder. All I did was hold the variables to something steady to get a baseline. You can’t reason about “does the harness matter” until you’ve pinned everything else down and measured one thing. This is one thing. It’s a starting point, not a verdict.

That’s the easy half. I can measure “does the harness matter.” Yes. Emphatically. Now the hard half.

The Unsolved Half Is Governance

Go back to that second quote. Skills decay like unowned code. A central harness team recreates the ops anti-pattern. Nobody has an answer.

I believe this problem is real because I live it. I run more than one coding agent — not because I’m indecisive, but because I’m testing harnesses, and the field spits out a new one every week. Claude Code, pi, hax, oh-my-pi, fx. C, Rust, Zig, TypeScript, Go. Nobody’s converging on anything, and I think that’s healthy — different constraints produce different tools.

But every one of those agents wants my skills in its own directory, and I got tired of cp -r and stale copies. Five agents, five directories, five slightly different opinions about where your stuff goes. That drift — that quiet divergence of the same skill across five homes — is the micro version of exactly what the retreat is scared of at org scale. Skills decay when nobody owns the canonical copy.

Skills Are the Atom of the Harness

Here’s the thing I keep coming back to: a skill is not complicated. It’s a directory with a SKILL.md in it — YAML frontmatter with a name, an optional description, then markdown. Any agent that reads the format understands it. The content is wonderfully portable.

What is not portable is where it lives. And that’s the whole governance problem in miniature. The standard that survives isn’t a packaging format — every ecosystem grows its own packaging and every packaging format is a wall. The standard that survives is the boring one: a git repo of SKILL.md directories. Nothing agent-specific. Clone it and the files are right there.

So I wrote sm — skills-mapper. It’s a small Go binary that does exactly one thing: it takes your library of skills, keeps it fresh from git, and puts it where every agent expects to find it. It never copies and it never transforms. There is one canonical copy of every skill, and every agent points at it with a symlink. Update the source, run sm sync, and all five agents get the new version simultaneously — because they were all looking at the same file the whole time.

That’s not a clever tool. It’s a symlink manager with a manifest. But the boring mechanism is the governance answer hiding in plain sight.

My Actual Proposal: Distributed, Versioned, Tested — Not a Priesthood

The retreat’s fear is a false binary: either skills rot from neglect, or you build a central harness team that becomes the new bottleneck. I don’t think those are the only two options. Harness pieces should be treated the way we already treat good code — as versioned, testable, distributed artifacts owned where the work actually happens. Four opinions I’d defend, each of which is baked into sm’s design for a reason:

One canonical source, never copies. The moment you have two copies of a skill, you have drift, and drift is decay. Symlinks-from-git means there is exactly one truth and it’s version-controlled. Governance starts with “there is one of these, and it has a git history.”

Fail loud, never guess. If two sources both ship a skill called debugging, sm stops and tells you. It does not pick a winner and it does not quietly alphabetize its way into a decision you discover three weeks later when the wrong skill fires. Silent success is the worst failure mode in tooling. A harness that guesses is a harness that lies to you.

Touch only what you made. sm tracks every link it created and refuses to clobber anything it didn’t. I have a healthy paranoia about tools that write into ~/.claude unsupervised, and governance without that paranoia is just a new way to break people’s setups.

Ownership lives with the work, not in an ops team. This is the one that actually answers the retreat’s anti-pattern worry. sm has a project scope: a skills.toml at your repo root that teaches any agent — Claude Code, Copilot, whatever clones the repo next Tuesday — how to work on that codebase. That’s ownership living with the code, versioned in the same repo, reviewed in the same PR. It’s CODEOWNERS for your harness. Nobody stands up a central team to own it, because it’s owned exactly where it’s used. That’s how you get governance without a priesthood.

And there’s a fifth piece I haven’t built yet but I’m convinced is the missing link. The retreat wants “self-improving feedback loops” — agents that fail, reflect, and propose edits to their own harness. I love that idea and it terrifies me in equal measure, because a skill that rewrites itself with nothing checking it is just a very sophisticated way to introduce a regression. I’ve argued for a while that the tests are the real spec. Skills need the same discipline. A skill should ship with conformance tests — a small, mechanical check that says “this skill still does what it claims.” A self-improving loop is only safe if there’s a test on the other side of it that turns red when the improvement is actually a regression. That’s the governance layer nobody’s built yet, and it’s the same insight as the retreat’s other headline finding: verification, not generation, is the bottleneck. It applies to the harness too.

What Doesn’t Work Yet

I’m not going to pretend this is finished, because it isn’t. sm’s add, remove, and update commands are stubs — you edit the manifest by hand and run sync, which is genuinely fine for how I use it but is not done done. The conformance-test layer for skills is a conviction, not a shipping feature — I don’t have a cross-agent skill-test harness yet, and building one is the next real piece of work. And the governance model above is a proposal backed by one small tool and one person’s paranoia, not a validated org-scale practice. The retreat had forty sessions of smart people and landed on “unsolved.” I’m not claiming to have solved it. I’m claiming to have found the right shape of the answer, and to have built the smallest possible version of it.

Conclusion

Harness engineering is real, it’s ownable, and as the models commoditize it’s where the differentiation is going to live. A room full of CTOs said so, and I’ve got the benchmark that backs it up: with the model held constant, the harness moved the result nearly 2x and halved the token bill. That’s not a rounding error. That’s the whole ballgame.

But the win isn’t hiring a harness team and centralizing your skills into a new ops silo — that’s just the old bottleneck wearing a new badge. The win is treating your harness the way you treat your best code: one canonical source, versioned in git, loud when it conflicts, owned where the work happens, and — the part we all still owe each other — tested, so a self-improving loop can improve instead of quietly rot. Skills as distributed, testable artifacts. Not a priesthood.

The retreat also warned that the hype cycle is compressing, and the smart move is to build durable capability that holds its value wherever the cycle lands. This is that. Models will keep leapfrogging each other and it won’t matter much. A well-governed harness is capability you keep.

If you’re running more than one agent — or if you think my “ownership lives with the code, not an ops team” stance is naive and want to tell me why — drop me a note on LinkedIn. I’d genuinely rather be told what I’m getting wrong now than find out at org scale later.

 Share!

• Share on Facebook
   
• Tweet
   
• Share on Google+
   
• Post to Tumblr
   
• Pin it
   
• Submit to Reddit