Tokens Are NRE, Not COGS

ai / inference / costs / agentic-coding / architecture / platform-engineering / hardware / finops

Right now, as an industry, we are running almost entirely on COGS and calling it innovation. Kelsey Hightower gave a talk at PlatformCon called Zero Token Architecture, and the whole thing compresses down to one sentence: infer once, export, run without inference. He’s right. I want to give you the vocabulary that makes it land in a budget meeting, because I work at a company that ships hardware, and we already have words for this.

He opens with a story about buying a very fancy track saw. He watched something like ten hours of YouTube picking one out and another fifty hours learning to use it, and by the end he felt like a carpenter. Then he plugged it in, moved too fast, and the blade jumped the track and came back at him at ten thousand RPM.

That’s his setup for the real point — asking Claude to deploy a Kubernetes cluster gets you a cluster, it does not make you a platform engineer — and it’s a good one. But I want to stay on the track saw a minute longer than he did, because I think the tool itself answers the question the talk raises.


The track costs you once. Every cut after that is free.

A track saw is not a better circular saw. It’s a circular saw plus a track. And the track is a jig. You align it once — carefully, slowly, measuring twice, cursing a little — and then every cut after that is straight, repeatable, and takes almost no thought at all. The expensive part is the setup. The cuts are free.

Freehand a circular saw and you pay the full cost of attention, judgment, and error on every single cut, forever. And eventually you will get one wrong.

That’s the whole argument. Everything below is bookkeeping.

NRE and COGS, For the Software People

If you’ve done hardware you already have this vocabulary in your bones. If you haven’t, here it is.

NRE is non-recurring engineering. It’s the money you spend once to make a thing manufacturable. The board layout. The injection mold. The test fixture. The tooling. It’s real money, it’s often uncomfortably large, and you write the check exactly one time.

COGS is cost of goods sold. It’s what each unit costs you after that. Parts, labor, freight.

You make an injection mold for plastic parts once - usually for a lot for actual money - and then every part that falls out of it costs you cents. Nobody 3D-prints a production run at forty dollars a part because 3D printing felt easier at the beginning. Well — some people do, briefly, and then someone in finance finds out.

I’ve sat in the rooms where these two numbers get argued about. When you ship physical product, the difference between NRE and COGS isn’t a metaphor. It’s the difference between a business and a hobby.

So here’s the thing. Tokens are a budget line now. My post from May on inference costs laid out the hard numbers on where that’s heading, and nothing since has made me less worried. Tokens are a real, growing, variable line item — which means they get the same treatment as any other cost, and the question for any token you spend is simple:

Which column does it land in?

Spend in the NRE column and you bought an asset. A library. A schema. A code generator. A binary. A jig. It cost you a pile of inference once, and it will run for years on an ordinary CPU for approximately nothing.

Spend in the COGS column and you bought one cut. Tomorrow the same request comes in and you buy it again, at the same price, forever.


One of these shapes is an asset. The other is a subscription you can't cancel.

So back to Kelsey’s point: right now, as an industry, we are running almost entirely on COGS and calling it innovation.

Think about that for a minute.

Your Agent Will Get This Wrong, and It’s Not Its Fault

Here’s the part I think is under-discussed, and it’s a naive-solution trap, not a malice problem.

Ask an agent to integrate with a service and it will write a direct API call. Ask it again in another file and it will write another direct API call. It will hand-roll the auth, the retries, the pagination, the backoff, the error mapping, and the timeout, from scratch, each time, slightly differently. And it will do this cheerfully, instantly, and without complaint, because it can, and because nothing in your request told it not to.

And each call site works. That’s what makes this dangerous. There’s no failing test. There’s no red build. Nothing is broken. You just quietly acquired forty implementations of the same idea.

Oh, and you are using Claude web or Claude Desktop? Yeah. Every time too.

Look — the software industry did not build abstraction layers because typing was expensive. We built them because understanding and maintenance are expensive. DRY was never a keystroke optimization. It was about having exactly one place where you have to be right, and exactly one place to go when you turn out to have been wrong.

Agents dropped the cost of keystrokes to roughly zero, and a lot of very smart people have quietly concluded that the reason for abstraction went with it.

It did not. It got worse. Forty hand-rolled call sites means forty places the retry logic is subtly different, forty things to review, forty things to patch when the vendor deprecates a header — and not one human being who has read all forty.

This is Kelsey’s ORM story running in reverse. My version of the story was the time we didn’t test a system built with an ORM at scale - and when we deployed it the sheer number of objects it created ran us out of memory hard and dropped the system. And of course, it did that on every startup. Days of outage. Almost lost us our largest customer (Eric Luke do you remember?).

Kelsey’s observation: ORMs hid the database, developers stopped learning how databases work, the queries got absurd, and the DBAs had to rewrite them by hand and hardcode them into the app. Knowledge rotted. The agentic version is worse, because agents skip the abstraction entirely and the knowledge never forms in the first place.

The library is not overhead. The library IS the exported artifact. It’s the durable form of an understanding you paid inference to reach. Throw it away and re-derive it at every call site and you paid for the understanding and then declined to take delivery. The *code may in fact be fungible - but the design of the abstraction is most definitly NOT.

Build the Lego blocks. Spend real money on them — this is the one place where lavish token spend is exactly right. Get them reviewed by a human who actually understands what they do. And test the hell out of them. Then spend a much smaller amount of tokens wiring blocks together, and let the runtime burn a thin trickle of inference on the fifteen percent of cases your if/else tree never covered.

The Three Tiers of Spend

Tier What it buys How often Where the tokens go
Build Libraries, schemas, generators, binaries — the jig Once, then on change Heavy. Deep reasoning, iteration, human review
Assemble Wiring blocks into a working system Occasionally Moderate
Run The residual cases the rules don’t cover Every request A trickle

Now read that table upside down and you have a depressing number of production agent architectures in 2026.

Teams are spending Tier 1 money at Tier 3 frequency and reporting it upward as a platform capability. Hoo boy.

The Part That Should Scare Your CFO

Software ate the world for exactly one reason. Say it with me: the marginal cost of another copy is zero.

Build it once, ship it a billion times, and your unit economics get better with every customer. That’s the whole magic trick. That is why software companies get valued the way they do and cement companies do not.

Runtime inference on every request puts a variable cost back on every transaction.

Sit with that one. Welcome to a cement company.

A system that pays for a model call per operation has a cost curve that does not bend. Ten times the traffic is ten times the bill. You have taken a software business and reintroduced a bill of materials. You are un-inventing software, one agentic loop at a time, and the only reason nobody has stopped you is that the volume is still small enough that the invoice is survivable. Mostly because it was being subsidized.

And the invoice, as Kelsey points out, is the one piece of technology in this entire stack that reliably tells the truth. Twenty bucks a month, nobody blinks. Two thousand, questions start. When the token burn matches somebody’s salary, we’ve gotten silly. Or fired.

Same Mistake, One Layer Down: RAG vs. CAG

This isn’t only an agent problem. The identical error shows up in retrieval, and it’s worth calling out because RAG-vs-CAG is having a moment on every architecture slide I see.

IBM has a clean explainer on RAG vs. CAG. Short version: RAG indexes your corpus into a vector database and pulls the top few relevant chunks per query. CAG — cache-augmented generation — front-loads the entire corpus into the context window, lets the model process it in one pass, and keeps the resulting KV cache around so later queries skip the reprocessing. CAG wins on latency and simplicity, loses on scale, and needs recomputation when the underlying data changes.

(Their numbers are dated, by the way — the video talks about 32K to 100K context windows and we are well past that. Which is precisely why CAG got popular. Bigger windows made “just put everything in” look free.)

Oh yeah… the OpenClaw/Karpathy approach that’s so popular of just putting all the data in markdown in a big folder? Unless you do something to control how much gets read into the LLM you are just burning NRE.

Here’s what I want to point out, and I think it’s the interesting bit: CAG, done properly, IS Zero Token Architecture. You pay an enormous cost once, capture the digested state, and reuse it. OK, so you periodically refresh it (using a very low cost, possibly even local embedding model). That’s firing the mold. That’s aligning the track. The instinct is exactly right.

But most of what gets called CAG in practice is not that. It’s stuffing the whole corpus into the prompt on every single call and hoping a cache discount covers it. Prompt caching is a discount, not an elimination. You are still paying to move and attend over the same material forever. Same corpus, same answer, new bill. That’s the freehand cut again, just with a bigger blade.

And honestly? The whole framing is too narrow. “RAG or CAG” assumes the knowledge has to be consulted by a language model at all. Often it doesn’t. Escalate deliberately:

  1. Precompute. If a question has one stable answer, that’s not retrieval — that’s a value. Put it in a table.
  2. Look up. Deterministic key, deterministic answer. No embeddings required.
  3. Retrieve. Now you genuinely need semantic matching over something large or fresh. Fine: RAG.
  4. Infer over context. Now you need synthesis across a bounded, stable corpus. Fine: CAG.
  5. Reason from scratch. Genuinely novel. Spend freely — and then export whatever you learned.

Each rung costs roughly an order of magnitude more than the one below it. Or so. Let’s not argue about the specifics since if we keep spending like that we’ll be unemployed.


The question was never RAG or CAG. It's how far down you can push each request.

Most designs I see start at rung four or five and never look down. Take IBM’s own help desk example — a 200-page manual, updated a few times a year. CAG is a perfectly defensible answer there. But twenty questions are going to be most of your ticket volume, and those twenty have fixed answers. Precompute them, route the tail to the model, and you just moved eighty percent of your traffic off the meter entirely.

The interesting question was never RAG or CAG. It’s: how far down this ladder can I push each class of request? You want to know what happens to Engineering in the AI age? THAT.

Tokens Are Joules

One more thing that I don’t think gets said plainly enough.

Inference is not an abstraction. It is electricity through silicon in a building that needs water and a grid interconnect. Tokens are joules.

Money can be raised quickly. Substations cannot. Sooner or later, “we’ll just buy more capacity” stops being a sentence you can finish.

We’ve run this arc before, slower. Early cloud was “just spin up more instances,” because compared to a procurement cycle it felt free. Then the bills got big enough to be visible at the board level, and an entire discipline appeared to deal with it — rightsizing, reserved instances, spot, and eventually a wave of repatriation by companies who did the arithmetic and did not care for the answer. Nobody arrived at FinOps because it was intellectually exciting. They arrived because somebody in finance asked a question they could not answer.

That question is coming for inference. It’s the same question it always was: what is our cost per transaction, and is it going down?

FOMO is what’s keeping that question from being asked out loud today. No executive wants to be the one who slowed down the AI initiative. That protection has a shelf life, and it expires the instant a competitor demonstrates the same capability at a tenth of the run rate.

It’s already starting. Kelsey described a company he visited in France that revoked unlimited token budgets and got something close to a revolt — teams genuinely could not articulate how they’d meet their previous throughput without it. He compares it to the “GitHub is down” reflex on social media, and that’s funny, but look at what it actually is.

That’s not a cost story. That’s an operational risk finding. If the tokens going away means the work stops, you have a dependency you never signed off on.

Yes, the Naysayers Have a Point

Let me not get too high on my own supply. There are three objections here that deserve real answers, and I’ve made two of them myself.

“Inference costs are collapsing, so this is a temporary problem.” Per token? Yes, dramatically. Per task? No. We spent every single efficiency gain immediately — on longer context, more reasoning, more loop iterations. Falling unit price met rising unit consumption and the bill went up anyway. Jevons has not been repealed. And volume growth outruns price decline: if your unit cost drops 10x while your traffic grows 100x, you have a bigger problem, not a smaller one.

“Premature abstraction is real.” Absolutely it is, and building the wrong Lego blocks is worse than building none. But this isn’t an argument against exporting, it’s an argument about when. You use inference to explore, to find the shape, to be wrong a few times. Then you export. That’s “infer once” in full — the first pass is supposed to be expensive and exploratory. The failure mode is never coming back to harvest it.

“Some things genuinely can’t be enumerated.” True, and important. Open-ended natural language, novel inputs, actual judgment calls. That’s the tail, and the tail is exactly where inference earns every penny. Spend there. The argument was never “no tokens.” It’s: know which of your requests are the tail, and stop paying tail prices for the head.

BOTH can be true. The technology is genuinely astonishing AND most of us are currently deploying it with the unit economics of a cement company. The industry keeps trying to pick one of those. Baloney. It’s both.

Conclusion

The engineering skill that’s about to matter is not prompting. I’ll say that again because there’s an entire cottage industry betting otherwise: it is not prompting.

It’s deciding what deserves inference at all — and having the taste to recognize, mid-loop, that the expensive thing you just did could be done once and exported.

And that’s not a new skill. It’s the oldest one we’ve got. It’s caching. It’s compiling. It’s why we have libraries instead of retyping the standard library into every program. It’s a machinist looking at a job and knowing the first hour goes into the fixture so the next four hundred parts fall out identical without anybody thinking about it.

I started coding on a TRS-80 with 4K of RAM, where you could feel every byte. Then memory got cheap and we stopped thinking about it, and mostly that was fine. Tokens are having their expensive era right now. Some of the habits we’re forming in this era are going to look really dumb in five years, and some of them are going to look like the thing that saved the company.

Align the track. Then make the cut, and every cut after it, for free.

So: go look at your token bill and ask which column it’s in. If you find something in the COGS column that should have been NRE, I’d genuinely love to hear about it — that’s the interesting part of this whole thing, and I don’t think anybody has a good playbook yet.

Drop me a note on LinkedIn: linkedin.com/in/gherlein.

 Share!