I spent last week reading the source of two serious pieces of agent infrastructure — Google’s Agent Substrate and ax, its “Agent Executor” — and holding them up next to my own little launcher, agentdev. I went in half-expecting to feel foolish about my homegrown thing. And I did. Substrate and ax are better. Genuinely, no-contest, better-engineered better. AND… they’re better at a problem I don’t have. Both of those are true.
A naval reactor will absolutely heat your coffee. It is also the wrong way to heat your coffee.
Let me set the stage, because the three things in this post operate at wildly different altitudes and the whole point is the distance between them.
The Three Things
Agent Substrate is a zero-trust runtime for running millions of sandboxed agents on Kubernetes. Not a figure of speech — millions is the design target. Every agent (“actor” in their vocabulary) runs inside its own gVisor or Cloud Hypervisor microVM, gets a short-lived SPIFFE cryptographic identity, starts on a default-deny network, and — this is the clever part — gets snapshotted out of RAM the moment it goes idle and restored in under 500ms when a request shows up. Because agents spend most of their lives waiting on a model or a human, Substrate multiplexes a huge registry of stateful actors onto a tiny pool of warm worker pods. 10x to 30x oversubscription. You pay for compute only while an agent is actually thinking.
ax sits on top of Substrate and makes it declarative. If you’ve used Kubernetes, ax feels like home: you write a Task, a Workspace, and a Model as YAML manifests, ax apply -f task.yaml, and the control plane schedules it as a Substrate actor. It even stores its state in PostgreSQL instead of etcd specifically because etcd falls over when you try to cram millions of short-lived task records into it. That’s not a toy decision. That’s someone who has hit the wall and engineered around it.
agentdev is mine. It is a bash script and a Python file. It launches one coding agent in one rootless podman container on one machine — mine — from one JSON config file. No cluster. No control plane. No database. Its entire dependency list is podman and python3. I wrote about the workstation it lives on in Where My Coding Agents Live; agentdev is the part that actually boxes the agent in.
So. One of these runs a planetary-scale multi-tenant agent cloud. One of them heats my coffee. Let’s be honest about the gap.
Where Substrate and ax Win (Give the Devil His Due)
I’m not going to soft-pedal this. There are real ideas in Substrate that my thing simply does not have, and a couple of them I’m a little jealous of.
Credential isolation. This is the big one. In Substrate, the agent never holds a secret. No API keys, no cloud tokens, nothing. When the agent makes an outbound call, a credential-injecting egress gateway — running outside the sandbox — adds the credential on the way out. So a prompt-injected or outright compromised agent has nothing to exfiltrate. It can’t leak a key it never had. That is a genuinely beautiful piece of security design, and it is the one gap in the analysis that actually nags at me. More on that below.
Per-agent identity. Every actor gets its own SPIFFE X.509 cert and can be minted a per-request JWT, carrying its identity and namespace, designed to federate with external identity systems. When you’re running agents for many mutually-hostile tenants on shared hardware, you need this. Each agent is a distinct, verifiable principal.
VM-grade isolation, always. Plain Linux containers aren’t even supported. Every actor is gVisor or a microVM, full stop. Mutually hostile tenants can share a physical node without sharing a security boundary.
Suspend/resume teleport. Snapshot an idle agent’s full state — RAM and filesystem — and restore it intact on any worker in the pool. ax exposes this as ax suspend / ax resume. In-flight tool state, working memory, all of it, preserved across hibernation. This is what makes the 30x oversubscription possible, and it’s hard systems engineering done well.
Scale and density. 500+ activations per second. Millions of actors on a small pool of warm pods. If you are a platform running agents for other people — a SaaS, a CI fleet, a research cluster burning through RL rollouts — this is the thing you want, and nothing I could write in a weekend comes within a thousand miles of it.
I mean it. It’s better. If the problem is “safely run a million untrusted agents for a thousand customers,” there is no argument to have. You run Substrate. You’d be insane to write your own.
But at What Complexity?
Here’s the other side of the ledger, and it’s the side the glossy READMEs never total up for you.
To run ax and Substrate, you need: a Kubernetes cluster. A container registry your cluster can pull from. PostgreSQL. A WorkerPool (which nothing creates for you — forget it and your first task hangs on ResourceExhausted: no free workers available). gVisor or Cloud Hypervisor on your nodes. An Envoy-based networking layer. A SPIFFE trust domain. An egress gateway with a credential-provider policy you have to configure per-namespace or every injection is default-denied. And — my favorite detail, the one that tells you exactly what world this software lives in — if your Linux host is going to run more than about a thousand sandboxes, you need to hand-tune the kernel’s neighbor table (net.ipv4.neigh.default.gc_thresh1/2/3) or the thing warns you it’s going to fall over.
Read that list again. That is not a tool. That is a platform. It is a standing system with an operations burden that never goes to zero. Someone has to own that cluster, patch it, watch it, pay for it, and get paged when the control plane degrades at 3am.
Now here’s agentdev’s entire operational footprint:
{
"image": "ghcr.io/gherlein/agentdev:latest",
"workspace": { "source": ".", "mode": "rw" },
"network": { "mode": "allowlist", "allow": ["api.anthropic.com"] },
"mounts": { "agent_folder": "~/.agentdev/homes/myproject" }
}
That’s it. That’s the whole run. podman and python3 are the only things that have to exist on the box. Read the file and you know exactly what the container gets — the workspace, the network reach, the folders mounted in. The config is validated fail-closed before anything launches, so a typo in a security setting aborts the run instead of silently turning the setting off. The agent’s home isn’t some opaque snapshot in a control plane — it’s a plain host folder I own. cp to fork it. tar to move it. git to version it. rm to wipe it. Because it’s rootless podman with --userns=keep-id, every file the agent writes is owned by me, not root.
Time from “clean machine” to “agent boxed and running”: a couple of minutes. Time from “clean machine” to “first Substrate actor serving a request”: well, clear your afternoon.
This is the whole argument in one line: complexity is not a one-time cost. It’s a subscription. You pay it every day you own the system, forever.
Different Problems Wearing the Same Words
The trap here — the thing that had me feeling foolish on day one — is that both projects use the same vocabulary. Sandbox. Agent. Workspace. Network policy. Isolation. It sounds like they’re solving the same problem and mine is just the sad, underpowered version.
They are not solving the same problem.
Substrate’s problem is multi-tenancy at scale: many untrusted agents, many hostile tenants, shared hardware, run economically. Every single one of its hard parts — SPIFFE identity, VM-grade isolation, credential injection outside the sandbox, snapshot/restore teleport, oversubscription — falls directly out of that problem. Take multi-tenancy and scale away and most of that machinery has nothing left to do.
agentdev’s problem is software for one: I am a single developer who wants to run a handful of agents on my own box without them trashing my home directory, reading my real ~/.claude, or phoning home to wherever. Single tenant. The tenant is me. I’m not mutually hostile with myself.
“Better” is not an absolute. It’s a ratio, and the denominator is your actual problem. Substrate is better per dollar of risk when you’re running a million agents for strangers. agentdev is better per minute of my life when I’m running six agents for me. A naval reactor is an objectively superior heat source to my kitchen stove. I am still not going to install one to make coffee, and the nuclear-trained part of my brain is the part that’s most sure about that.
The One Gap I Can’t Wave Away
I won’t pretend the comparison comes out totally clean, because it doesn’t, and I’d rather be honest than tidy.
The credential-isolation idea is right, and it’s right at every scale, including mine. In agentdev, secrets live in a .env or .envrc in the workspace — which means the agent can read them. My mitigations are real but indirect: the network can be locked to an allowlist with a filtering proxy and a sink that refuses DNS and logs every bypass attempt, so even if an agent grabbed a key, it has nowhere to send it that I didn’t bless. That shrinks the blast radius a lot. But it’s defense-in-depth around a secret the agent still physically holds, not Substrate’s cleaner “the agent never had the secret in the first place.”
That’s the one I’m going to keep chewing on. I think there’s a narrower version of the egress-gateway trick that could work for a single box — park the credentials in the proxy sidecar and inject them on the way out, so the agent only ever sees an unauthenticated local endpoint. It wouldn’t be SPIFFE and it wouldn’t federate and it absolutely would not scale to a million tenants. It’d just close the one gap that actually matters when the threat is prompt injection rather than a hostile co-tenant. Good artists borrow. I know where I’m borrowing from.
So What’s the Lesson?
Read the better system. Steal its best idea. Do not copy its complexity.
That’s the whole thing. Substrate and ax are a master class in what agent infrastructure looks like when you take untrusted, mostly-idle, stateful agents seriously at scale — and reading their source made my own tool better by showing me exactly which of my gaps are real (credential isolation) and which are just scale problems I’ll never have (SPIFFE, teleport, oversubscription). That’s worth a week of anyone’s time.
But the instinct to reach for the heavyweight platform because it’s “more serious” or “what Google uses” is how solo builders and small teams drown themselves in operational burden they will never pay back. You don’t need etcd’s successor and a tuned kernel neighbor table to run coding agents on your laptop. You need a container, an honest config file, and a network you can reason about. If you’re buying your own metal to run agents or just reclaiming the ability to ship as one person, the right answer is almost never “stand up a cluster first.”
Pick the tool for the scale you actually have — not the scale that makes you feel like a real engineer. Those are rarely the same number.
And yes — if your scale really is millions, go run Substrate, and send the team a thank-you note, because the work is excellent. Just don’t install a reactor to heat your coffee. I say that with love, and with four years of nuclear-power training backing it up.
Both can be true.