Agent Substrate Review: Running Millions of Sandboxed AI Agents on Kubernetes, and What the Density Claim Depends On
Agent Substrate suspends idle AI agents to snapshots and resumes them onto warm gVisor or microVM workers on Kubernetes. We read the v1.0.1 code, checked the numbers, sized a pool with a runnable script, and found what the headline omits: your app decides when to suspend, and egress rules scope credentials by host.
BY LUKAS HOFFMANN · FDEINTERVIEWS EDITORIAL · UPDATED NOVEMBER 20, 2026 · 10 MIN READ
PRACTICE THIS:Agent sandboxing, the concept ·Design the sandbox for agent-written code ·Self-hosted Claude Managed Agents ·System design questions
Agent Substrate is an open-source runtime from Google that runs sandboxed AI agents on Kubernetes at far higher density than one Pod per agent. It does this by noticing that agents spend most of their time waiting. A suspended agent becomes a snapshot of its memory and files; when a request arrives, the runtime restores it in under a second onto a worker Pod that was already running, possibly a different one from last time. Every agent runs inside gVisor or a Cloud Hypervisor microVM, holds no credentials, and reaches the network only through gateways. We read the repository at v1.0.1 (commit dc747ef8, 20 November 2026), its security docs and threat model, and Google's announcements, and ran our own numbers. It is a serious piece of infrastructure. Two things matter before you plan around it: the density you get depends on your traffic, and the decision to suspend is yours.
What it is
The project started on GitHub in May 2026, reached 1.0 on 5 November 2026, and on the same day Google donated it, with gVisor, to the CNCF at Sandbox status. Google's Cloud blog announced it on GKE in September 2026: open to all GKE customers for non-production workloads, with production support by allowlist. GKE's documentation requires Standard clusters on version 1.37 or later; Autopilot is not supported. The repository had 7,985 GitHub stars when we checked.
The core idea is to separate the actor (one agent instance and its state) from the worker (a warm sandbox Pod in a pool). Kubernetes still owns the slow, infrequent work: nodes, Pods, autoscaling. A small control plane owns the fast work: which actor is where, resume, suspend, routing. The reason is spelled out in the architecture doc. The Kubernetes API server "is not designed to handle millions of resources", and getting a Pod running in a couple of seconds is fine for a long-lived service but too slow for an agent that runs for milliseconds to seconds and then waits again.
The figure follows one request to a suspended agent, which is the case that matters. Your application sends it to atenet-router with a header naming the target actor. The router asks ate-api-server, which keeps actor and worker records in PostgreSQL, to resume the actor; the API server claims a warm worker, and the node agent atelet restores the actor's latest snapshot into the sandbox on that worker. Only then does the router open an mTLS tunnel to the worker and forward the request. No Pod is created and the Kubernetes scheduler is never asked.
| Piece | What it is | Where it lives |
|---|---|---|
| WorkerPool | A pool of warm worker Pods for one sandbox class | Kubernetes custom resource |
| ActorTemplate | Image, memory and settings for one kind of agent; used to build a "golden snapshot" | Substrate API, not Kubernetes |
| Actor | One agent instance: state (RUNNING, SUSPENDED), current worker, snapshots | PostgreSQL, through ate-api-server |
| Atespace | The grouping unit for policy, like a namespace | Substrate API |
Snapshots come in three cumulative levels, set per template: VOLUMES (durable directories only), ROOTFS (plus files written to the container's filesystem) and MEMORY (plus the running processes and RAM, the default). A MEMORY snapshot is tied to the sandbox runtime version it was taken on. If no worker offering that version has room, the resume falls back to ROOTFS and cold-boots, so the agent comes back with its files but without its in-memory state. If your agent keeps a conversation in a Python variable, that fallback is a lost conversation; keep anything you cannot afford to lose on a durable volume.
Suspend is your call
The hero image says idle agents suspend and the quickstart says that "when the actor becomes idle, Substrate checkpoints" it. The architecture doc is more precise: suspension is "triggered by an explicit SuspendActor call", and GKE's documentation talks about "letting your application suspend idle agents". The code agrees with the precise version. A package that lets a worker ask the control plane to suspend its actor exists at internal/ateworker/suspend, with a comment that "an idleness signal is a fact about the past, and it decays", but at commit dc747ef8 nothing imports it.
Resume, on the other hand, is automatic: any request routed to a suspended actor wakes it. So the orchestration layer you build on top (or a framework that integrates Substrate, such as kagent or Google's Agent Executor) owns the policy. That is the right place for it, because only your harness knows whether the agent is waiting for a model call that returns in three seconds or for a human who has gone to lunch. It also means a design review should ask who calls SuspendActor, on what signal, and what happens when that caller is down. Agents that are never suspended do not save anything.
The default suspend is also asynchronous. SuspendActor returns once the checkpoint is on the node's disk; atelet uploads it to object storage in the background. The docs say plainly that asynchronous suspends "risk state loss if the node fails before background upload completes". A synchronous mode exists for agents whose state you cannot lose.
What the numbers are
| Number | Source | What it is |
|---|---|---|
| Resume in less than 500 ms | README, architecture doc, Google blogs | Product claim |
| Resume "in under a second" | GKE documentation | Product claim |
| More than 500 activations per second | README, Google Cloud blog | Product claim |
| 10x density over standard containers | README, Google Cloud blog | Product claim |
| 10x to 30x oversubscription | README, about the demo video | Demo: about 250 actors on 8 worker Pods |
| 100 ms at p95, 1 billion agents, 1,000 wakeups per second per cluster | Architecture doc, its metrics section | Targets |
None of the published pages says what workload or hardware produced the claims. The repository does ship the means to find out: a Locust-based benchmark harness, a synthetic workload called glutton that holds a set amount of RAM and re-dirties part of it each cycle, and a runbook that breaks every suspend and resume into phases (download, restore, wake-up probe). Run it with your own memory size. A restore has to move or demand-page the snapshot, so an agent with a 4 GB working set will not resume like a counter demo.
The density claim needs a different kind of care. How many agents share a slot is not a property of the runtime; it is a property of how often your agents are busy and whether they are busy at the same time. A short script makes the point. It asks how many concurrently running slots a pool needs so that demand exceeds it no more than 0.1% of the time, assuming each agent is independently busy some fraction of the time, with an optional burst where everyone gets busier at once (a Monday 9 a.m.).
from math import lgamma, log, exp
def tail(n, p, s):
"""P(more than s of n actors are busy at once), each busy with probability p."""
total = 0.0
for k in range(s + 1, n + 1):
total += exp(lgamma(n + 1) - lgamma(k + 1) - lgamma(n - k + 1)
+ k * log(p) + (n - k) * log(1 - p))
return total
def slots_needed(n, mix, risk=0.001):
"""Smallest slot count s with P(demand > s) <= risk.
mix is a list of (share of time, busy probability) regimes."""
for s in range(n + 1):
if sum(w * tail(n, p, s) for w, p in mix) <= risk:
return s
cases = [
("250 agents, busy 5%", 250, [(1.0, 0.05)]),
("250 agents, busy 10%", 250, [(1.0, 0.10)]),
("250 agents, 5% + a 30% burst 10% of the time", 250, [(0.9, 0.05), (0.1, 0.30)]),
("10,000 agents, busy 5%", 10_000, [(1.0, 0.05)]),
]
print(f"{'case':<48}{'mean busy':>10}{'slots':>7}{'agents/slot':>13}")
for name, n, mix in cases:
mean = n * sum(w * p for w, p in mix)
s = slots_needed(n, mix)
print(f"{name:<48}{mean:>10.1f}{s:>7}{n / s:>13.1f}")
case mean busy slots agents/slot
250 agents, busy 5% 12.5 24 10.4
250 agents, busy 10% 25.0 41 6.1
250 agents, 5% + a 30% burst 10% of the time 18.8 92 2.7
10,000 agents, busy 5% 500.0 569 17.6
We checked the two 250-agent edge cases with a Monte Carlo run of 40,000 draws each: 24 slots exceed demand 0.08% of the time and 23 slots 0.2%; 92 and 91 slots land at 0.075% and 0.10%.
Read the chart from top to bottom. Agents busy 5% of the time, independently, give you the advertised 10x. Double their busy time and density falls to 6x. Add a correlated burst and it falls to 2.7x, because the pool has to be sized for the burst even though the average barely moved. A larger pool recovers density, since independent demand smooths out as it adds up: 10,000 agents busy 5% need only 14% more slots than their average. So the first thing to measure for a customer is not resume latency. It is the busy fraction per agent and how correlated it is across agents, which their own logs usually answer.
When the pool does run short, the router parks the request and retries the resume with backoff for up to five seconds by default, then returns 503. Parking absorbs a moment when every worker is taken. It does not fix a pool sized for the average.
The security model, read carefully
The design treats every actor as untrusted and every pair of actors as mutually hostile, and most of the hard parts are done. Every actor runs in gVisor or a microVM. Each gets its own network namespace with no path to other actors, the node, the Substrate API, the Kubernetes API or the database. Ingress arrives only through the authenticated router; egress leaves only through atenet-egress, which fails closed. Every active actor gets a one-hour SPIFFE X.509 certificate, and the gateway can mint per-actor JWTs for upstream services that trust Substrate's issuer. Sandboxes never touch snapshot storage; on GCS, atelet moves snapshots with credentials scoped to one actor.
The project's own security page marks every principle as met, in progress, future or out of scope, which is more candid than most. These are the rows that are not met yet:
| Principle | Status at v1.0.1 |
|---|---|
| Scrub local state between actors on a reused worker | In progress |
| Worker Pod privileges | In progress: still root with SYS_ADMIN and NET_ADMIN, no user namespace |
| Contain a compromised node to its own actors | In progress |
| Per-actor snapshot encryption; re-check authorization on restore | Future |
| Network policy and quotas between atespaces | In progress |
| Actor-to-actor authorization; rate limits and quotas | Future |
| Taint snapshots of a compromised actor so they cannot resume | Future |
None of these is unusual for a project that hit 1.0 two weeks ago, and the sandbox is still the primary boundary. They do decide whether you put two of a customer's tenants on the same nodes this quarter.
The credential design deserves its own look, because it is the part people quote. An actor never holds an API key. It sends a request with a placeholder header, the egress gateway checks the request against the actor's EgressPolicy, and when the deciding rule says so, the gateway swaps in the real credential from a provider (Kubernetes Secrets or Google Secret Manager). HTTPS works because the gateway terminates the actor's TLS for the hostnames a rule lists, so the actor has to trust the gateway's CA. The key never enters the sandbox, so it never lands in a snapshot.
The figure shows what that protects and what it does not. We read the rule evaluation in internal/egresspolicy: a request is decided on its hostname and port, and the rule schema has no field for HTTP method or path. A prompt-injected agent cannot leak the GitHub token, but it can call any GitHub endpoint the token allows, with the token attached for it. The README's phrasing, that "a prompt-injected or compromised agent has nothing to exfiltrate", is true of the bytes and not of what they authorize. Two fixes work today: issue the narrowest token the task needs (a read-only, single-repository token injected on a read path), or put your own policy enforcement point in front, which the extensibility doc supports for egress, although the installer does not have a bring-your-own mode yet.
The project also lists what it leaves to you. Your application authenticates end users and decides which actors they may reach, keeps the registry of agents, and records lineage (which agent created which sub-agent).
When it fits, and when it does not
Agent Substrate fits a platform that hosts many stateful, per-user agents that are idle most of the time and run code nobody reviewed: coding agents, per-user assistants, tool servers. Its demos run Claude Code and an Alpine shell sandbox, and it hosts any OCI image without an SDK, so ADK, LangChain or MCP servers run unchanged.
It is the wrong tool in three cases. A handful of long-running agents is simpler as ordinary Pods, or Agent Sandbox if you want Pod semantics with stronger isolation. One-shot code execution that is thrown away after a single run gains nothing from snapshots. And a team without people who run Kubernetes, PostgreSQL and an Envoy data plane will spend its time on those instead of the agents.
For an FDE, the evaluation is short. Measure the customer's busy fraction and its correlation before quoting a density. Run the benchmark harness at the customer's memory size and read the phase breakdown. Decide who calls SuspendActor and whether it is synchronous. Then scope every credential the gateway injects as if the agent could use all of it, because it can.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
Agent Substrate (ate.dev) is an Apache-2.0 runtime from Google, donated to the CNCF at Sandbox status in November 2026, that runs sandboxed agents on Kubernetes. It keeps a record for every agent (an actor) in its own control plane, snapshots an actor's memory and files when it is suspended, and restores it onto any free worker Pod from a pre-started pool when a request arrives. Every actor runs inside gVisor or a Cloud Hypervisor microVM; plain containers are not supported.
Not at v1.0.1. The architecture documents suspension as an explicit SuspendActor call, and Google's GKE documentation describes the application suspending idle agents. A worker-side path for idleness signals exists in the code, but no shipped binary uses it at commit dc747ef8. Resume is automatic: a request routed to a suspended actor wakes it.
The README says resume takes less than 500 ms and that it handles more than 500 activations per second; GKE's documentation says under a second. The project's stated targets are 100 ms at the 95th percentile and 1,000 wakeups per second per cluster. The repository ships a benchmark harness but no published results, so measure your own memory size: a restore moves the actor's snapshot, and bigger working sets restore slower.
The API is stable as of 1.0, and Google offers production support on GKE by allowlist; open use on GKE is for non-production workloads. The project's own security table lists scrubbing between actors, per-actor snapshot encryption, quotas and actor-to-actor authorization as in progress or future, and worker Pods still run as root with SYS_ADMIN. Read that table against your threat model before putting hostile tenants on shared nodes.
The agent sends a placeholder header and the egress gateway, atenet-egress, replaces it with the real credential on the way out, so the key never enters the sandbox or its snapshots. Rules match on hostname and port only. A prompt-injected agent cannot read the key, but it can make any request to an allowed host with it, so the token itself still needs narrow scopes.
Agent Sandbox gives each sandbox standard Kubernetes Pod semantics. Agent Substrate builds on its sandboxing and snapshot capabilities but moves the per-agent lifecycle into its own control plane backed by PostgreSQL, so millions of mostly idle agents do not become millions of Pods and resumes skip the Kubernetes scheduler. Google says the two run side by side.
Discussion (4)
The async suspend default deserves a line in every design review. SuspendActor returns once the checkpoint is on local disk and uploads in the background, so a node that dies in that window takes the agent's latest state with it. For a coding agent that is a lost turn. For an agent that just told a customer it filed their refund, it is a disagreement between what the agent remembers and what happened.
Reader question: why not just scale Pods to zero with an autoscaler and skip all of this?
Scale-to-zero throws the process away, so the agent restarts and has to rebuild its state, and a new Pod pays for scheduling and image pull. Substrate keeps the process, its memory and its files, and restores them onto a Pod that is already running. If your agent keeps all state in a database and starts in a second, scale-to-zero is simpler and you should use it.
The snapshot is the agent's memory, so it holds whatever the agent read: customer records, code, the contents of a file a user uploaded. Credential injection keeps API keys out of it. It does nothing for data. Per-actor snapshot encryption is still listed as future work, so ask who can read the bucket.
