GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo

Agent Substrate Review: Running Millions of Sandboxed AI Agents on Kubernetes, and What the Density Claim Depends On

Agent Substrate suspends idle AI agents to snapshots and resumes them onto warm gVisor or microVM workers on Kubernetes. We read the v1.0.1 code, checked the numbers, sized a pool with a runnable script, and found what the headline omits: your app decides when to suspend, and egress rules scope credentials by host.

BY LUKAS HOFFMANN · FDEINTERVIEWS EDITORIAL · UPDATED NOVEMBER 20, 2026 · 10 MIN READ

PRACTICE THIS:Agent sandboxing, the concept ·Design the sandbox for agent-written code ·Self-hosted Claude Managed Agents ·System design questions

Agent Substrate is an open-source runtime from Google that runs sandboxed AI agents on Kubernetes at far higher density than one Pod per agent. It does this by noticing that agents spend most of their time waiting. A suspended agent becomes a snapshot of its memory and files; when a request arrives, the runtime restores it in under a second onto a worker Pod that was already running, possibly a different one from last time. Every agent runs inside gVisor or a Cloud Hypervisor microVM, holds no credentials, and reaches the network only through gateways. We read the repository at v1.0.1 (commit dc747ef8, 20 November 2026), its security docs and threat model, and Google's announcements, and ran our own numbers. It is a serious piece of infrastructure. Two things matter before you plan around it: the density you get depends on your traffic, and the decision to suspend is yours.

What it is

The project started on GitHub in May 2026, reached 1.0 on 5 November 2026, and on the same day Google donated it, with gVisor, to the CNCF at Sandbox status. Google's Cloud blog announced it on GKE in September 2026: open to all GKE customers for non-production workloads, with production support by allowlist. GKE's documentation requires Standard clusters on version 1.37 or later; Autopilot is not supported. The repository had 7,985 GitHub stars when we checked.

The core idea is to separate the actor (one agent instance and its state) from the worker (a warm sandbox Pod in a pool). Kubernetes still owns the slow, infrequent work: nodes, Pods, autoscaling. A small control plane owns the fast work: which actor is where, resume, suspend, routing. The reason is spelled out in the architecture doc. The Kubernetes API server "is not designed to handle millions of resources", and getting a Pod running in a couple of seconds is fine for a long-lived service but too slow for an agent that runs for milliseconds to seconds and then waits again.

One request to a suspended agentYour application: Authenticates users, names the target actor, and decides when to suspend it.. atenet-router: Envoy with ext_proc. Holds a request up to 5 s when no worker is free.. resume + locate. SuspendActor. ate-api-server: Actor and worker state in PostgreSQL. Claims a warm worker; authorizes every call per atespace.. Object storage: GCS or S3. Durable snapshots; sandboxes never touch it.. snapshot. restore here. atelet, one per node: Fetches the snapshot, keeps a local copy for fast re-resume, drives the worker.. RestoreWorkload. Sandbox: gVisor or a Cloud Hypervisor microVM. The agent: memory, files, no secrets.. atunnel: The router reaches it over mTLS on 8443; it forwards to the agent.. every outbound call. atenet-egress: Allows by hostname and port, and swaps placeholder headers for credentials on the way out.. Suspend is a call your application makes. At this commit nothing in Substrate decides that an agent is idle. One request to a suspended agent Agent Substrate at commit dc747ef8, from its architecture and networking docs. Kubernetes stays out of this path. Your application Authenticates users, names the target actor, and decides when to suspend it. atenet-router Envoy with ext_proc. Holds a request up to 5 s when no worker is free. resume + locate SuspendActor ate-api-server Actor and worker state in PostgreSQL. Claims a warm worker; authorizes every call per atespace. Object storage GCS or S3. Durable snapshots; sandboxes never touch it. snapshot restore here atelet, one per node Fetches the snapshot, keeps a local copy for fast re-resume, drives the worker. WORKER POD: WARM, POOLED RestoreWorkload Sandbox gVisor or a Cloud Hypervisor microVM. The agent: memory, files, no secrets. atunnel The router reaches it over mTLS on 8443; it forwards to the agent. every outbound call atenet-egress Allows by hostname and port, and swaps placeholder headers for credentials on the way out. Suspend is a call your application makes. At this commit nothing in Substrate decides that an agent is idle.

The figure follows one request to a suspended agent, which is the case that matters. Your application sends it to atenet-router with a header naming the target actor. The router asks ate-api-server, which keeps actor and worker records in PostgreSQL, to resume the actor; the API server claims a warm worker, and the node agent atelet restores the actor's latest snapshot into the sandbox on that worker. Only then does the router open an mTLS tunnel to the worker and forward the request. No Pod is created and the Kubernetes scheduler is never asked.

PieceWhat it isWhere it lives
WorkerPoolA pool of warm worker Pods for one sandbox classKubernetes custom resource
ActorTemplateImage, memory and settings for one kind of agent; used to build a "golden snapshot"Substrate API, not Kubernetes
ActorOne agent instance: state (RUNNING, SUSPENDED), current worker, snapshotsPostgreSQL, through ate-api-server
AtespaceThe grouping unit for policy, like a namespaceSubstrate API

Snapshots come in three cumulative levels, set per template: VOLUMES (durable directories only), ROOTFS (plus files written to the container's filesystem) and MEMORY (plus the running processes and RAM, the default). A MEMORY snapshot is tied to the sandbox runtime version it was taken on. If no worker offering that version has room, the resume falls back to ROOTFS and cold-boots, so the agent comes back with its files but without its in-memory state. If your agent keeps a conversation in a Python variable, that fallback is a lost conversation; keep anything you cannot afford to lose on a durable volume.

Suspend is your call

The hero image says idle agents suspend and the quickstart says that "when the actor becomes idle, Substrate checkpoints" it. The architecture doc is more precise: suspension is "triggered by an explicit SuspendActor call", and GKE's documentation talks about "letting your application suspend idle agents". The code agrees with the precise version. A package that lets a worker ask the control plane to suspend its actor exists at internal/ateworker/suspend, with a comment that "an idleness signal is a fact about the past, and it decays", but at commit dc747ef8 nothing imports it.

Resume, on the other hand, is automatic: any request routed to a suspended actor wakes it. So the orchestration layer you build on top (or a framework that integrates Substrate, such as kagent or Google's Agent Executor) owns the policy. That is the right place for it, because only your harness knows whether the agent is waiting for a model call that returns in three seconds or for a human who has gone to lunch. It also means a design review should ask who calls SuspendActor, on what signal, and what happens when that caller is down. Agents that are never suspended do not save anything.

The default suspend is also asynchronous. SuspendActor returns once the checkpoint is on the node's disk; atelet uploads it to object storage in the background. The docs say plainly that asynchronous suspends "risk state loss if the node fails before background upload completes". A synchronous mode exists for agents whose state you cannot lose.

What the numbers are

NumberSourceWhat it is
Resume in less than 500 msREADME, architecture doc, Google blogsProduct claim
Resume "in under a second"GKE documentationProduct claim
More than 500 activations per secondREADME, Google Cloud blogProduct claim
10x density over standard containersREADME, Google Cloud blogProduct claim
10x to 30x oversubscriptionREADME, about the demo videoDemo: about 250 actors on 8 worker Pods
100 ms at p95, 1 billion agents, 1,000 wakeups per second per clusterArchitecture doc, its metrics sectionTargets

None of the published pages says what workload or hardware produced the claims. The repository does ship the means to find out: a Locust-based benchmark harness, a synthetic workload called glutton that holds a set amount of RAM and re-dirties part of it each cycle, and a runbook that breaks every suspend and resume into phases (download, restore, wake-up probe). Run it with your own memory size. A restore has to move or demand-page the snapshot, so an agent with a 4 GB working set will not resume like a counter demo.

The density claim needs a different kind of care. How many agents share a slot is not a property of the runtime; it is a property of how often your agents are busy and whether they are busy at the same time. A short script makes the point. It asks how many concurrently running slots a pool needs so that demand exceeds it no more than 0.1% of the time, assuming each agent is independently busy some fraction of the time, with an optional burst where everyone gets busier at once (a Monday 9 a.m.).

python
from math import lgamma, log, exp

def tail(n, p, s):
    """P(more than s of n actors are busy at once), each busy with probability p."""
    total = 0.0
    for k in range(s + 1, n + 1):
        total += exp(lgamma(n + 1) - lgamma(k + 1) - lgamma(n - k + 1)
                     + k * log(p) + (n - k) * log(1 - p))
    return total

def slots_needed(n, mix, risk=0.001):
    """Smallest slot count s with P(demand > s) <= risk.
    mix is a list of (share of time, busy probability) regimes."""
    for s in range(n + 1):
        if sum(w * tail(n, p, s) for w, p in mix) <= risk:
            return s

cases = [
    ("250 agents, busy 5%",                     250,    [(1.0, 0.05)]),
    ("250 agents, busy 10%",                    250,    [(1.0, 0.10)]),
    ("250 agents, 5% + a 30% burst 10% of the time", 250, [(0.9, 0.05), (0.1, 0.30)]),
    ("10,000 agents, busy 5%",                  10_000, [(1.0, 0.05)]),
]
print(f"{'case':<48}{'mean busy':>10}{'slots':>7}{'agents/slot':>13}")
for name, n, mix in cases:
    mean = n * sum(w * p for w, p in mix)
    s = slots_needed(n, mix)
    print(f"{name:<48}{mean:>10.1f}{s:>7}{n / s:>13.1f}")
Code
case                                             mean busy  slots  agents/slot
250 agents, busy 5%                                   12.5     24         10.4
250 agents, busy 10%                                  25.0     41          6.1
250 agents, 5% + a 30% burst 10% of the time          18.8     92          2.7
10,000 agents, busy 5%                               500.0    569         17.6

We checked the two 250-agent edge cases with a Monte Carlo run of 40,000 draws each: 24 slots exceed demand 0.08% of the time and 23 slots 0.2%; 92 and 91 slots land at 0.075% and 0.10%.

Density is a property of your traffic250 agents, each busy 5%: 24 slots, 10.4 agents per slot. 250 agents, each busy 10%: 41 slots, 6.1 agents per slot. 250 agents, 5%, with a 30% burst 10% of the time: 92 slots, 2.7 agents per slot. 10,000 agents, each busy 5%: 569 slots, 17.6 agents per slot. Same runtime, a 2.7x to 17.6x spread. Correlated bursts erase density; a larger pool recovers it. Density is a property of your traffic Agents per running slot, sizing the pool so demand exceeds it at most 0.1% of the time. Hypothetical traffic, computed by the script below. 250 agents, each busy 5% 10.4x 24 slots 250 agents, each busy 10% 6.1x 41 slots 250 agents, 5%, with a 30% burst 10% of the time 2.7x 92 slots 10,000 agents, each busy 5% 17.6x 569 slots Same runtime, a 2.7x to 17.6x spread. Correlated bursts erase density; a larger pool recovers it.

Read the chart from top to bottom. Agents busy 5% of the time, independently, give you the advertised 10x. Double their busy time and density falls to 6x. Add a correlated burst and it falls to 2.7x, because the pool has to be sized for the burst even though the average barely moved. A larger pool recovers density, since independent demand smooths out as it adds up: 10,000 agents busy 5% need only 14% more slots than their average. So the first thing to measure for a customer is not resume latency. It is the busy fraction per agent and how correlated it is across agents, which their own logs usually answer.

When the pool does run short, the router parks the request and retries the resume with backoff for up to five seconds by default, then returns 503. Parking absorbs a moment when every worker is taken. It does not fix a pool sized for the average.

The security model, read carefully

The design treats every actor as untrusted and every pair of actors as mutually hostile, and most of the hard parts are done. Every actor runs in gVisor or a microVM. Each gets its own network namespace with no path to other actors, the node, the Substrate API, the Kubernetes API or the database. Ingress arrives only through the authenticated router; egress leaves only through atenet-egress, which fails closed. Every active actor gets a one-hour SPIFFE X.509 certificate, and the gateway can mint per-actor JWTs for upstream services that trust Substrate's issuer. Sandboxes never touch snapshot storage; on GCS, atelet moves snapshots with credentials scoped to one actor.

The project's own security page marks every principle as met, in progress, future or out of scope, which is more candid than most. These are the rows that are not met yet:

PrincipleStatus at v1.0.1
Scrub local state between actors on a reused workerIn progress
Worker Pod privilegesIn progress: still root with SYS_ADMIN and NET_ADMIN, no user namespace
Contain a compromised node to its own actorsIn progress
Per-actor snapshot encryption; re-check authorization on restoreFuture
Network policy and quotas between atespacesIn progress
Actor-to-actor authorization; rate limits and quotasFuture
Taint snapshots of a compromised actor so they cannot resumeFuture

None of these is unusual for a project that hit 1.0 two weeks ago, and the sandbox is still the primary boundary. They do decide whether you put two of a customer's tenants on the same nodes this quarter.

The credential design deserves its own look, because it is the part people quote. An actor never holds an API key. It sends a request with a placeholder header, the egress gateway checks the request against the actor's EgressPolicy, and when the deciding rule says so, the gateway swaps in the real credential from a provider (Kubernetes Secrets or Google Secret Manager). HTTPS works because the gateway terminates the actor's TLS for the hostnames a rule lists, so the actor has to trust the gateway's CA. The key never enters the sandbox, so it never lands in a snapshot.

The secret stays out. Its authority does not.One https rule: hostnames: [api.github.com], replaceHeaders: Authorization from a stored token. No field for method or path.. GET /repos/acme/app/issues: token injected. The task the agent was given. DELETE /repos/acme/app: token injected. Same host and port, so the same rule decides. POST https://paste.example.net: denied. No rule matches the host. A prompt-injected agent cannot read or leak the token, but it can still spend it on that host. Scope the token itself, or put your own enforcement point in front. The secret stays out. Its authority does not. How an atenet-egress rule decides, from internal/egresspolicy at dc747ef8. The requests are illustrative. One https rule hostnames: [api.github.com], replaceHeaders: Authorization from a stored token. No field for method or path. GET /repos/acme/app/issues The task the agent was given token injected DELETE /repos/acme/app Same host and port, so the same rule decides token injected POST https://paste.example.net No rule matches the host denied A prompt-injected agent cannot read or leak the token, but it can still spend it on that host. Scope the token itself, or put your own enforcement point in front.

The figure shows what that protects and what it does not. We read the rule evaluation in internal/egresspolicy: a request is decided on its hostname and port, and the rule schema has no field for HTTP method or path. A prompt-injected agent cannot leak the GitHub token, but it can call any GitHub endpoint the token allows, with the token attached for it. The README's phrasing, that "a prompt-injected or compromised agent has nothing to exfiltrate", is true of the bytes and not of what they authorize. Two fixes work today: issue the narrowest token the task needs (a read-only, single-repository token injected on a read path), or put your own policy enforcement point in front, which the extensibility doc supports for egress, although the installer does not have a bring-your-own mode yet.

The project also lists what it leaves to you. Your application authenticates end users and decides which actors they may reach, keeps the registry of agents, and records lineage (which agent created which sub-agent).

When it fits, and when it does not

Agent Substrate fits a platform that hosts many stateful, per-user agents that are idle most of the time and run code nobody reviewed: coding agents, per-user assistants, tool servers. Its demos run Claude Code and an Alpine shell sandbox, and it hosts any OCI image without an SDK, so ADK, LangChain or MCP servers run unchanged.

It is the wrong tool in three cases. A handful of long-running agents is simpler as ordinary Pods, or Agent Sandbox if you want Pod semantics with stronger isolation. One-shot code execution that is thrown away after a single run gains nothing from snapshots. And a team without people who run Kubernetes, PostgreSQL and an Envoy data plane will spend its time on those instead of the agents.

For an FDE, the evaluation is short. Measure the customer's busy fraction and its correlation before quoting a density. Run the benchmark harness at the customer's memory size and read the phase breakdown. Decide who calls SuspendActor and whether it is synchronous. Then scope every credential the gateway injects as if the agent could use all of it, because it can.

PRACTICE THIS

Turn it into offers. Work the real questions and concepts this maps to:

FAQ

Agent Substrate (ate.dev) is an Apache-2.0 runtime from Google, donated to the CNCF at Sandbox status in November 2026, that runs sandboxed agents on Kubernetes. It keeps a record for every agent (an actor) in its own control plane, snapshots an actor's memory and files when it is suspended, and restores it onto any free worker Pod from a pre-started pool when a request arrives. Every actor runs inside gVisor or a Cloud Hypervisor microVM; plain containers are not supported.

Discussion (4)

Arjun MehtaEditor

The async suspend default deserves a line in every design review. SuspendActor returns once the checkpoint is on local disk and uploads in the background, so a node that dies in that window takes the agent's latest state with it. For a coding agent that is a lost turn. For an agent that just told a customer it filed their refund, it is a disagreement between what the agent remembers and what happened.

Mei LinEditor

Reader question: why not just scale Pods to zero with an autoscaler and skip all of this?

Lukas HoffmannEditor

Scale-to-zero throws the process away, so the agent restarts and has to rebuild its state, and a new Pod pays for scheduling and image pull. Substrate keeps the process, its memory and its files, and restores them onto a Pod that is already running. If your agent keeps all state in a database and starts in a second, scale-to-zero is simpler and you should use it.

Hannah BryantEditor

The snapshot is the agent's memory, so it holds whatever the agent read: customer records, code, the contents of a file a user uploaded. Credential injection keeps API keys out of it. It does nothing for data. Per-actor snapshot encryption is still listed as future work, so ask who can read the bucket.