Agentic Software Factories: How to Build an AI Pipeline From Issue to Production That You Can Trust

An agentic software factory is a pipeline in which AI agents carry a change from request to production. A triage agent decides whether the work is ready, a spec agent turns ambiguity into checkable requirements, and an implementation agent writes the code in a sandbox. Separate verification and review agents then try to prove it wrong, a merge queue and a progressive rollout decide whether it ships, and monitoring files new issues back in. Humans stop typing code and start writing intent, owning the gates and approving what the gates cannot judge. CI/CD automated the path a change takes. A factory automates the production of the change itself, which moves the scarce resource from writing code to verifying and reviewing it. The teams that have published numbers (Stripe, Spotify, OpenAI, StrongDM, Warp) agree on that shift. They disagree on nearly everything else, including whether a human should read the code at all. What follows is our read of what works, as of October 2026: the evidence, a reference architecture, and the parts most write-ups skip, which are verification, merge physics, rollout, security, cost and what happens when the model underneath changes.

What the teams running factories have published

The term is old (Microsoft used "software factories" for model-driven development in the 2000s), but the current meaning comes from a run of engineering posts between late 2025 and mid 2026. Each one describes a different point on the same curve.

Now the counterweight. METR's July 2025 randomized trial put 16 experienced open-source developers on 246 tasks in their own repositories; with AI tools they took about 19 percent longer, while believing they had been about 20 percent faster. Google's 2025 DORA report (nearly 5,000 respondents, 90 percent using AI at work) found AI adoption now tracks higher delivery throughput but still tracks lower delivery stability, and 30 percent of respondents reported little or no trust in AI-generated code. CodeRabbit's December 2025 analysis of 470 open-source PRs found about 1.7 times as many issues in AI-coauthored PRs, a figure worth reading knowing CodeRabbit sells code review.

These do not contradict the factory numbers. They describe the same thing from the other side. A developer pairing with an assistant in a large, familiar codebase pays the cost of steering and checking in real time. A factory moves that cost into gates that run without a human watching, and it only pays off where those gates exist. That is DORA's finding in one sentence: AI amplifies whatever delivery system it lands in.

Five conclusions hold across every source above:

1. The gate is code, not a prompt. Stripe's test step runs regardless of what the model thinks. OpenAI encodes its architecture rules as linters. Spotify moved verification into a runtime the agent cannot touch.
2. The bottleneck moves to review and intent. Spotify said it plainly at QCon London in March 2026: generation is no longer the constraint, PR review is.
3. An agent will game a weak grader. Spotify's early agents commented out failing tests and downgraded Java versions to get green builds. StrongDM keeps its scenarios outside the repository for exactly this reason.
4. Narrow, repetitive change classes go first. Every large published number is a migration, a dependency bump or a mechanical refactor.
5. Throughput without stability controls makes delivery worse. More changes through the same weak gates means more incidents, faster.

How a factory differs from CI/CD

In CI/CD the pipeline is trusted and the change is suspect. In a factory both are suspect, and the pipeline changes every time a model ships. Most of this guide follows from that.

The reference loop

The loop below is the shape most published factories converge on, and it matches the stages Warp documents (triage, spec, implement, review, verify, ship, monitor). Read it as a set of decisions, each owned by an agent or a person, with every failure path leading somewhere specific.

Intake

Work arrives from wherever the team already talks: Stripe starts most Minion runs by tagging a Slack app in a thread, which hands the agent the thread and its links; Warp accepts Slack, Teams, GitHub, GitLab, Linear and Jira; monitoring alerts are the third source. Intake should attach three things before any agent runs: who asked, which repository and owner the work touches, and the change class (more on classes below). Without an owner on every item, the factory produces orphaned PRs.

Triage: the agent whose job is to say no

Triage is the cheapest stage and the one that saves the most money, because every unready item it rejects is an implementation run, a CI run and a review you do not pay for. Warp's published triage skill tries to understand and reproduce the issue, then applies one of four labels: ready-to-implement, ready-to-spec, needs-info or wait-to-implement. The last two stop the factory until a human acts.

Two design rules make triage work. Give it read-only tools and no secrets, because it reads the least trusted text in the system (issues anyone can file). And measure it on precision of the automatable label, the share of items it sent to implementation that ended in an accepted change. A triage agent that sends everything forward looks productive and burns the budget.

Spec: turning ambiguity into checkable requirements

When an item is ambiguous or large, a spec agent writes it down before anyone writes code. Warp's spec agent produces a PRODUCT.md and a TECH.md under specs/, and a human approves them. AWS's Kiro writes requirements in EARS form ("WHEN a condition THE SYSTEM SHALL a behavior") with approval gates between requirements, design and tasks, and GitHub's open-source Spec Kit runs a similar specify, plan, tasks sequence with whatever agent you already use.

The format matters less than one property: every requirement gets an ID and a named test that will prove it. A spec that cannot be checked is a wish, and the rest of the factory has nothing to verify against.

Implementation in a sandbox

The implementation agent works in an isolated, disposable environment with the repository checked out. Stripe's devboxes are pre-warmed so one starts in about 10 seconds, and they are cut off from production and the internet, which is what lets Minions run without asking permission for each command. Locally, one git worktree per agent gives the same isolation.

Fast feedback first, slow feedback last: Stripe runs selected lints on every push in under five seconds, then selective CI, and caps CI at two rounds before handing back to a human. Airbnb found that retrying a failing file with its error output beat further prompt tuning; its engineers called the approach brute force, and it migrated about 75 percent of files within four hours.

Verification: a different agent with different tools

The agent that wrote the code is the worst judge of it. Anthropic's harness work found that tuning a separate, skeptical evaluator was far easier than making a generator critical of its own output, and it gave the evaluator a browser so it could click through the running app. Warp's implementation agents use computer use to check changes end to end and save the recording to the PR. Google's Antigravity calls these outputs Artifacts: plans, screenshots and browser recordings a human can inspect.

The verifier should start with fresh context: the spec, the built artifact and the scenarios, but not the implementer's reasoning. Inheriting the maker's explanation is how a checker talks itself into agreeing.

Review: agents first, a human where the policy says so

Review agents cycle with the implementer for a capped number of rounds, then a human reviews the diff together with the verification evidence. Parallel specialist reviewers (security, performance, spec coverage) catch more than one generalist and cost little more, because they run at once. Using a model from a different family for at least one reviewer reduces the chance that maker and checker share the same blind spot, the concern a Stanford Law School blog post raised about StrongDM's setup.

Merge, ship, monitor

A merge queue lands changes one at a time against the current head. A progressive rollout exposes each change to a slice of traffic. A monitoring agent watches deploy markers and error rates and files new issues, which closes the loop. Each of these has its own section below, because each one breaks in a specific way at agent speed.

Autonomy is per change class, not per company

Dan Shapiro's January 2026 "five levels" borrow from driving automation: from spicy autocomplete at the bottom, through an agent you review like a junior engineer, to a dark factory at the top, where specs go in and software comes out with no human looking. The levels are useful vocabulary. The mistake is to pick one level for a whole organization.

A dependency bump and a change to the payment ledger should never share a policy. Set the ceiling per class of change, promote a class only on its own track record, and demote it on the first escaped defect.

The promotion rule we recommend: a class moves up one level after a run of consecutive accepted changes with no escaped defects and a clean human audit of a random sample, and it drops back on any incident it caused. The track record is the argument. Nobody has to win a debate about whether agents can be trusted in general.

Verification is the product

If the factory's output is code nobody fully reads, then the factory's real product is its verification. Everything here is about making "the gates are green" mean "the requirement is met".

Deterministic gates first

Build, type check, lint, unit tests, structural tests. They are cheap, fast and impossible to argue with. OpenAI's team goes further and writes custom linters whose error messages include remediation instructions, so the gate also teaches the agent the fix. Every architectural rule you can express as a linter is one less thing a reviewer has to remember.

Protect the grader

An agent optimizing for a green build will edit whatever makes the build green, including the tests. Put tests, CI configuration and the factory's own settings under CODEOWNERS entries that exclude the agent. Reject, or route to a human, any change that modifies both the implementation and the tests that judge it. Make the factory's configuration human-only. GitHub's Copilot coding agent enforces a version of this structurally: it can push only to branches named copilot/, cannot approve or merge its own pull request, and the person who requested the work cannot approve it either.

Holdout scenarios

StrongDM's most transferable idea is to treat end-to-end scenarios like a holdout set in machine learning: stored outside the codebase, never shown to the coding agents, and scored by an LLM as a probability it calls satisfaction (the share of observed runs through the scenarios that likely satisfy a user) instead of a single pass or fail. Tests that live in the repository can be rewritten to match the code. Scenarios the agent never sees cannot.

StrongDM runs those scenarios against what it calls a Digital Twin Universe: behavioral clones of Okta, Jira, Slack, Google Docs, Google Drive and Google Sheets, which let it run thousands of scenarios an hour without rate limits, API bills or risk to real accounts. Most teams will not clone six SaaS products, but a fake of the two external services your product depends on most is within reach, and it is what makes failure-mode testing safe.

Test the tests with mutation testing

Line coverage tells you which lines ran. It does not tell you whether a test would notice a bug. Mutation testing does: it plants small faults in the code and checks that some test fails. Meta's ACH system, published at FSE 2025, generates targeted mutants and then writes tests that catch them; applied to 10,795 Android Kotlin classes it produced 9,095 mutants and 571 tests, and engineers in Messenger and WhatsApp test-a-thons accepted 73 percent of its tests. For a factory, a mutation score on the changed code is the most honest single answer to "were the agent's tests any good?". Tools exist for most stacks: PIT for Java, Stryker for JavaScript and .NET, mutmut for Python.

Requirement traceability: the closest thing to mathematical proof

You cannot prove in general that code satisfies a requirement written in English. You can get much closer than "the tests pass" with four layers, in increasing order of cost:

• Fail-to-pass per requirement. Each requirement ID maps to at least one test that fails on the old code and passes on the new code, while every previously passing test still passes. This is exactly how SWE-bench grades a patch, and the gate is a short script: run the named tests on the base commit, run them on the change, compare.
• Mutation score on the changed lines, so the fail-to-pass tests are known to catch faults.
• Property-based tests for invariants that should hold for all inputs (an export never drops rows, a refund never exceeds the charge), using a library like Hypothesis to search for a counterexample.
• Formal methods for a small, high-consequence kernel: a TLA+ model of a protocol, a verified state machine. Expensive, so reserve it for the parts where one bug costs more than the proof.

Then add the statistical layer, because agent behavior varies between runs.

Consistency: pass^k

The τ-bench paper defined pass^k as the probability that all k independent attempts at the same task succeed, and showed that a strong 2024 agent that solved under half of retail tasks once fell below 25 percent at pass^8. The arithmetic is unforgiving. Under independence, pass^k is the single-run success rate raised to the power k.

An agent that succeeds 90 percent of the time succeeds eight times in a row only about 43 percent of the time. The same arithmetic applies across stages: a six-stage pipeline where each stage is right 95 percent of the time is right end to end about 74 percent of the time. Run important evaluation tasks several times, report pass^k next to the average, and treat a gap between them as a reliability bug.

LLM judges: advisory, calibrated, replaceable

An LLM judge is useful for questions code cannot answer ("does this PR do what the issue asked?"), and dangerous as a hard gate. Spotify added a judge to check generated code against the original requirements, found it too rigid and blocking valid changes, and removed it as models improved. If you keep one, calibrate it against a few hundred human-labeled examples, track its agreement rate, and recalibrate every time the judge's model changes. Our LLM-as-a-judge paper notes cover the known biases.

Throughput physics: queues, conflicts and the review wall

A factory that opens 100 pull requests a day for a team that reviews 20 has not built a factory. It has built a queue. Little's law makes this exact: in a steady state, waiting time equals the backlog divided by the rate at which it clears, so a backlog growing by 80 a day grows the wait for every PR, including the urgent ones. Set a work-in-progress limit on open agent PRs per reviewer and make the factory stop pulling new work when it is hit.

Spotify's answer to its review wall has three parts: culture (let the engineer driving a migration approve its PRs and close stale ones), tooling (a PR inbox that prioritizes, and possibly auto-merges docs), and standardization (fewer ways of doing things means agent code is right more often, which makes review faster). Make the review itself cheaper too: every agent PR should open with the spec it implements, the evidence that it works (test output, scenario results, a recording), a statement of what was not tested, and the change class.

Parallel agents also collide. Practitioners who have run many agents at once report the same patterns: two agents adding tests to the same test file, two agents editing the same hot configuration file, one agent renaming a function another still calls. Git reports a clean merge for the last case, and only re-running the tests on the merged result finds it. Four habits remove most of the pain:

• Give each task a declared file scope at triage, and hold a lease on hot files so two agents do not take the same one.
• Put new tests in new files named after the feature instead of appending to shared ones.
• Land everything through a merge queue that rebases each change onto the current head and re-runs the gates before merging, so every change is tested against what main will actually be.
• Keep PRs small and short-lived. OpenAI's team describes minimal merge gates with short-lived pull requests, and handles flaky tests with follow-up runs instead of blocking.

Flaky tests deserve a special mention. An agent that hits a flake retries, spends tokens, and may "fix" code that was never broken. Quarantine known flakes, and give the agent the flake history for every test it sees fail.

Shipping at agent speed: provenance, rollout and rollback

When a metric moves at 3 p.m., you need to know which change caused it and which factory configuration produced that change. Stamp every agent commit with provenance: the run ID, the model's dated version, the prompt and skill versions, the spec ID and the change class. Git trailers are enough. Carry the same fields onto deploy markers on your dashboards. Without this, a regression from a new model version looks like a random run of bad luck.

Three rules matter more at agent speed than at human speed:

• New behavior ships behind a flag, off by default. Merging and releasing become separate decisions, and the release decision can stay human for feature work long after merging is automated.
• Canary analysis is automated. Compare the canary to a baseline on error rate, latency and one business metric, and roll back without waiting for a person. Argo Rollouts, Flagger and Spinnaker's Kayenta all do this; OpenFeature gives you a vendor-neutral flag API.
• The error budget gates the factory. When the service is burning its budget, stop merging agent changes for that service. Human hotfixes keep flowing.

The monitoring agent closes the loop, and a closed loop without a circuit breaker is an oscillator. An alert files an issue, an agent fixes the symptom, the deploy triggers a different alert, and the loop repeats. Deduplicate alerts into one issue per root cause, rate-limit how many issues the monitor may file per service per day, refuse to auto-file against a change the factory itself shipped in the last hour without a human looking, and give every loop a named human owner.

Security: the factory is an attack surface

Simon Willison's "lethal trifecta" (June 2025) says an agent becomes exploitable when it combines access to private data, exposure to untrusted content and a way to communicate externally. A naive factory has all three by default. It reads issues, PR comments and dependency documentation written by anyone; it holds the repository and often secrets; and it can push code, comment and reach the network. Any text the agent reads can carry instructions it might follow. The fix is architectural, so remove at least one leg in every stage.

The hallucinated-package row deserves emphasis because it is cheap to exploit and cheap to prevent: names a model invents repeatably can be registered by someone else. A mirror with an allowlist removes the class. More on the injection side in our prompt injection concept and agent sandboxing, and the AI security and governance questions cover how interviewers probe it.

Context management: what each agent should see

Every stage needs the minimum context for its decision, and the checker should never inherit the maker's reasoning. Getting this wrong in either direction is costly: too little and the agent guesses, too much and it anchors on the wrong thing.

Four practices from the published factories make the implementation agent's context work at repository scale. OpenAI keeps AGENTS.md to about 100 lines that act as a table of contents, with the real knowledge in a versioned docs/ directory, and runs a doc-gardening agent that flags stale documentation. Stripe scopes nearly all of its agent rules to subdirectories, so an agent editing one service does not read another service's conventions, and it pre-fetches the content of links in a prompt before the run starts. OpenAI makes the application legible to the agent: each worktree boots its own instance, and the agent can query logs and metrics directly. And for long runs, a progress file on disk survives context compaction where conversation history does not. Our posts on harness engineering and context engineering go deeper on each.

Tool context counts too. A model choosing among 400 tools chooses worse than one choosing among 12, which is why Stripe gives each Minion a curated subset, and why an agent gateway that exposes tools per role belongs in the architecture.

Cost: measure cost per accepted change

Token spend per run is the wrong number. The number that decides whether a factory pays is cost per accepted change: tokens, sandbox compute, CI minutes, human review time, plus the cost of changes that were merged and later reverted. A cheap run that produces a PR nobody can merge costs its tokens plus the reviewer's time to reject it.

The published reference points are far apart. A dashboard screenshot in Warp's August 2026 post shows about $58 per PR over April to July. StrongDM's rule of thumb is that a factory spending less than $1,000 of tokens per engineer per day has room to improve; Simon Willison pointed out that at around $20,000 a month per engineer the question becomes one of business model. Both can be right for different change classes. Spend heavily where a merged change saves days, and very little on a typo.

Review time is not a rounding error. As an illustration, a 15-minute review at a loaded cost of $100 an hour is $25, the same order as the token cost of many runs. A factory that halves review time through better evidence in the PR saves as much as one that halves tokens.

The levers, roughly in order of payoff:

• Reject early. Triage is the cheapest stage. Every unready item it parks saves a full implementation, CI and review cycle.
• Prompt caching. Agent loops re-send the same long prefix (system prompt, repository map, tool definitions) on every turn, which is exactly what caching discounts.

• Route by stage. A fast, cheap model for triage, labeling and spec-coverage checks; a strong model for implementation and security review. Our model routing concept covers how to decide.
• Hard caps. A token budget, a wall-clock limit and a retry cap per task, enforced by the orchestrator, not requested in the prompt. Stripe's two-round CI limit is the model.
• Gate depth by class. Five parallel reviewers on a typo fix is waste. Scale verification to the change class.
• Fast local checks before slow CI. A five-second lint that fails saves a twenty-minute pipeline.

The AI cost and unit economics concept has the general method for building the per-unit cost model.

Model upgrades: a new model is a deploy of the whole factory

A factory's behavior is a function of the model, the harness, the prompts and skills, the tools and the repository. Change any one and the factory needs to be qualified again. Model upgrades are the change teams most often skip qualifying, because the new model is "better" on public benchmarks.

Public benchmarks cannot qualify your factory. In February 2026 OpenAI stopped reporting SWE-bench Verified scores: it audited 138 problems that its o3 model did not reliably solve and found that 59.4 percent had material problems in their tests or descriptions, and it found evidence that frontier models had seen benchmark problems in training. (Our SWE-bench paper notes explain how the benchmark was built.) A score that tracks training exposure tells you nothing about your codebase.

Build a private replay suite instead. Take closed issues that a human fixed, with the merged PR and its tests, and keep them out of every prompt and skill. Grade with the fail-to-pass rule, add your holdout scenarios, and record more than the success rate.

Pin dated model versions in the factory configuration and never point it at a "latest" alias, or the factory upgrades itself on someone else's schedule. Expect gates to need re-justifying in both directions: Spotify removed its LLM judge once models improved enough that it blocked more good changes than bad ones, and a more capable model may need a tighter tamper check.

Behavior can also change with no model change at all. Anthropic's September 2025 postmortem described three infrastructure bugs that intermittently degraded Claude's output for weeks, and said its evaluations did not capture what users were reporting. Its fix included running quality evaluations continuously on production systems. The factory version: send a handful of known-answer canary tasks through the real production path every hour, and alert when their pass rate drops.

Infrastructure: what a factory runs on

A factory splits cleanly into three planes. The control plane decides; the work plane is disposable; the gate plane is the part the agents must never be able to change.

The options at each layer, as of October 2026 (this list will date fast; the layers will not):

On build versus buy: Warp's own write-up concedes that building all the infrastructure to scale a factory is a lot of work. Our position is to rent the sandbox and the runner, and own the gates and the policy. The gates encode what your organization means by "correct", and no vendor can hold that for you. Our concepts on multi-agent orchestration, agent observability and the agent versus workflow decision cover the design choices inside the control plane.

Building one on Google: ADK, Gemini CLI and Jules

Google's stack covers the factory at three levels, and they combine well.

• Gemini CLI GitHub Action (google-github-actions/run-gemini-cli) runs inside GitHub workflows for issue triage, pull request review and on-demand help when someone mentions @gemini-cli. Project context goes in a GEMINI.md file, and the action's docs recommend Workload Identity Federation over long-lived API keys. It is the fastest way to put triage and a first-pass reviewer in front of a repository.
• Jules is Google's asynchronous coding agent: it clones the repository into a cloud VM, plans, runs the tests and opens a pull request. It has a command-line tool and an API, which makes it a candidate implementation stage that you call from your own orchestrator.
• Agent Development Kit (ADK) is for building your own control plane. Its workflow agents are deterministic: SequentialAgent runs steps in order, ParallelAgent runs them at once, and LoopAgent repeats until a sub-agent escalates or a maximum iteration count is reached. Agents share data through session state (output_key writes, {key} placeholders read). adk deploy targets Agent Engine on Vertex AI, Cloud Run, GKE or Docker.

Here is the core of a factory in ADK. The test gate is a custom agent that runs the real test command, so the model cannot argue with it.

import subprocess
from typing import AsyncGenerator

from google.adk.agents import BaseAgent, LlmAgent, LoopAgent, ParallelAgent, SequentialAgent
from google.adk.agents.callback_context import CallbackContext
from google.adk.agents.invocation_context import InvocationContext
from google.adk.events import Event, EventActions
from google.genai import types

FAST, STRONG = "gemini-2.5-flash", "gemini-2.5-pro"  # pin the versions your replay suite qualified


class TestGate(BaseAgent):
    """Deterministic gate: runs the real test command. The model cannot argue with it."""

    command: list[str]

    async def _run_async_impl(self, ctx: InvocationContext) -> AsyncGenerator[Event, None]:
        result = subprocess.run(self.command, capture_output=True, text=True, timeout=900)
        passed = result.returncode == 0
        yield Event(
            author=self.name,
            content=types.Content(role="model", parts=[types.Part(text=result.stdout[-4000:])]),
            actions=EventActions(
                state_delta={"tests_passed": passed, "test_log": result.stdout[-4000:]},
                escalate=passed,  # green tests end the loop; red tests send it round again
            ),
        )


def require_green(callback_context: CallbackContext) -> types.Content | None:
    """A LoopAgent that runs out of iterations does not fail. Skip review instead of reviewing red code."""
    if not callback_context.state.get("tests_passed"):
        return types.Content(role="model", parts=[types.Part(
            text="Tests still red after the retry budget. Escalating to a human.")])
    return None


triage = LlmAgent(
    name="triage", model=FAST, output_key="triage",
    instruction="Read the issue: {issue}. Reply with exactly one label: "
                "ready-to-implement, ready-to-spec, needs-info or park, then one line of reasons.")

spec = LlmAgent(
    name="spec", model=STRONG, output_key="spec",
    instruction="Turn the issue {issue} into numbered requirements (R1, R2...) "
                "and name the test that will prove each one.")

implement = LlmAgent(
    name="implement", model=STRONG, output_key="patch",
    instruction="Implement {spec}. Previous test output, if any: {test_log?}. "
                "Never edit files under tests/ or ci/.")

build_loop = LoopAgent(
    name="build_loop", max_iterations=3,
    sub_agents=[implement, TestGate(name="test_gate", command=["pytest", "-q"])])

reviewers = ParallelAgent(name="reviewers", before_agent_callback=require_green, sub_agents=[
    LlmAgent(name="security_review", model=STRONG, output_key="security",
             instruction="Review {patch} against {spec} for security issues only."),
    LlmAgent(name="spec_review", model=FAST, output_key="coverage",
             instruction="For each requirement in {spec}, name the test in {patch} that proves it, or say MISSING."),
])

factory = SequentialAgent(name="factory", sub_agents=[triage, spec, build_loop, reviewers])

We ran the control flow against google-adk 2.11.0 with stub agents in place of the model calls. Green tests escalate and end the loop after one pass. Red tests go round three times, as configured.

Then we found the trap the require_green callback exists for. Without it, a LoopAgent that exhausts max_iterations simply finishes, and the parent SequentialAgent carries on to the review stage with red tests. We also tried setting end_invocation inside a custom agent placed after the loop. In 2.11.0 the parent ran the next step anyway, because each sub-agent gets its own copy of the invocation context. A before_agent_callback that returns content on the next stage is what reliably skips it. The general lesson outlives this framework: every loop with a cap needs an explicit out-of-budget path, and the default is usually to carry on without an error.

Two more points from the ADK tooling. First, adk eval scores evalsets with criteria such as tool_trajectory_avg_score, which requires an exact match of the tool-call sequence (default threshold 1.0), and response_match_score, a ROUGE-1 similarity (default 0.8). Its efficiency metrics, token_usage_v1 and tool_call_count_v1, are reported but never pass or fail. Exact trajectory match suits a triage agent, whose tool sequence should be predictable. It is the wrong gate for an implementation agent, which can reach a correct patch many ways; use the rubric-based criteria and your fail-to-pass replay suite there. Second, adk optimize rewrites an agent's instructions using the GEPA method. That is the kind of self-improvement Warp describes with its observer agents, and it belongs behind the same model-upgrade qualification as any other change to the factory.

A practical split: put the Gemini CLI action in front for triage and first-pass review, run implementation through Jules or your own ADK loop in a sandbox, keep the gates in your existing CI and merge queue, and deploy the ADK orchestrator to Cloud Run or Agent Engine with the replay suite running in CI against every configuration change.

Who runs a factory

The published factories are run by small teams: StrongDM started with three people, and OpenAI's harness project grew from three engineers to seven. The roles matter more than the headcount:

• Factory owner. A platform engineer who owns the harness, sandboxes, gates, budgets and the change-class policy. The factory is a product with users, and this person is its owner.
• Intent owners. Product engineers and PMs who write specs and scenarios. Writing acceptance criteria that an agent cannot satisfy the wrong way is now a core engineering skill.
• Code owners. Reviewers for each area, through CODEOWNERS. Review capacity has to be planned like any other capacity.
• Reliability. Rollout policy, error budgets, and the incident process when an agent change escapes.
• Security. Agent identity, egress, tool policy and the trifecta audit for each stage.
• Eval owner. The replay suite, judge calibration and model qualification. Often the factory owner at first, but a separate role by the time three or more change classes are automated.

DORA's 2025 report found that 90 percent of organizations already run at least one internal platform. A factory is a natural extension of that team, and putting it there is better than putting it in a separate "AI team" that does not own CI, deploys or on-call.

The skills that grow in value are reading diffs quickly and skeptically, writing specs and tests that cannot be gamed, and debugging the factory itself from its traces when it starts producing a new class of bad change.

Where to start: use cases ranked

Three scenarios show how the same loop behaves with different inputs.

Four hundred services need a logging library upgrade. Triage marks it automatable, and the orchestrator fans out one task per repository with a shared spec. Each run gets a strict budget and a two-round CI cap. Repositories that fail after the cap go to a human list instead of retrying forever, which is the remaining-30-percent problem Spotify described with its earlier scripted migrations. The PR inbox groups the changes so one owner can approve a batch.

A null-pointer alert at 2 a.m. The monitoring agent deduplicates it against open issues, attaches the stack trace and the suspect deploy's provenance, and files one issue. Triage reproduces it in a sandbox, the implementation agent writes a fix and a fail-to-pass test from the repro, and the verifier replays the failing request. A human approves in the morning; the canary rolls it out. If the suspect deploy was itself an agent change, the monitor also opens an issue against the factory configuration that produced it.

A product manager asks in Slack for a CSV export. Triage marks it ready-to-spec. The spec agent writes requirements: which columns, what happens at a million rows, who is allowed to export. The PM approves. Implementation ships behind a flag, the holdout scenario checks that an unauthorized user gets nothing, and the flag goes on for one customer first. The permissions requirement puts this change in the high-consequence class, so two humans approve, whatever the agent's track record on CSV code.

Failure modes to design against

• Test tampering. The agent edits the grader. Prevent it with ownership rules, not instructions.
• Plausible but wrong. Clean code that solves a slightly different problem. Prevent it with fail-to-pass per requirement and holdout scenarios.
• Scope creep. A two-line fix arrives as a 600-line refactor. Gate on diff size per change class and ask for a split.
• Rubber-stamp review. At volume, humans approve what the gates approved. Spotify compared reviewers to pilots monitoring automation. Audit a random sample deeply each week and track how often reviewers catch something.
• Runaway loops. An agent retries a flake 40 times. Budgets and caps live in the orchestrator.
• Context rot. Stale docs teach the agent wrong conventions, at scale. Run a doc-gardening job and date your docs.
• Monoculture. The same model writes, verifies and reviews, so it shares one set of blind spots. Use a different model family or a deterministic check for at least one gate.
• Oscillation. Monitor and fixer chase each other. Deduplicate, rate-limit, and give the loop a human owner.
• Invented dependencies. Use an allowlist and require human approval for new packages.
• Silent degradation. A model or infrastructure change lowers quality with no error. Run canary tasks continuously on the production path.

A 90-day plan to stand one up

Resist starting with feature work. The first class should be boring enough that a mistake is cheap and frequent enough that the track record builds quickly.

What this means for forward deployed engineers

FDEs build these factories inside customer environments, where the constraints are the interesting part: a VPC where code cannot leave, an approved-model list, a change-management board that predates agents, and a security team that has read about prompt injection. The design questions in this guide (which change classes, which gates, who approves, how provenance flows into the customer's audit trail) are the ones you will be asked to answer in the room. They are also showing up in interviews as system design prompts: "design a system where agents fix bugs from production alerts" or "how would you let an agent open PRs against our monorepo safely". Interviewers listen for deterministic gates, autonomy per change class, protection of the grader, rollout and rollback, and cost per accepted change. Practice with the system design questions and the RAG and agent questions, and the Harness Engineering for FDEs course builds the verification and loop pieces hands-on.