GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo

CanyonOS explained: deploy, trace and scale multi-agent Python workflows

A practical CanyonOS guide to per-agent replicas, queues, OpenTelemetry and deployment. Learn the runtime constraints and how to test scaling without confusing it with correctness.

BY ARJUN MEHTA · FDEINTERVIEWS EDITORIAL · UPDATED NOVEMBER 15, 2026 · 4 MIN READ

PRACTICE THIS:Design a production agent ·Practise system design ·Study agent observability

CanyonOS deploys Python agents as separately operated services and makes their calls visible across a workflow. Its useful promise is selective capacity: add replicas to the busy research agent while leaving a lightly used planner alone. It supplies deployment machinery, communication and telemetry around your code. Adopt it when a working multi-agent application has become difficult to operate; first establish that the application needs multiple agents at all.

The official repository is licensed under AGPL-3.0. This guide reviews the November 15, 2026 source and focuses on the engineering questions that its quickstart cannot answer for your workload.

What runs around your agents?

CanyonOS uses a Global Controller, per-replica Local Controllers, an HTTP endpoint and Redis coordination. Each replica has an LLM Gateway that records model usage. Docker runs the local deployment; the documented options also include EC2 and Kubernetes, with different placement of the control components. The architecture overview explains those boundaries.

Consider a proposed support workflow: a planner assigns document research, researchers collect evidence, and a synthesis agent drafts the response. This is an illustrative design, not a measured CanyonOS benchmark. The blue stage in the figure is where adding capacity might help.

Scale the bottleneck, then measure the whole task3 calls / task. Planner: 12 workflows / minute. One task launches three research calls.. Research queue: 36 calls / minute arrive. Track queue age as well as length.. Worker 1: 12 calls / min assumed capacity. Worker 2: 12 calls / min assumed capacity. Worker 3: 12 calls / min assumed capacity. Worker 4: 12 calls / min assumed capacity. Join and synthesize: 48 calls / minute of theoretical capacity. Verify the end-to-end outcome.. A provider limit of 24 calls / minute still wins. More containers cannot remove that cap, and retries consume it too.. A planner sends three research calls per task into a queue. Four replicas each have an assumed capacity of twelve calls per minute. Their results join before synthesis. The figures are planning assumptions, not measured CanyonOS performance. Scale the bottleneck, then measure the whole task Illustrative workload arithmetic. Replicas add capacity only within upstream limits. RESEARCH AGENT: FOUR INDEPENDENT REPLICAS 3 calls / task Planner 12 workflows / minute. One task launches three research calls. Research queue 36 calls / minute arrive. Track queue age as well as length. Worker 1 12 calls / min assumed capacity Worker 2 12 calls / min assumed capacity Worker 3 12 calls / min assumed capacity Worker 4 12 calls / min assumed capacity Join and synthesize 48 calls / minute of theoretical capacity. Verify the end-to-end outcome. A provider limit of 24 calls / minute still wins. More containers cannot remove that cap, and retries consume it too.

Use our single-loop versus multi-agent question before introducing that split. Separate services earn their cost when work can run independently, needs different resources, or requires a separate failure boundary. Splitting every function into an agent adds coordination without necessarily improving an answer.

Find the bottleneck before adding replicas

Suppose the planner receives 12 workflows per minute. Each launches three research calls, giving the research stage 36 calls per minute. If one replica completes 12 calls per minute, three replicas only match average demand. Four provide a theoretical capacity of 48, before provider limits and variability.

That arithmetic is a planning estimate. A model provider capped at 24 calls per minute still caps the whole stage at 24, regardless of container count. Retries can consume that allowance before useful work does.

Measure arrivals, completed calls, oldest queued request, active work, error rate and provider throttling per agent. Compare backpressure and circuit breaking with exponential backoff: the first limits incoming work, while the second controls when failed work returns. More replicas cannot repair an unbounded retry loop.

The current scaling guide supports completed requests per minute per replica or total queue length. One workflow policy applies separately to each agent. It waits for four qualifying measurements and adjusts by one replica, with a default five-second poll. Starting a new replica adds further delay. It never scales to zero.

An illustrative config/scaling.yaml is:

yaml
scaling:
  min_replicas: 1
  max_replicas: 4
  metric: queue_length_total
  scale_up_above: 8
  scale_down_below: 2

Those numbers are teaching choices, not recommended defaults. Dashboard edits must be copied back to the file if they should survive a configuration reload. Test a burst and a sustained load separately: a policy that catches a long backlog may react too late for a short interactive deadline.

Plain Python still has a runtime contract

“No decorators” does not mean arbitrary Python behaves unchanged after distribution. The runtime contract requires a module-level class, a constructor without arguments and declared callable methods. Methods are invoked synchronously; an async def is not awaited. Calls share an instance across a thread pool, so mutable state must be thread-safe.

For a minimal agent, save this as agents/normalize_agent.py:

python
class NormalizeAgent:
    def run(self, query: str) -> dict:
        if not isinstance(query, str) or not query.strip():
            raise ValueError("query must contain text")
        return {"query": " ".join(query.split())}

Its matching config/workflow_config.yaml is:

yaml
target: docker
provider: local
agents:
  - name: NormalizeAgent
    entrypoint: agents/normalize_agent.py
    replicas: 1
    functions:
      - name: run
        arguments:
          - name: query
            type: str
        returns:
          type: dict
endpoint:
  agent: NormalizeAgent
  method: run

The explicit input check is intentional. A declared type does not convert or validate a runtime value for you. This small example needs no model credential and returns a JSON-compatible object. Test it directly before distribution, then run the CLI's validation against the folder. The configuration reference defines the accepted keys.

We checked this fixture with canyonos==0.2.2: normalization and four invalid inputs behaved as expected, and canyonos validate . --json returned zero errors. This verifies the local contract, not a deployed cluster or scaling benchmark. The CLI guide documents default anonymous command reporting; use canyonos contact --usage off before validation if that reporting is unwanted.

For an existing application, canyonos convert asks a coding agent to create a separate deployment under .canyon. Review that output, particularly imports, dependencies, state storage and the methods made callable. Keep a versioned copy: the conversion guide notes that the generated directory is normally added to .gitignore.

Deploy with a failure experiment in mind

Use a disposable machine or isolated development environment for the first Docker deployment. Inspect listening interfaces before exposing it: the documented local endpoint and dashboard ports can bind on all interfaces. Put authentication, network restrictions and TLS where your deployment requires them. Keep real customer prompts out of the first telemetry experiment.

Start with a request that calls two independent workers and joins their results. Then stop one worker mid-request. Record what the endpoint returns, whether the task is retried, and what happens if the original worker later finishes. An idempotency key belongs to the business operation, not merely the HTTP request that happened to carry it.

Pin the code revision and configuration together. Keep an existing working deployment available while you test the new one. A successful container build does not verify imports in every generated image, state recovery or safe draining.

Read the trace as evidence

The useful trace separates queue wait, agent work, model calls and join wait. Add outcome information such as whether the response cites the required evidence and whether a human had to finish the task. Token cost alone cannot identify a useful agent.

For the proposed support workflow, compare one and four researcher replicas on the same request set. Accept the change only if end-to-end latency improves within provider quotas, duplicate actions remain absent, and cost per completed case stays acceptable. Our agent-observability concept explains the trace vocabulary; the production agents course develops the surrounding outcome and recovery contract.

PRACTICE THIS

Turn it into offers. Work the real questions and concepts this maps to:

FAQ

CanyonOS is an AGPL-3.0 platform for deploying and operating multi-agent Python workflows. It creates agent containers, routes calls through generated stubs, and exposes workflow telemetry and model-cost information.

Discussion (3)

Lukas HoffmannEditor

Queue age deserves a graph beside queue length. Ten jobs waiting for a slow research tool can be worse than a hundred short classification jobs. Autoscaling needs the service-time distribution to make those counts useful.

Emily CarterEditor

Treat conversion as a code change with an acceptance suite. The original files can remain untouched while the deployed behavior changes through generated stubs, serialization and concurrency.

Mei LinEditor

A dashboard cost total should carry its pricing version and missing-usage count. A zero can mean no charge, missing provider usage, or a model whose price is absent from the mapping.