GO TO A SECTION

↑ ↓ to navigate · Enter to openExplore all sections →
FDEInterviews logo
Lesson 39 of 44 · FreeCourse syllabus
Course outline · choose a lesson

Choose a runtime by the controls you can demonstrate

Compare runtimes using the same task, permission, recovery and verification contract. Current product features are useful only when the configured integration demonstrates the behavior your customer requires.

24 MIN

By FDEInterviews · Updated

TL;DR: Choose a runtime by demonstrating the controls your customer requires on the same task. Compare extension boundaries, state persistence, permissions, observation and recovery; product names and feature lists do not establish the behavior of your configured integration.

Where you are. You have built the course's control mechanisms. This public lesson helps you evaluate current coding and domain-agent runtimes against those contracts.

A portable acceptance packet

Northstar's packet contains a broken threshold function, exact acceptance cases, protected evaluator, scoped purchasing evidence, a pending approval and a worker-crash scenario. Run the same packet through each candidate integration. Keep the starting artifact, task scope and measurement policy fixed.

A useful comparison asks what the runtime owns and what your application has to supply: whether it persists a session transcript, a resumable workflow state or both; whether a hook can prevent an action at the required boundary; where credentials are injected; which code can write acceptance results; and how a model refusal reaches your controller.

Do not collapse those questions into a single “supports agents” checkbox. A runtime can be excellent at interactive coding while leaving business approvals and durable effect reconciliation to the host application, and that can be the right choice as long as you implement those responsibilities explicitly.

Mechanisms before products

Design choiceUseful whenEvidence to request
Small extensible coreYou need a narrow custom interaction and can own more integration codeExtension lifecycle, permission boundary and compatibility tests
Integrated coding workflowRepository navigation, editing and review dominate the taskEffective instructions, workspace controls and actual verification path
Plugin-oriented runtimeTools and services need independent extension pointsPlugin provenance, resource disposal, event contracts and privilege limits
Durable workflow engineTasks pause, branch and resume across processesPersistent checkpoints, replay behavior, ownership and effect reconciliation
Managed execution serviceYou want the provider to operate part of the runtimeConfigured isolation, retention, region, export and failure semantics

The categories overlap: one product can expose several layers, and one application can combine them. What you are deciding is which layer owns each required contract in your deployment.

An interface match is not proof of the required authority, recovery or acceptance behavior. Customer acceptance and authority contract → Same portable test packet. Same portable test packet → Runtime adapter A. Same portable test packet → Runtime adapter B. Runtime adapter A → Observed control and recovery evidence. Runtime adapter B → Observed control and recovery evidence. Observed control and recovery evidence → Required behavior demonstrated?. Required behavior demonstrated? → Implement missing boundary or reject fit when no. Required behavior demonstrated? → Review cost, operations and maintainability when yes. Test runtime choices against the same contractSERVICECustomer acceptance and authority contractSTATESame portable test packetSERVICERuntime adapter ASERVICERuntime adapter BSTATEObserved control and recovery evidenceDECISIONRequired behavior demonstrated?HOLDImplement missing boundary or reject fitCHECKReview cost, operations and maintainabilitynoyesAn interface match is not proof of the required authority, recovery or acceptance behavior.

Required behavior is the first gate. In a workflow that changes customer records, lower cost or a convenient API cannot make up for missing authority enforcement.

Normalizing the model boundary

Your adapter should distinguish completed output, refusal, incomplete output and transport failure. A truncated JSON object is not a partial action to execute. A model timeout can leave usage unknown even though no customer tool has run. Keep provider request IDs and bounded usage metadata for diagnosis without copying credentials into traces.

The companion's adapters.py defines a provider-neutral reply contract and tests those states with recorded replies. It is not a tested integration with every current SDK, and it does not claim to be. The dated framework worksheet identifies the live checks to perform: actual instruction loading, timeout behavior, tool-call normalization, permissions, process restart and retained evidence.

The contract is quick to see in action. From the companion directory, send five recorded replies through invoke_decision; test_adapters covers the same states and passes 4 of 4:

python
from adapters import ModelReply, invoke_decision

valid = '{"kind": "check", "task_id": "T-17", "args": {"name": "threshold"}}'
replies = {
    "completed, valid": ModelReply("completed", valid, "req-1", 812, 31),
    "completed, truncated": ModelReply("completed", valid[:40], "req-2", 812, 16),
    "refused": ModelReply("refused", "", "req-3", 812, 9),
    "incomplete": ModelReply("incomplete", valid[:30], "req-4", 812, None),
}
for name, reply in replies.items():
    result = invoke_decision(lambda request, r=reply: r, {}, "T-17")
    print(f"{name:22} {result['status']:9} {result.get('reason', ''):16} usage_known={result['usage_known']}")

def hung(request):
    raise TimeoutError
result = invoke_decision(hung, {}, "T-17")
print(f"{'transport timeout':22} {result['status']:9} {result['reason']:16} usage_known={result['usage_known']}")
text
completed, valid       proposal                   usage_known=True
completed, truncated   blocked   invalid_decision usage_known=True
refused                blocked   refused          usage_known=True
incomplete             blocked   incomplete       usage_known=False
transport timeout      unavailable model_timeout    usage_known=False

Only the first line produces something the controller may consider, and even then it is a proposal, not an action. The truncated reply is the instructive one: the provider reported completed, the usage numbers are complete, and the text is still useless, so the adapter's own parse is what blocks it. The last two lines show the usage rule from the paragraph above: an incomplete reply missing its output count, and a timeout with no reply at all, both record usage_known=False instead of a zero.

For a real adapter, enforce the transport deadline where the call is made. A wrapper that catches TimeoutError cannot interrupt an underlying function that never raises one. Test cancellation during an actual call and inspect whether any already-issued tool effect still needs reconciliation.

The Premium companion also includes http_model.py, a complete optional HTTP adapter with local-service tests and a separate live command. It makes one bounded request and returns a proposal plus available usage; it never executes that proposal. Its tested protocol and untested external-provider boundary are documented in FRAMEWORKS.md.

Do this before moving on

Two synthetic runtime candidates both pass the threshold repair. Candidate A can persist a transcript but loses its workflow checkpoint on process death. Candidate B resumes the graph but requires you to implement the buyer-approval adapter. The customer requires unattended recovery and exact buyer approval before amendments.

Deliverable: write a fit decision and the minimum missing work for each candidate. Do not invent a winner from the incomplete evidence.

Worked review. A needs durable workflow state and demonstrated restart recovery; a saved transcript alone does not establish either. B needs an authenticated, proposal-bound approval adapter and service-side enforcement. Evaluate whether those missing boundaries can be implemented and operated within the engagement. Neither passes the full contract yet.

Laid out against the portable packet, the gap is easier to cost:

Required behaviorCandidate ACandidate B
Threshold repair with protected evaluatordemonstrateddemonstrated
Scoped purchasing evidenceto testto test
Unattended recovery after worker deathmissing: checkpoint lost on process deathdemonstrated: graph resumes
Exact buyer approval before amendmentsto testmissing: host must build the adapter
Service-side enforcement of the approvalhost's job either wayhost's job either way

Two rows say "to test" because the supplied evidence is silent on them, and silence is not a pass. The last row is the same for both: neither runtime can enforce approval inside the customer's order service, so that work belongs in the plan whichever candidate wins.

Change the input. The task becomes an interactive, read-only coding assistant where a human owns each session and no business write occurs. Durable approval graphs may no longer be necessary. Reassess the required packet, keeping repository permissions and honest verification. The right runtime can change because the task changed, not because one product became better at everything.

The field notes below were checked against official documentation in October 2026. Treat them as a starting map, and verify the specific version and configuration you deploy.

Go deeper

Key takeaways

  • Choose a runtime by a portable packet of required behavioral evidence.
  • Normalize refusal, incomplete output and timeout separately from valid actions.
  • Recheck product-specific assumptions against the deployed version and configuration.

Check yourself

Put the notes aside. Make a decision, then check the reasoning behind it.

Scratch answers stay on this page. They are not saved or graded, and this practice does not mark the lesson complete.

  1. 1A runtime saves every conversation. Does that establish durable workflow recovery?

  2. 2What should a provider adapter do with incomplete JSON output?

  3. Apply it to a new situation

    3Practice case: runtime A demonstrates the full customer acceptance packet in staging. Runtime B advertises more built-in recovery features, but its configured adapter has only a happy-path demo. Both fit the budget. What should the selection memo say?

Clears these scratch answers and hides the worked answers.

Sign in free to save your progress and return to this lesson.

Back to the course syllabus