Course outline · choose a lesson
Choose a runtime by the controls you can demonstrate
Compare runtimes using the same task, permission, recovery and verification contract. Current product features are useful only when the configured integration demonstrates the behavior your customer requires.
24 MIN
By FDEInterviews · Updated
TL;DR: Choose a runtime by demonstrating the controls your customer requires on the same task. Compare extension boundaries, state persistence, permissions, observation and recovery; product names and feature lists do not establish the behavior of your configured integration.
Where you are. You have built the course's control mechanisms. This public lesson helps you evaluate current coding and domain-agent runtimes against those contracts.
A portable acceptance packet
Northstar's packet contains a broken threshold function, exact acceptance cases, protected evaluator, scoped purchasing evidence, a pending approval and a worker-crash scenario. Run the same packet through each candidate integration. Keep the starting artifact, task scope and measurement policy fixed.
A useful comparison asks what the runtime owns and what your application has to supply: whether it persists a session transcript, a resumable workflow state or both; whether a hook can prevent an action at the required boundary; where credentials are injected; which code can write acceptance results; and how a model refusal reaches your controller.
Do not collapse those questions into a single “supports agents” checkbox. A runtime can be excellent at interactive coding while leaving business approvals and durable effect reconciliation to the host application, and that can be the right choice as long as you implement those responsibilities explicitly.
Mechanisms before products
| Design choice | Useful when | Evidence to request |
|---|---|---|
| Small extensible core | You need a narrow custom interaction and can own more integration code | Extension lifecycle, permission boundary and compatibility tests |
| Integrated coding workflow | Repository navigation, editing and review dominate the task | Effective instructions, workspace controls and actual verification path |
| Plugin-oriented runtime | Tools and services need independent extension points | Plugin provenance, resource disposal, event contracts and privilege limits |
| Durable workflow engine | Tasks pause, branch and resume across processes | Persistent checkpoints, replay behavior, ownership and effect reconciliation |
| Managed execution service | You want the provider to operate part of the runtime | Configured isolation, retention, region, export and failure semantics |
The categories overlap: one product can expose several layers, and one application can combine them. What you are deciding is which layer owns each required contract in your deployment.
Required behavior is the first gate. In a workflow that changes customer records, lower cost or a convenient API cannot make up for missing authority enforcement.
Normalizing the model boundary
Your adapter should distinguish completed output, refusal, incomplete output and transport failure. A truncated JSON object is not a partial action to execute. A model timeout can leave usage unknown even though no customer tool has run. Keep provider request IDs and bounded usage metadata for diagnosis without copying credentials into traces.
The companion's adapters.py defines a provider-neutral reply contract and tests those states with recorded replies. It is not a tested integration with every current SDK, and it does not claim to be. The dated framework worksheet identifies the live checks to perform: actual instruction loading, timeout behavior, tool-call normalization, permissions, process restart and retained evidence.
The contract is quick to see in action. From the companion directory, send five recorded replies through invoke_decision; test_adapters covers the same states and passes 4 of 4:
from adapters import ModelReply, invoke_decision
valid = '{"kind": "check", "task_id": "T-17", "args": {"name": "threshold"}}'
replies = {
"completed, valid": ModelReply("completed", valid, "req-1", 812, 31),
"completed, truncated": ModelReply("completed", valid[:40], "req-2", 812, 16),
"refused": ModelReply("refused", "", "req-3", 812, 9),
"incomplete": ModelReply("incomplete", valid[:30], "req-4", 812, None),
}
for name, reply in replies.items():
result = invoke_decision(lambda request, r=reply: r, {}, "T-17")
print(f"{name:22} {result['status']:9} {result.get('reason', ''):16} usage_known={result['usage_known']}")
def hung(request):
raise TimeoutError
result = invoke_decision(hung, {}, "T-17")
print(f"{'transport timeout':22} {result['status']:9} {result['reason']:16} usage_known={result['usage_known']}")
completed, valid proposal usage_known=True
completed, truncated blocked invalid_decision usage_known=True
refused blocked refused usage_known=True
incomplete blocked incomplete usage_known=False
transport timeout unavailable model_timeout usage_known=False
Only the first line produces something the controller may consider, and even then it is a proposal, not an action. The truncated reply is the instructive one: the provider reported completed, the usage numbers are complete, and the text is still useless, so the adapter's own parse is what blocks it. The last two lines show the usage rule from the paragraph above: an incomplete reply missing its output count, and a timeout with no reply at all, both record usage_known=False instead of a zero.
For a real adapter, enforce the transport deadline where the call is made. A wrapper that catches TimeoutError cannot interrupt an underlying function that never raises one. Test cancellation during an actual call and inspect whether any already-issued tool effect still needs reconciliation.
The Premium companion also includes http_model.py, a complete optional HTTP adapter with local-service tests and a separate live command. It makes one bounded request and returns a proposal plus available usage; it never executes that proposal. Its tested protocol and untested external-provider boundary are documented in FRAMEWORKS.md.
Do this before moving on
Two synthetic runtime candidates both pass the threshold repair. Candidate A can persist a transcript but loses its workflow checkpoint on process death. Candidate B resumes the graph but requires you to implement the buyer-approval adapter. The customer requires unattended recovery and exact buyer approval before amendments.
Deliverable: write a fit decision and the minimum missing work for each candidate. Do not invent a winner from the incomplete evidence.
Worked review. A needs durable workflow state and demonstrated restart recovery; a saved transcript alone does not establish either. B needs an authenticated, proposal-bound approval adapter and service-side enforcement. Evaluate whether those missing boundaries can be implemented and operated within the engagement. Neither passes the full contract yet.
Laid out against the portable packet, the gap is easier to cost:
| Required behavior | Candidate A | Candidate B |
|---|---|---|
| Threshold repair with protected evaluator | demonstrated | demonstrated |
| Scoped purchasing evidence | to test | to test |
| Unattended recovery after worker death | missing: checkpoint lost on process death | demonstrated: graph resumes |
| Exact buyer approval before amendments | to test | missing: host must build the adapter |
| Service-side enforcement of the approval | host's job either way | host's job either way |
Two rows say "to test" because the supplied evidence is silent on them, and silence is not a pass. The last row is the same for both: neither runtime can enforce approval inside the customer's order service, so that work belongs in the plan whichever candidate wins.
Change the input. The task becomes an interactive, read-only coding assistant where a human owns each session and no business write occurs. Durable approval graphs may no longer be necessary. Reassess the required packet, keeping repository permissions and honest verification. The right runtime can change because the task changed, not because one product became better at everything.
The field notes below were checked against official documentation in October 2026. Treat them as a starting map, and verify the specific version and configuration you deploy.
Go deeper
- Agent framework failure modes: Compare runtime responsibilities without confusing an SDK feature with an enforced contract.
- Harness design decisions: Use the design vocabulary while reading the survey’s evidence limits carefully.
- Harness experiments: Evaluate planning, tools and context choices on the model and workload you actually use.
- Production AI Agents: Extend the adapter comparison to a complete support-agent implementation.
Key takeaways
- Choose a runtime by a portable packet of required behavioral evidence.
- Normalize refusal, incomplete output and timeout separately from valid actions.
- Recheck product-specific assumptions against the deployed version and configuration.
Check yourself
Put the notes aside. Make a decision, then check the reasoning behind it.
Scratch answers stay on this page. They are not saved or graded, and this practice does not mark the lesson complete.
1A runtime saves every conversation. Does that establish durable workflow recovery?
2What should a provider adapter do with incomplete JSON output?
Apply it to a new situation
3Practice case: runtime A demonstrates the full customer acceptance packet in staging. Runtime B advertises more built-in recovery features, but its configured adapter has only a happy-path demo. Both fit the budget. What should the selection memo say?
Clears these scratch answers and hides the worked answers.
Sign in free to save your progress and return to this lesson.
