Hard Deployments
The deployments that break people: open-weight models on hardware you did not buy, environments with no internet, one platform serving tenants who must never see each other, an engagement already on fire when you arrive, and the arithmetic that decides all of it. The hardest material on this site.
Begin with: Budget weights, resident cache and runtime memory in consistent units
Syllabus
Free lessons need no account. Sign in free for Plus lessons. Premium unlocks the full course.
Serving on hardware you did not choose
The memory arithmetic that decides whether a model runs at all, the difference between throughput and latency once you are batching, and how to size a box before the customer buys it rather than after.
- Budget weights, resident cache and runtime memory in consistent unitsVideo explainer · 11:3830 minfree
- Measure throughput and user latency before choosing a batching policyVideo explainer · 11:0130 minsign in
- Hardware arrives on a procurement cycle and your estimate arrives in a meetingVideo explainer · 12:2630 minpremium
- Measure throughput without losing the requests that failedproject180 minpremium
- Build and verify an inference service under failureproject240 minpremium
The air gap
Building for an environment with no route to the internet: vendoring everything your build silently assumes, moving an update through a one-way door, and producing the evidence a regulator will accept without ever phoning home.
One platform, many customers
Isolation you can defend to a security reviewer rather than describe in a diagram, scheduling that stops one tenant ruining the median for everyone, and a blast radius small enough that a bad hour is not a bad quarter.
- Specify and test tenant isolation at every layerVideo explainer · 13:0630 minsign in
- Bound shared demand and measure the tenants hidden by the medianVideo explainer · 11:2130 minpremium
- Calculate failure scope from dependencies and rollout stateVideo explainer · 11:2230 minpremium
- Test the model server's cache, adapter and shared-failure boundariesVideo explainer · 12:2130 minpremium
Cost and latency across a portfolio
Finding where the money and the milliseconds actually go when you run many deployments, pulling the three levers that move them without quietly trading away quality, and planning capacity against a price ceiling somebody else set.
Autonomy over long horizons
What changes when an agent's work spans hours rather than seconds: state that survives the process dying, memory that stays useful instead of accumulating, and oversight of a fleet where the correlated failure is the one that hurts.
- An agent that keeps its plan in memory has a plan that ends with the processVideo explainer · 10:5030 minpremium
- Keep remembered claims current enough and preserve the decisions that matterVideo explainer · 13:1230 minpremium
- Two hundred agents making the same mistake is one incident, not two hundredVideo explainer · 12:3330 minpremium
Rescue engineering
Walking into a deployment that is already failing: forming a defensible diagnosis in 48 hours without the people who built it, proving what actually happened from data rather than from accounts of it, and a fortnight's plan that restores confidence before it restores features.
- You have two days, no context, and everybody has a different theoryVideo explainer · 15:1430 minfree
- The database remembers what the meeting does notVideo explainer · 13:0330 minpremium
- Build a recovery plan that fits the evidence, dependencies and available timeVideo explainer · 13:5230 minpremium
- Lab: diagnose a failing deployment from exports that disagree with the meetingproject120 minpremium
Reliability across a portfolio
Service levels that mean something when you run twenty deployments rather than one, running an incident inside an environment you cannot reach, and deciding in advance what the system does when the model it depends on is unavailable.
- Twenty deployments, twenty different definitions of workingVideo explainer · 12:0130 minpremium
- Running an incident where every action needs somebody else's handsVideo explainer · 13:0430 minpremium
- Decide now what the system does on the day the model is unavailableVideo explainer · 12:1730 minpremium
Extracting a platform from field code
Turning three bespoke builds into one thing worth maintaining: finding the abstraction that is really there rather than the one you hoped for, designing for variation without a configuration language nobody can read, and moving existing customers across without breaking them.
- Three customers doing the same thing differently is not yet a productVideo explainer · 11:1230 minpremium
- Your configuration file has become a programming language nobody can debugVideo explainer · 9:1830 minpremium
- The customers on the old code did not ask for your platformVideo explainer · 10:4330 minpremium
The practice
What makes the work repeatable by people who are not you: evaluation infrastructure that serves a fleet rather than a demo, field evidence packaged so the product roadmap can act on it, capability transfer, and the admission, hiring, staffing and funding decisions needed to run a sustainable practice.
- Twenty deployments need one harness and twenty sets of casesVideo explainer · 9:4930 minpremium
- Turn field observations into a product decision someone can evaluateVideo explainer · 9:0630 minpremium
- A bottleneck you are proud of is still a bottleneckVideo explainer · 11:1630 minpremium
- Admit work the practice can finish20 minpremium
- Hire for the work and ramp with evidence25 minpremium
- Plan the portfolio by skill and calendar25 minpremium
- Lab: defend the practice investmentproject90 minpremium
Capstone
One engagement that uses every module: an air-gapped, multi-tenant, already-failing deployment with agents in flight, three bespoke units to consolidate, and a price ceiling. The deliverable is a design and a set of decisions you can defend for two hours against somebody trying to break them.
About this course
Before you start: The FDE Engagement, or several years running deployments end to end. You should already be comfortable with an eval harness, an entitlement model, and owning a production system inside somebody else's environment. This course assumes all of that and starts from there.
Build a service and verify it under failure
The streaming inference service project provides a complete source download and a staged written implementation guide. Build a small HTTP service with admission limits, cancellation and deadlines; inject failures; deploy its container; and retain every request in a load report. The Premium companion includes setup instructions, working reference code, tests, deployment files and follow-up requirements.
The core runs locally with synthetic text and no model account. An optional real-model adapter and a cloud VM deployment runbook are included. Those stages need your own runtime and authorized environment; fixture measurements do not establish model quality or GPU capacity.
Course format and samples
Hard Deployments combines detailed written lessons, executable examples, practical projects and a capstone. Narrated videos are available on lessons marked with a video badge. They explain lesson mechanisms and examples; the course does not currently include a continuous instructor-led recording of a complete application built from scratch and deployed to the cloud.
The project source is supplied as a downloadable ZIP, not a hosted GitHub repository. Other labs contain their own examples and exercises. The final capstone produces an engineering dossier for a supplied scenario; it is separate from the runnable service project.
Read two complete free lessons before choosing the course: Does it fit? and The first 48 hours. The Does it fit? narrated video shows the current video teaching style. These are lesson samples, not complete cloud build demonstrations.
For a larger local Python system, Production AI Agents for FDEs has a connected companion with recorded model decisions, durable state and recovery tests. For browser-side network failures, Repair a customer's streaming assistant includes a broken client, local server, automated checks and a separate written solution walkthrough. Their public overviews describe the exercises and access.
The deployment problems this course covers
Every other course on this site is about doing the job well. This one is about the deployments where doing the job well is not sufficient, because the environment itself is hostile to the way software is normally built.
There is no internet. The hardware was bought two years ago by somebody who has left. The model weights have to cross a boundary on physical media, with a signature, and be validated by a person who is not you. Eleven tenants share one platform and two of them are competitors. The engagement has been running for five months, nothing works, and you have been given a fortnight.
None of that is exotic. It is where the largest contracts live, it is why the role exists at all, and it is the work that separates an engineer who can run a deployment from one an account will ask for by name.
This course is technical first. The arithmetic of whether a model fits in the memory available. The scheduler behavior that decides whether one tenant can ruin the median for everyone else. The forensic method for proving what happened from rows in a table. Leadership is the last module rather than the theme, because at this level the influence follows the judgment rather than replacing it.
Two of these modules assume you will never have the access you want. Read them in order: the sequence is a deployment getting harder, and the later modules assume the constraints established in the earlier ones.
When you finish, you can
- Size and serve an open-weight model on hardware you do not control, and say why it fits before anyone buys it
- Deploy, update and evidence a system inside an air gap, where nothing calls home and every change is a one-way door
- Architect multi-tenant isolation you can actually defend, with quotas, fair scheduling and a bounded blast radius
- Find where the money and the milliseconds go across a portfolio, and pull the three levers that move them
- Run agents over long horizons: durable state, memory that does not rot, and oversight across a fleet
- Diagnose a failing deployment in 48 hours from logs and data, in a codebase nobody will explain
- Extract a real platform from three bespoke builds and migrate customers off the one-off without breaking them
- Build a practice decision packet covering admission, hiring evidence, skill-constrained capacity and a reconciled investment case
Premium companion · JavaScript
Build and verify a streaming inference service
Get the complete source project, staged implementation walkthrough, failure tests, container deployment files, load generator and review packet. Build admission limits, streaming, cancellation and deadlines, then measure the system through its HTTP interface.
Node 22 or later; Docker is optional. The core runs locally with synthetic text and no model account. A real-model adapter and cloud VM runbook are included; those stages need your own runtime and environment. This is a written project, with no recorded coding walkthrough.