From Proof-of-Concept to Production

TL;DR: A POC proves the idea can work once; production proves it keeps working when you are asleep and the customer is touching it. The gap is idempotency, error handling, observability, runbooks, and an owner who is not you. Customers do not buy demos, they buy systems they can operate.

The demo that lies to you

The POC runs in a notebook. You paste in a clean input, the model returns a good answer, the room nods. That demo is honest about one thing (the idea has signal) and silently lies about everything else. It ran once, on input you chose, on your machine, with you watching, with no second user, no retry, no malformed row, no expired token, no pager.

Production is the set of guarantees that survive your absence. The work to add them is rarely interesting and almost never demoed, which is exactly why customers pay an FDE to do it. The first failed overnight batch is what converts a champion into a skeptic, so the gap is where deals are actually won or lost.

What "done" actually means

Walk the same pipeline as if it will run unattended for a year:

• Idempotency. The job will get retried, double-clicked, or replayed from a queue. Re-running it must not double-charge, double-insert, or double-send. Use an idempotency key per logical operation and upserts instead of blind inserts.
• Error handling. Decide per failure: retry (transient, with backoff), skip and quarantine (one bad row), or halt (the whole batch is suspect). A notebook does none of these; it throws and dies on row 4,000 of 50,000.
• Observability. You cannot operate what you cannot see. Structured logs with a request id, a metric for throughput and error rate, and one trace that follows a single record through every stage. For an LLM system add token cost and a sample of inputs and outputs.
• Monitoring and alerts. A threshold that pages a human before the customer notices. Silent degradation, answers slowly getting worse, is the failure mode that erodes trust fastest.
• A runbook. The plain-language doc that lets the customer's on-call engineer answer "it is down at 2am, what do I do?" without calling you. This is the artifact that ends your dependency.

Every step below the first is something the demo never had to survive. None of it is interesting to build and all of it is what the customer is actually buying.

A concrete scenario

A POC enriches support tickets with an LLM and writes them to the CRM. In the demo it processes ten tickets perfectly. In production: the API rate-limits at ticket 200, a ticket with no body throws, the run is retried after a timeout and writes every enrichment twice, and nobody notices the CRM is full of duplicates for three days because there was no metric on write counts. None of those are model problems. All of them are the production gap.

Why interviewers probe this

This question screens for whether you have actually shipped, or only prototyped. The tell is in what you bring up unprompted: a candidate who immediately asks "what happens on a retry?" and "who operates this after I leave?" has run real deployments. The reserved follow-up is usually a triage one ("it worked in staging and fails in prod, the customer is angry, walk me through your first hour"), and they want to see you reach for logs, metrics, and a recent-change diff rather than guesses.

Common misconceptions

• "It works, so it is done." Working once on your input is the start of the work, not the end. Done means it survives unattended on the customer's input.
• "We will add monitoring later." Later is after the silent failure already cost trust. Observability is part of go-live, not a follow-up ticket.
• "Production-ready means scale." For most FDE deployments it means reliable and operable at the current volume, not horizontally scaled to millions. Solve the failure that will actually happen.
• "The customer can read the code." They cannot and will not. They need a runbook and an alert, not your repo.

Key takeaways

• The POC-to-production gap is the FDE job; the hardening work is unglamorous and rarely demoed, which is why it is valuable.
• "Done" = idempotent, handles errors deliberately, observable, monitored, and documented in a runbook the customer can act on.
• Design for your own absence: the deliverable is a system someone else can operate at 2am, not a notebook only you understand.
• Trust is built by surviving the first overnight failure cleanly, and lost by the first silent one.