FDE System Design: Designing AI Systems Under Customer Constraints

FDE system design asks you to design an AI system that works inside one specific customer's environment, not a hypothetical service at internet scale. The constraints that decide your design are the deployment boundary (often a private VPC or an air-gapped network), data residency and compliance rules, identity and SSO, a fixed latency or cost budget the customer sets, and an evaluation plan that proves the thing works before it goes live. A textbook distributed systems round rewards sharding and throughput. This round rewards fit: a design that survives the customer's security review, runs under their budget, and can be measured. Below is what changes, followed by a worked example of a RAG system deployed in a regulated VPC.

What makes it different from the textbook round

A standard system design interview is a scaling exercise. Design a URL shortener for a billion links, a news feed for a billion users, a rate limiter for a global API. The implicit goal is throughput, and the constraints are ones you invent to justify your own architecture.

The FDE round inverts that. The customer supplies the constraints, and they are rarely about scale. A single bank deploying an internal assistant might have a few thousand users, which is trivial load. The hard parts are everywhere else: the data cannot leave their network, every query has to respect a permission model their identity provider already enforces, the legal team needs to know where each byte is stored, and the security review can kill your design in one sentence. Your job is to design backward from those walls.

Five constraint families show up in almost every FDE design round. Learn to ask about each one before you draw a single box.

This is why the system design and RAG and agent design categories overlap so heavily for FDEs. The architecture is half the answer. The other half is the deployment and the proof.

The worked example: a RAG assistant in a regulated VPC

Here is the prompt as a customer would phrase it. "We are a regulated financial services firm. We want an internal assistant our analysts can ask questions of, grounded in our research documents and filings. Our security team will not let any of that data leave our AWS VPC, and answers have to respect who is allowed to see which documents. We need answers in a couple of seconds, and our compliance team needs to sign off before this touches a real user."

Resist the urge to architect. Confirm the constraints first, because each one removes options.

• Boundary. It is a private VPC, not air-gapped, so you have controlled outbound access through a proxy. That means a hosted model API is possible if the customer's security team approves a data processing agreement, but you should design so that an on-prem model is a drop-in fallback if they say no. Always have that fallback ready.
• Residency. Everything stays in their AWS account and region. So the vector database, the document store, the embedding service, and the logs all live inside the VPC. Nothing transits a third-party SaaS.
• Identity. They use Okta. Answers must respect existing document permissions, so access control cannot be an afterthought bolted on at the end.
• Budget. A couple of seconds end to end. That is your hard latency budget, and it constrains how many retrieval and reranking passes you can afford.
• Evaluation. Compliance must sign off, which means a documented evaluation plan is part of the deliverable, not a nice-to-have.

The architecture

Walk the panel through the request path and tie each hop to a constraint.

A query enters through the app API behind Okta SSO. The user's identity and group membership travel with the request, because retrieval has to be access-aware. The retriever queries the vector database but filters candidates by the access control list before they ever reach the model. This is the part candidates most often miss: if you filter after retrieval or, worse, rely on the prompt to tell the model not to reveal restricted content, you have built a data leak. The filter belongs at the retrieval layer, enforced by the same group claims Okta provides.

Retrieved chunks go through a reranker to lift precision, then to the model, which generates an answer with citations back to the source documents. Citations matter doubly here: analysts need to verify, and compliance needs an audit trail. Every request and response is written to an audit log inside the VPC.

Spending the latency budget

With a couple of seconds end to end, do the arithmetic out loud. Retrieval against a warm vector index is fast, on the order of tens of milliseconds. A cross-encoder reranker over a few dozen candidates adds a slice. Generation dominates, and it depends on the model and answer length. If the budget gets tight, the levers are: rerank fewer candidates, use a smaller or distilled model, or stream the first tokens so the perceived latency drops even if total time does not. Naming those levers and the order you would pull them is the senior signal.

The model decision

This is the fork the interviewer is waiting for. If the security team approves a VPC-routed hosted model API under a data processing agreement, you get frontier quality with near-zero ops, and you should take it. If they refuse, you self-host an open-weight model on GPUs inside the VPC, served with something like vLLM for throughput, and you size the hardware honestly. State the tradeoff plainly: self-hosting trades model quality and operational convenience for full control and an easier compliance story. The wrong answer is to pick one silently. The right answer is to make the call and name the condition that flips it.

The evaluation plan is not optional

A regulated customer will not let a system that affects analysts go live on vibes. Reserve real time for the evaluation plan, because in this round it is what gets you past compliance.

Build a golden set of representative questions with known-good answers, labeled by the customer's own domain experts. Measure retrieval quality (did the right documents surface) separately from answer quality (was the generated answer faithful and correct), because they fail for different reasons and the fix differs. Set gating metrics that a release has to clear before it ships, and run an automated eval suite on every change so you catch regressions. For a regulated customer, add a check for the failure that scares them most: did the system ever surface a document the user was not allowed to see. That number should be zero, and you should be able to prove it. The ML infrastructure work of making evals repeatable is what turns a demo into a deployment.

How to practice this round

Take any RAG or agent design and re-run it five times, each time with a different customer constraint bolted on: air-gapped, sub-second latency, a tiny GPU budget, strict data residency across two regions, an existing identity provider you must integrate. The architecture should change each time, and being able to explain why is the skill. Start with the system design and RAG and agent banks, then pressure-test your instincts against the must-know set.

The one-line version

Design backward from the customer's walls, not forward from a clean whiteboard. Ask where the data lives, who can see it, what the budget is, and how you will prove it works, then build the smallest system that fits all four. That is the design an enterprise security team actually approves.