Scale AI RAG & Agent System Design interview questions

The warm-up that filters out tutorial-level candidates. Interviewers aren't grading the diagram, they're grading whether you know the three places real pipelines fail and how you'd see it happening.

Everyone says '500 tokens with overlap.' The candidates who get hired explain why chunk boundaries are a recall problem, name three strategies, and describe the experiment that picks between them.

'Evals first' is the #1 hiring signal in FDE design rounds. This is the foundational version: what a 50-case golden set looks like, which metrics split retrieval from generation, and the launch gate that convinces interviewers.

A reported FDE design question that punishes one-size-fits-all answers. Each corpus has a different 'semantic unit': get the three designs, the parsing tools to name, and the eval that proves the split was worth it.

The modal FDE design question of the last two years. There's a scoring trap in the first 60 seconds, and most candidates jump to fixes and fail. Here's the diagnostic tree that wins it.

The unglamorous design question that exposes who has run LLMs in production. Batch APIs at 50% off, idempotency keys, poison records, spend kill-switches: the full checklist interviewers listen for.

The staff-level version of 'evals first.' Four layers, each with a job, a cost, and a blind spot, plus judge calibration numbers and the data flywheel that ties production failures back into CI.

The POC-to-production gap is the FDE's home turf. What actually breaks at each order of magnitude (index, ingestion, latency, cost) and the dependency math that makes 99.9% harder than it sounds.

Reciting the four definitions is table stakes; the differentiator is knowing which two need ground truth, what each dimension tells you to fix, and where the judge-model scores quietly lie. The 2x2 that turns RAGAS into a debugging tool.

The classic 'it regressed and the prompt is untouched' triage. The answer that scores walks a decisional tree from retrieval to generation and names the silent culprit most candidates miss: the embedding stack changed under you.

Hard budgets stop the bleeding; they don't fix the cause. The real fix is progress tracking plus semantic loop detection plus a planning layer that commits to a plan instead of re-deciding every step, validated on a fixed trajectory set offline before it ships.

You cannot unit-test a stochastic two-agent pipeline with a golden output. You evaluate it the way you stress a distributed system: thousands of simulated runs, controlled fault injection, and a judge calibrated against humans.

DSPy's pitch is to stop hand-writing prompts and instead declare what the pipeline does, define a metric, and let a compiler search for the prompts and examples. The catch is that the compiler is only as good as the metric, and an LLM-judge metric drags its own biases in.

The retrieval design question with a precise rubric: candidates who can say what each stage rescues, and what it costs in latency, clear it. Includes the RRF detail and the latency budget interviewers ask for.

Pure vector search fails on the exact queries enterprise users ask most: IDs, error codes, product names. Here's the failure-mode framing that turns a definition question into a design answer.

Three plausible fixes, one decision framework. Interviewers grade the ordering (cheapest-reversible first) and whether you can say what each option costs in data, time, and operational burden.

Text-to-SQL's nightmare isn't syntax errors, it's the query that runs, returns a confident number, and is silently wrong. The defense is a semantic layer plus transparency UX. Here's the full design.

Arguing against agents reads senior; it's an explicit rubric signal at AI labs. Learn the cost ladder from single call to autonomous loop, and the three conditions that actually justify an agent.

Sierra's signature design exercise. The grading hinges on three things most candidates underweight: tool API shape, the read/write trust boundary, and eval cases written before the agent. Walkthrough inside.