Hebbia RAG & Agent System Design interview questions

Harvey's signature design question. The twist: the architecture is the easy half. The graded half is designing for reviewer trust, where one hallucinated clause costs more than a hundred correct ones earn.

Slides are the worst RAG corpus: text in shapes, meaning in charts, the message in the image. A query like 'the blue bar chart with Q3 revenue' fails if you embed only the extracted text, and citing the wrong slide number kills trust instantly.

Before you touch a model, check whether the two numbers measure the same thing. Field accuracy of 96% across twelve fields is 61% of documents fully correct, which means half of the drop may be arithmetic. The other half is the documents the pilot never saw.

The eval set says recall fell 2.3 points, which is inside the noise you would expect from a re-index. Support says the product got worse. Both are right, and the reason is that one document class lost 49 points of recall while everything else improved slightly.

Recall@10 is 0.94 and climbing, and answer accuracy is going down. The two facts are consistent, and reconciling them is the whole question: every passage you add to raise recall also adds a distractor the generator has to survive.

A reranker cannot promote a document the retriever never returned, which makes candidate-set recall a hard ceiling on the entire exercise. Compute the ceiling and the disappointment becomes arithmetic: three points of headroom were available, twelve were sitting upstream, and the latency was an implementation detail.

The warm-up that filters out tutorial-level candidates. Interviewers aren't grading the diagram, they're grading whether you know the three places real pipelines fail and how you'd see it happening.

Everyone says '500 tokens with overlap.' The candidates who get hired explain why chunk boundaries are a recall problem, name three strategies, and describe the experiment that picks between them.

'Evals first' is the #1 hiring signal in FDE design rounds. This is the foundational version: what a 50-case golden set looks like, which metrics split retrieval from generation, and the launch gate that convinces interviewers.

Sierra's signature design exercise. The grading hinges on three things most candidates underweight: tool API shape, the read/write trust boundary, and eval cases written before the agent. Walkthrough inside.

The retrieval design question with a precise rubric: candidates who can say what each stage rescues, and what it costs in latency, clear it. Includes the RRF detail and the latency budget interviewers ask for.

A reported FDE design question that punishes one-size-fits-all answers. Each corpus has a different 'semantic unit': get the three designs, the parsing tools to name, and the eval that proves the split was worth it.

Pure vector search fails on the exact queries enterprise users ask most: IDs, error codes, product names. Here's the failure-mode framing that turns a definition question into a design answer.

Arguing against agents reads senior; it's an explicit rubric signal at AI labs. Learn the cost ladder from single call to autonomous loop, and the three conditions that actually justify an agent.

Three plausible fixes, one decision framework. Interviewers grade the ordering (cheapest-reversible first) and whether you can say what each option costs in data, time, and operational burden.

A Sierra-signature scenario where 'put it in the system prompt' scores zero. The winning answer is defense-in-depth with a measured catch-rate per layer, plus an honest number for what still gets through.

The modal FDE design question of the last two years. There's a scoring trap in the first 60 seconds, and most candidates jump to fixes and fail. Here's the diagnostic tree that wins it.

Text-to-SQL's nightmare isn't syntax errors, it's the query that runs, returns a confident number, and is silently wrong. The defense is a semantic layer plus transparency UX. Here's the full design.

Citations are the trust interface of enterprise RAG, and the easiest thing to fake badly. Covers inline citation generation, span verification, and the verification-rate metric that interviewers actually want.

Calibrated abstention is a system property, not a prompt line. Learn the four abstention signals, the precision-at-coverage framing, and why 'just tell it to say I don't know' fails the interview.

The staff-level version of 'evals first.' Four layers, each with a job, a cost, and a blind spot, plus judge calibration numbers and the data flywheel that ties production failures back into CI.