RAG & AI Agent Interview Questions

The warm-up that filters out tutorial-level candidates. Interviewers aren't grading the diagram, they're grading whether you know the three places real pipelines fail and how you'd see it happening.

Everyone says '500 tokens with overlap.' The candidates who get hired explain why chunk boundaries are a recall problem, name three strategies, and describe the experiment that picks between them.

Pure vector search fails on the exact queries enterprise users ask most: IDs, error codes, product names. Here's the failure-mode framing that turns a definition question into a design answer.

Arguing against agents reads senior; it's an explicit rubric signal at AI labs. Learn the cost ladder from single call to autonomous loop, and the three conditions that actually justify an agent.

'Evals first' is the #1 hiring signal in FDE design rounds. This is the foundational version: what a 50-case golden set looks like, which metrics split retrieval from generation, and the launch gate that convinces interviewers.

A reported FDE design question that punishes one-size-fits-all answers. Each corpus has a different 'semantic unit': get the three designs, the parsing tools to name, and the eval that proves the split was worth it.

The retrieval design question with a precise rubric: candidates who can say what each stage rescues, and what it costs in latency, clear it. Includes the RRF detail and the latency budget interviewers ask for.

The modal FDE design question of the last two years. There's a scoring trap in the first 60 seconds, and most candidates jump to fixes and fail. Here's the diagnostic tree that wins it.

Three plausible fixes, one decision framework. Interviewers grade the ordering (cheapest-reversible first) and whether you can say what each option costs in data, time, and operational burden.

Sierra's signature design exercise. The grading hinges on three things most candidates underweight: tool API shape, the read/write trust boundary, and eval cases written before the agent. Walkthrough inside.

A Sierra-signature scenario where 'put it in the system prompt' scores zero. The winning answer is defense-in-depth with a measured catch-rate per layer, plus an honest number for what still gets through.

Perceived latency is a design surface, not an infrastructure detail. Covers progressive disclosure, SSE vs WebSockets, resumability, and the mid-stream failure question most candidates have never thought about.

Calibrated abstention is a system property, not a prompt line. Learn the four abstention signals, the precision-at-coverage framing, and why 'just tell it to say I don't know' fails the interview.

Citations are the trust interface of enterprise RAG, and the easiest thing to fake badly. Covers inline citation generation, span verification, and the verification-rate metric that interviewers actually want.

Text-to-SQL's nightmare isn't syntax errors, it's the query that runs, returns a confident number, and is silently wrong. The defense is a semantic layer plus transparency UX. Here's the full design.

An Anthropic-signature integration question. The hidden rubric: tools are a curated product surface, not a proxy for your OpenAPI spec, plus the auth model and confused-deputy trap most candidates miss.

The unglamorous design question that exposes who has run LLMs in production. Batch APIs at 50% off, idempotency keys, poison records, spend kill-switches: the full checklist interviewers listen for.

A Glean-signature design question where deletions, not ingestion, are the trap. Covers the connector framework, change detection per source class, tombstones, and the freshness SLO customers actually sign.

The staff-level version of 'evals first.' Four layers, each with a job, a cost, and a blind spot, plus judge calibration numbers and the data flywheel that ties production failures back into CI.

Glean's signature design question. The two traps: post-filtering that drains your top-k, and the leak through generated answers. The winning architecture filters at retrieval, and here is how, at scale.

Memory is a retrieval problem plus a write-policy problem plus a privacy problem. Most candidates only design the first. Covers memory taxonomies, the curation pipeline, and the leak vectors that fail the question.

The question that separates demo-builders from production engineers. Idempotency keys, the saga pattern, dry-run modes, approval tiers: the distributed-systems toolkit applied to agents, with the trap most answers miss.

Harvey's signature design question. The twist: the architecture is the easy half. The graded half is designing for reviewer trust, where one hallucinated clause costs more than a hundred correct ones earn.

Serving 50 enterprise customers off one platform means four leak surfaces most designs miss, including the semantic cache and the eval set. A tier-by-tier isolation model with the noisy-neighbor math.

The most FDE question in the bank: every real deployment hits an unchangeable legacy system. The answer is an architectural principle, concentrate brittleness in one owned layer, plus what to do when the GUI is the only API.

The POC-to-production gap is the FDE's home turf. What actually breaks at each order of magnitude (index, ingestion, latency, cost) and the dependency math that makes 99.9% harder than it sounds.

Standard A/B machinery assumes a crisp conversion event; agents give you fuzzy, delayed, gameable outcomes. The metric hierarchy, the reopen-window trick, and the sample-size reality check that win this question.

Multi-agent is the most over-prescribed architecture in the industry. A staff-level answer is a decision framework driven by task structure and context limits, with the orchestration failure modes named before the interviewer asks.

Untrusted content plus write-capable tools is the lethal combination. Prompt-level defenses alone fail this question; the winning answer is an architecture where injection succeeding still can't move money.

Launch is the starting line. The staff-level capstone: how production traffic becomes eval cases, prompts ship like code, and a model deprecation becomes a Tuesday instead of an incident.

A real interview question with a trap in its framing: it offers you two expensive options and hides the cheap one. The escalation ladder from feature-based reranking to two-tower fine-tuning, with the cold-start and position-bias answers ready.

A deceptively short interview question that's really a trap: embeddings cannot do exact match, and the interviewer knows it. What to actually turn on (payload full-text indexes, sparse vectors, metadata filters) in the vector DBs you already run.

The verbatim Salesforce Agentforce FDE question. Strong answers name Topics, Actions, and grounding sources specifically, and refuse the word 'replace' without saying no. The component-by-component build with the staged-autonomy rollout.

The verbatim ElevenLabs case-study prompt. It looks like a UI question; it's a workflow-modeling question, and the candidates who pass start by asking what a 'unit of work' is in dubbing. The entity model, the review-state machine, and the v1 cut.

Reciting the four definitions is table stakes; the differentiator is knowing which two need ground truth, what each dimension tells you to fix, and where the judge-model scores quietly lie. The 2x2 that turns RAGAS into a debugging tool.

Asked constantly in GenAI loops, and most answers are marketing-page recitals. The version that scores: classify by what each actually is, data framework, integration toolkit, graph runtime, then give the selection criteria that outlive all three names.

The staff-level reliability question for agent platforms. Refunds went out, emails sent, tickets closed, and now you have to triage at fleet scale without double-refunding. The recovery model interviewers actually score.

The classic 'it regressed and the prompt is untouched' triage. The answer that scores walks a decisional tree from retrieval to generation and names the silent culprit most candidates miss: the embedding stack changed under you.

Most candidates reach for a smaller model first. The staff answer builds an explicit per-stage latency budget, parallelizes independent tool calls, and spends the time budget where the user actually feels it.

Golden sets catch what you anticipated. The harder problem is the live traffic you never labeled. The answer that scores layers cheap proxy signals against a small anchored golden set and knows what each can and cannot tell you.

This is the lethal trifecta in one sentence, and most candidates reach for a better system prompt. The answer that scores breaks the trifecta architecturally, because no amount of prompting stops indirect injection once all three capabilities are present.

The data-isolation answer everyone gives misses the harder surface: live OAuth tokens and API keys the agent uses to act in third-party systems. The model that keeps one tenant's credentials out of another's agent loop, and out of the LLM context entirely.

Hard budgets stop the bleeding; they don't fix the cause. The real fix is progress tracking plus semantic loop detection plus a planning layer that commits to a plan instead of re-deciding every step, validated on a fixed trajectory set offline before it ships.

A multi-hop question over a property graph is a query-generation problem, not a similarity-search one. The staff-level move is grounding generation in the live schema and closing an auto-healing loop on the DB's own error messages.

You cannot unit-test a stochastic two-agent pipeline with a golden output. You evaluate it the way you stress a distributed system: thousands of simulated runs, controlled fault injection, and a judge calibrated against humans.

The hard parts are not the four roles. They are the communication substrate, a termination condition that is not just a retry cap, and stopping the Coder from forgetting the ticket after five repair cycles.

Self-correcting RAG is easy to demo and easy to ruin. The discipline is a gate that grades evidence before drafting, a termination rule that is not just a loop cap, and a metric that proves correction adds facts rather than fluff.

DSPy's pitch is to stop hand-writing prompts and instead declare what the pipeline does, define a metric, and let a compiler search for the prompts and examples. The catch is that the compiler is only as good as the metric, and an LLM-judge metric drags its own biases in.

Slides are the worst RAG corpus: text in shapes, meaning in charts, the message in the image. A query like 'the blue bar chart with Q3 revenue' fails if you embed only the extracted text, and citing the wrong slide number kills trust instantly.

The hard part is not the retrieval pipeline, it is the boundary. Embeddings and the index live inside the VPC; the only question is whether the generation model is an in-VPC open model or an external API with a zero-retention DPA, and what crosses that line.

A 500-page doc is well over 200k tokens, and even if it fit, stuffing it in would rot the context. The real design is a working set in context plus an external memory the agent retrieves from and compacts into as it goes.

Deep-research agents run unsupervised for twenty minutes and their report goes straight to an executive. The failure mode is not a crash, it is confident citations that do not support the claims. Most candidates design the retrieval and skip the verification pass, the run budget, and the freshness story.

An agent loop optimizes whatever the verifier rewards, so a weak verifier gets gamed: tests edited, outputs hardcoded, inputs memorized. The fix is a verifier the agent cannot author or edit, run on evidence in a clean environment, with held-out and adversarial checks that track intent.

Every eval question in the prep canon is about answer quality. This agent does not produce answers, it moves freight. You cannot score a reroute for faithfulness, and the candidate who tries has not understood what they built.

The provider bill tells you the total and nothing else. This is a forensics question, and the answer is almost never a pricey model. It is that the transcript grows every turn, so a forty step run costs six times what everyone estimated.

Everyone reaches for a bigger machine. The two mechanisms that actually cause this both scale with corpus size, both hit the tail before the median, and one of them is the permission filter your security review insisted on.

Vectors from two versions of an embedding model are not comparable, and nothing in the stack will tell you so. The cosine between the same document's old and new vector is roughly zero, which is why a half-finished migration returns confident nonsense at full speed.

A wrong answer inside one conversation dies with the conversation. A wrong answer written to memory becomes a retrievable fact that conditions every later run, and the agent then derives new conclusions from it. Deleting it is the part nobody designed for.

Everyone sizes the pool on average run length, and average is the one statistic an agent fleet does not have. Simulated, the p99 sat at 160 seconds whether the system was half loaded or nearly saturated, which means capacity was never the problem.

Everyone computes the stale fraction as a share of the corpus, gets 0.03%, and moves on. Weighted by queries rather than documents it is 1.4%, roughly fifty times larger, because the documents that changed during the rebuild are the ones people are asking about.

The rate limiter is working correctly and that is the problem. It counts requests, and one hundred-thousand-token request occupies the KV cache of fifty short ones, so admitting both tenants equally hands one of them eighty percent of the machine.

A query that errors is a bug. A query that runs and returns a plausible wrong number is a product teaching a finance team, one figure at a time, not to trust it. The diagnosis is a classification exercise, and the fix is almost never a better prompt.

Before you touch a model, check whether the two numbers measure the same thing. Field accuracy of 96% across twelve fields is 61% of documents fully correct, which means half of the drop may be arithmetic. The other half is the documents the pilot never saw.

Nobody deployed anything is true of your repository and false of the system. The model provider, a tool behind an API, the knowledge base and the traffic mix all change without a commit. Segmentation comes before theory: a broken intent and a mix shift produce the same headline and need different fixes.

A nightly suite of fifty simulated conversations per intent catches a collapsed tool with certainty and a five-point drift almost never. Knowing which is which, and sizing each layer for what it can actually see, is the whole design. The grader checks outcomes, not transcripts, or the agent learns to sound resolved.

Three correct queries scored 0 of 3 on exact match and 3 of 3 on execution. The golden set is question, reference SQL and result set, stratified by the ways queries go wrong, with abstention as its own column. A hundred cases gives plus or minus eight points; four hundred gives four.

The eval set says recall fell 2.3 points, which is inside the noise you would expect from a re-index. Support says the product got worse. Both are right, and the reason is that one document class lost 49 points of recall while everything else improved slightly.

Recall@10 is 0.94 and climbing, and answer accuracy is going down. The two facts are consistent, and reconciling them is the whole question: every passage you add to raise recall also adds a distractor the generator has to survive.

The document count matches, nothing threw, and a specific set of documents is now unreachable by any query. That combination narrows the cause fast, and the two survivors are an approximate-search parameter that a rebuild reset and a metadata filter applied in the wrong order.

A reranker cannot promote a document the retriever never returned, which makes candidate-set recall a hard ceiling on the entire exercise. Compute the ceiling and the disappointment becomes arithmetic: three points of headroom were available, twelve were sitting upstream, and the latency was an implementation detail.

The citation is real and semantically relevant, yet the advice belongs to different equipment. Resolve applicability before ranking and preserve the version decision through generation and evaluation.

Extraction accuracy cannot reveal a document that never arrived. Reconcile expected evidence with received and parsed artifacts, and keep completeness separate from any claim decision.

Cheap typed judgments make it affordable to check every input, tool call and output. The design question is which checks are semantic, which are deterministic, and how you prove the layer catches what it claims to.

The pitch is a model that runs the loop. The design that survives production is a loop that code runs, with the model answering three narrow questions inside it: which tool, is this done, does this need a stronger model.

Most answers draw an API gateway with OAuth and stop. The bank's problem is the hop from agent to tool: whose identity reaches the CRM, which of 40 tools this agent may call, what stops a runaway loop, and what happens when the gateway itself is down.

70 percent is an average over tasks and runs, and it hides the number the customer cares about: does the same request get the same right outcome every time? Two agents with identical averages can differ from shippable to unusable.

Single-session evals pass while users complain, because memory fails across sessions and at three separate stages. The answer is a planted-fact harness that names which stage lost the fact, plus the two tests teams skip: forgetting on request and leaking across users.

The customer wants the agent to practice before it can write to production. Design the task world, reset semantics, tool boundary and verifier, then explain when its scores justify training or release.

A frozen policy jumps from 70% to 90% after parallelizing its evaluation. Reconstruct the denominator, separate runner faults from policy behavior, and design the experiment that could invalidate the gain.

Trace an old fact from durable storage into the next model call. Separate conversation history, checkpoints, summaries, and verified tool outcomes.