THE RESEARCH LIBRARY · PAPERS FOR FDES
AI research, explained for the work.
150 papers on building, evaluating and serving AI. Understand the mechanism, examine the evidence, and see what changes when you put the idea into production.
The guided starting pathProduction foundations → retrieval → agents → serving
STAGE 01
Understand production work
Describe a customer workflow, its dependencies, what success means and who owns it after launch.
STAGE 02
Understand the model and grounding
Explain the architecture and build a basic question-answering app over documents.
STAGE 03
Measure retrieval and answers
Build a small eval set and test evidence placement, retrieval quality and faithfulness.
STAGE 04
Add controlled actions
Give an agent a few well-designed tools, clear stopping rules and useful feedback.
STAGE 05
Evaluate reliability
Write a rubric, check the judge against humans, and run repeated and perturbed tests.
STAGE 06
Understand customization and serving
Say when adaptation could help, and how context length and concurrency set the memory bill.
Find your next paper
Browse by topic, or search for an idea.
150 papers
Foundations
The ideas every system you deploy is built on.Hidden Technical Debt in ML Systems
Google's 2015 field guide to why ML systems are cheap to build and expensive to keep alive, and where the cost hides when nobody is looking.
Attention Is All You Need
The 2017 Google paper that introduced the Transformer: attention and position information in place of recurrent and convolutional layers.
GPT-3: Few-Shot Learners
The 2020 OpenAI paper that showed a big enough model can learn a task from a few examples in its prompt, no fine-tuning, and made prompting a discipline.
InstructGPT
The 2022 OpenAI paper that turned GPT-3 into an assistant with human feedback, and showed a 1.3B model people preferred to the 175B original.
Chain-of-Thought Prompting
The 2022 Google paper that showed examples with their working shown, not just their answers, can triple a large model's accuracy on multi-step problems.
RAG
The 2020 paper that named RAG: a generator paired with a searchable index, so a model's knowledge can be checked, cited and updated without retraining.
Sentence-BERT
The 2019 paper that made BERT usable for semantic search: one vector per sentence, compared by cosine, turning a 65-hour job into 5 seconds.
Lost in the Middle
The 2023 Stanford paper that showed models use evidence at the start and end of a long prompt far better than in the middle, sometimes worse than no evidence at all.
BERT
Select 15% of tokens for prediction, train a Transformer encoder with context from both sides, then fine-tune for each task. BERT set records on eleven language benchmarks.
Mamba
Make a state space model's update depend on the current token, so it can choose what to remember, and run it as a fused GPU scan. Transformer-level quality to 2.8B parameters with no KV cache.
DeepSeek-V3
A 671B mixture-of-experts model, 37B active per token, trained in FP8 for 2.788M H800 GPU-hours, with a compressed attention cache and bias-balanced experts. Read the cost line closely.
Building and evaluating LLM apps
Agents, pipelines, structured output and how to grade them.ReAct
The 2022 paper that interleaves reasoning with tool use: think, act, observe, then revise the plan. A foundation for agents that adapt to what their tools return.
DSPy
The 2023 Stanford paper that treats an LLM pipeline as a program: declare what each call does, give a metric and a few examples, and let a compiler write the prompts.
Grammar-Constrained Decoding
The 2023 EPFL paper showing that masking a model's tokens with a grammar guarantees the output's shape, lifts accuracy without training, and has a trap: valid but empty.
LLM-as-a-Judge
The 2023 paper that made using a strong model to grade other models respectable, by measuring how often it agrees with people and exactly how it is biased.
Ragas
The 2023 paper that split RAG quality into three checks you can run without reference answers: is the context focused, is the answer grounded in it, does it address the question.
Retrieval, documents and memory
Getting the right context in front of the model, and keeping it.RAG or Long-Context LLMs?
The 2024 Google DeepMind study that found long context beats RAG on quality, RAG wins on cost, and a model that says when it cannot answer can route between them.
Late Chunking
The 2024 Jina AI paper that fixes chunks losing their context: run the whole document through the embedding model first, then cut the token vectors into chunks.
GraphRAG
Microsoft's 2024 paper on answering questions about a whole corpus, not one passage: build an entity graph, summarize its communities, then map-reduce over the summaries.
ColPali
The 2024 paper that skips PDF parsing: embed each page image with a vision-language model into many small vectors, and let tables, charts and layout count in search.
Qwen3-VL Embedding and Reranker
Alibaba's 2026 report on one embedding model and one reranker for text, images, document pages and video, and the trade-offs of dimensions, precision and reranking.
RAGPerf
The 2026 UIUC and IBM benchmark that profiles a RAG system stage by stage and finds where the time, memory and money really go, rarely where teams look.
Agentic Context Engineering (ACE)
The 2025 Stanford and SambaNova paper that improves agents by growing a detailed playbook of lessons in their context, with small edits instead of rewrites, and no weight updates.
Recursive Language Models
The 2025 MIT paper that stops stuffing long inputs into the context window: put the input in a variable, let the model write code to slice it and call itself on the pieces.
Mem0
The 2025 paper behind the Mem0 memory layer: extract facts from conversations, keep them current with add, update and delete, and trade a little accuracy for a lot of speed.
Self-RAG
Train a model to emit its own control tokens: whether to retrieve, whether a passage is relevant, whether each sentence is supported. Beam search on those scores; strong results, with fine print.
RAPTOR
Cluster a document's chunks, summarize each cluster, repeat to a root, and search every level at once: broad questions get summaries, narrow ones get chunks. Gains are modest.
RAFT
Fine-tune on questions paired with the right document plus distractors, with answers that quote the right one, so the model learns to read past bad retrieval. Strong on APIs; read the tables closely.
Sufficient Context
Google splits RAG errors by whether the retrieved context could answer the question at all. Models answer wrongly rather than abstain when it cannot, and right 35-62% of the time anyway.
HippoRAG 2
Graph retrieval that doesn't lose to plain embeddings on simple questions: LLM-extracted triples plus passage nodes, searched with Personalized PageRank. Real multi-hop gains, at 8x the indexing time.
ReasonIR
Meta trains an 8B retriever on LLM-written hard queries and hard negatives. It leads BRIGHT, helps RAG on GPQA, and shows why an LLM reranker that scores 0-5 needs a tie-breaker.
Nemotron ColEmbed V2
NVIDIA's open late-interaction models retrieve PDF pages from their images and top ViDoRe V3. The paper also prices them: about 6 TB of vectors per million pages for the 8B model, and how to cut it.
Agents, evaluation and security
What agents do in the real world, how they fail, and how they are attacked.SWE-agent
The 2024 Princeton paper that showed an agent's tools matter as much as its model: a purpose-built interface for searching, viewing and editing code beat a raw shell by seven points.
AI Agents That Matter
The 2024 Princeton critique of agent benchmarks: report cost next to accuracy, compare against simple baselines, keep a real held-out set, and stop mistaking leaderboards for procurement advice.
Why Multi-Agent Systems Fail
Berkeley read 1,642 traces from seven multi-agent frameworks and named 14 ways they fail. Most failures trace back to how the agents are organized and checked, not to the model inside them.
Measuring Agents in Production
Twenty deployment teams and 86 deployed systems describe how agents actually ship: frontier models prompted by hand, fixed workflows, short leashes and humans checking the output.
Reflexion
The 2023 paper that taught agents to learn from failure without training: after each failed attempt, the model writes itself a lesson and reads it on the next try.
SWE-bench
The benchmark that moved coding evaluation from toy functions to real repositories: 2,294 GitHub issues, each graded by the project's own tests. In 2023 the best model fixed under 2%.
τ-Knowledge
A support-agent benchmark built like a real deployment: 698 interlinked bank documents, tools you can only find by reading them, and a simulated customer. The best agent passed a quarter of tasks.
AI Agent Reliability
Princeton borrowed reliability engineering from aviation and nuclear power, turned it into twelve metrics for AI agents, and found that two years of accuracy gains bought only small reliability gains.
SkillsBench
Agent Skills, folders of instructions and scripts an agent loads on demand, lifted pass rates from 34% to 51% across 18 agent setups. Focused Skills helped most; self-written ones hurt.
ττ-bench (hyper-tau-bench)
A benchmark that turns an FDE engagement into the task: messy records, a client to interview, a flaky API and a serving budget. AI-built agents scored under a third of an expert's.
AgentDojo
The standard testbed for prompt injection against tool-using agents: 97 tasks, 629 attack cases, and a finding that the most capable models were often the easiest to hijack.
Indirect Prompt Injection Arena
464 red-teamers made 272,000 attempts to hijack 13 frontier agents and make them hide it. Every model fell at least sometimes, and attacks that beat the strongest model worked almost everywhere.
Customization and reasoning
Changing a model's behavior, and paying for it to think longer.LoRA
The 2021 Microsoft paper behind most fine-tuning today: freeze the model, learn each layer's change as two thin matrices, and match full fine-tuning of GPT-3 with 10,000× fewer trained parameters.
QLoRA
LoRA on top of a 4-bit base model: QLoRA fit a 65B fine-tune onto one 48 GB GPU with no measured loss, and showed that 9,000 good examples beat 450,000 mediocre ones.
DPO
The 2023 paper that fits preferences with a classification loss over chosen and rejected answers, removing the explicit reward-model fitting and reinforcement-learning loop used in PPO-based RLHF.
Tülu 3
AI2's fully open post-training recipe: every dataset, script and evaluation used to turn Llama 3.1 base models into assistants that matched closed models, plus RL with verifiable rewards.
DeepSeek-R1
DeepSeek combined reinforcement learning with cold-start examples to reach o1-level reasoning. R1-Zero separately showed reasoning emerging from RL without supervised fine-tuning first.
Scaling Test-Time Compute
When does thinking longer beat a bigger model? Spending inference compute by question difficulty used up to 4× less compute and beat a 14× larger model on questions within reach.
Serving, latency and cost
What makes inference fast, cheap and able to hold many users.FlashAttention
The 2022 Stanford kernel that made attention fast by counting memory trips, not math: exact attention computed in on-chip tiles, several times faster, with memory linear in sequence length.
PagedAttention and vLLM
The Berkeley paper behind vLLM: manage the KV cache like operating-system virtual memory, in small pages allocated on demand, and serve 2 to 4 times more requests on the same GPU.
AWQ
MIT's 4-bit weight quantization that asks the activations which weights matter, protects about 1% of channels by scaling instead of keeping them in 16-bit, and runs over 3× faster than FP16 on GPUs.
SGLang
A serving system for programs that call a model many times: it caches KV across requests in a radix tree, decodes fixed JSON text in one step, and ran agent workloads up to 6.4× faster.
DistServe
Prefill is compute-bound and decode memory-bound, so sharing one GPU makes each slow the other. DistServe splits them onto separate GPUs and served up to 7.4× more requests within latency targets.
LMCache
Treat the KV cache as data that outlives the request: store it beyond the GPU, move it in big chunks, and reuse it across queries and engines. With vLLM, up to 15× throughput.
RouteLLM
Train a small router on human preference data to send each query to a strong, costly model or a weak, cheap one. On MT Bench it reached 95% of GPT-4's score using GPT-4 for 13% of queries.
Sarathi-Serve (chunked prefill)
When a long prompt arrives, most servers pause all other output to process it. Sarathi-Serve slices prompts into chunks riding along with ongoing decodes: up to 3.7x more load within a latency target.
Parrot
LLM APIs see single requests, not the agent workflow behind them. Parrot has apps mark prompt inputs and outputs as variables, so the service chains, batches and shares work: up to 11.7x faster.
ThunderAgent
Agents spend much time in tool calls, and LLM servers evict their KV cache while they wait. ThunderAgent schedules whole agent programs: 1.5-3.6x serving throughput, up to 3.9x faster RL rollouts.
FlashInfer
The attention library inside SGLang, vLLM and MLC-Engine: every KV layout as one block-sparse format, custom variants compiled on demand, uneven batches balanced. 29-69% lower latency in SGLang.
FlashAttention-4
Blackwell doubled tensor-core speed but not the exponential unit or shared memory, so softmax and data movement now limit attention. FlashAttention-4 rebalances: 1,613 TFLOPS on B200, 1.3x cuDNN.
Speculative decoding
A small model guesses the next few tokens and the big model checks them all in one pass. With the right acceptance rule the output is exactly what the big model alone would produce, only 2-3x faster.
EAGLE-3
EAGLE-3's draft head reads the target model's own hidden states from three layers and is trained on its own guesses. Lossless decoding runs 3-6.5x faster, and it still helps at batch 64 in SGLang.
Beyond text
Vision, speech and the tabular baseline you should always try first.CLIP
Train image and text encoders on 400 million image-caption pairs. Classify images using class names in prompts, without training a separate classifier on labels for each downstream task.
Whisper
Train one transformer on 680,000 hours of web audio and its imperfect transcripts. With no benchmark fine-tuning, it made about half the errors of a LibriSpeech-trained model on other test sets.
XGBoost
Gradient boosted trees with a regularized objective, built-in handling of missing values and serious systems engineering. Still the first model to try on a customer's tables.
Harness design and boundaries
The engineering around the model: what it may touch, what it is told, and how you tell a change helped.Agent harness design decisions
A source-reading survey of 70 public agent projects that names the five decisions every harness makes. Its vocabulary is useful; several of its headline statistics cannot be reproduced.
Harness design for coding agents
Hold the agent loop fixed, vary one harness component at a time, and measure. Context management, planning and tool choice each helped some models and hurt or merely cost others.
Meta-Harness
Let a coding agent rewrite the code around a model, reading every earlier attempt's code, scores and traces. It beat hand-built harnesses; its biggest lesson is that raw traces beat summaries.
Evaluating AGENTS.md
Coding agents follow the instructions in AGENTS.md files closely. On 438 real tasks, those files did not make them solve more, and every agent spent 8 to 50% more per task.
Auditing agent harness safety
An agent can hand back the right answer after reading records it should not, leaking data between agents, or calling tools it was never given. This benchmark audits the whole run, not the reply.
CaMeL: prompt injection by design
Stop asking the model to resist injected instructions. Have it write a plan before it sees any untrusted data, then let an interpreter track where every value came from and enforce who may receive it.
AgentSpec
Write safety rules as trigger, check and enforce, and run them before every agent step. Hand-written rules stopped most risky actions in milliseconds; rules written by an LLM were patchier.
Prompt injection design patterns
Six ways to build agents that resist prompt injection by limiting what untrusted text can touch, worked through ten real applications. The trade is plain: every pattern gives up some generality.
Actions, tools and orchestration
How an agent expresses actions, runs independent work at once, and divides a task among roles.CodeAct: code as the action format
Let an agent act by writing Python instead of one JSON tool call at a time. On multi-tool tasks the best model solved 74% against 52% with JSON; on single calls JSON was as good or better.
Toolformer
A model writes candidate API calls into ordinary text, keeps those whose results make the next words easier to predict, and trains on them. A 6.7B model learned when to call a calculator.
PAL: program-aided language models
Have the model translate a word problem into Python and let the interpreter compute. Codex beat a 540B model on grade-school math, and led by 38 points once the numbers got large.
ReWOO: plan before you observe
Write the whole tool plan up front with placeholders for results, run the tools, answer once. Tokens fell about fivefold versus ReAct, but the plan cannot adapt to what tools return.
LLMCompiler: parallel function calls
Plan tool calls as a dependency graph, then run every call whose inputs are ready at the same time. Against ReAct this cut latency up to 3.7× and cost up to 6.7×, with similar accuracy.
Agentless
Fix GitHub issues with a fixed localization, repair and validation pipeline. Agentless resolved 32% of SWE-bench Lite for $0.70 per issue, leading the open-source systems in the paper's comparison.
OpenHands
An open platform for agents that work like developers: one event stream of actions and observations, a sandboxed runtime with shell, Python and browser, and agents that hand off to each other.
AutoGen
Microsoft's framework for building LLM apps as conversations between agents that write code, run it, ask humans and check each other, and what its own results say about when more agents help.
Magentic-One
Microsoft's five-agent team led by an Orchestrator that keeps a task ledger and a progress ledger, re-plans when stuck, and was scored with error bars that show what a leaderboard can't tell you.
AFlow
Treat an LLM workflow as code and let Monte Carlo tree search, with an LLM proposing one edit at a time, find a better one. It beat hand-built workflows on six benchmarks. Read the evaluation closely.
Planning, feedback and improvement
Search, critique and optimization loops, and the evidence that says whether they helped.GEPA
A prompt optimizer that reads its failures in plain language, rewrites one module's prompt at a time, and keeps every prompt that wins somewhere. It beat GRPO on five of six tasks with fewer rollouts.
TextGrad
Backpropagation with words: an LLM critiques each step of a pipeline, critiques flow backward through the graph, and an LLM rewrites prompts, code or answers. A PyTorch-style API, uneven evidence.
Tree of Thoughts
Let the model propose several next steps, judge which are still promising, and search with backtracking. On the Game of 24, GPT-4 went from 4% to 74%, at a cost the paper itemizes.
Self-Consistency
Sample many chain-of-thought answers and take the majority vote. A one-line change that added up to 17.9 points on GSM8K, and whose agreement rate doubles as a confidence score.
Self-Refine
One model drafts, critiques its own draft, and rewrites it, up to four times. Big gains on style and constraint tasks, none on math, and a headline average that mixes win rates with scores.
LLMs Cannot Self-Correct Reasoning Yet
Asked to review and fix their own reasoning with no outside signal, GPT-3.5, GPT-4 and Llama-2 got worse on every benchmark. The earlier gains came from answer keys and unequal compute.
The Prompt Report
A systematic survey of 1,565 prompting papers: a shared vocabulary, 58 text techniques in six families, a small benchmark, and a 47-step case study that shows how fragile hand prompt engineering is.
Memory and context
What persists across turns and tasks, and how long work fits inside a context window.MemGPT
Treat the context window like RAM: a small editable working memory in the prompt, everything else in searchable storage, and the model paging data in and out itself through function calls.
Generative Agents
Twenty-five simulated townspeople with a memory stream, reflection and plans. The retrieval score behind them, recency plus importance plus relevance, became the template for agent memory.
Voyager
A GPT-4 agent that teaches itself Minecraft by proposing its own next task, writing code, checking the result, and saving what works as a reusable skill. Memory here is a library of verified programs.
Agent Workflow Memory
Web agents that turn successful runs into reusable, abstracted workflows and improve as they work: WebArena success from 23.5% to 35.5% with no human-written workflows. Check what is relative.
A-MEM
Memory as Zettelkasten: each interaction becomes a note with LLM-written keywords and context, linked to related notes; old notes are rewritten as new ones arrive. Big temporal gains, mixed elsewhere.
MemOS
A memory operating system for LLM apps: text, KV-cache and parameter memories in one governed unit, the MemCube, with scheduling, lifecycle and access control. Strong benchmarks, run by the vendor.
LongMemEval
500 hand-written questions buried in user-assistant chat histories up to 1.5M tokens, testing five memory abilities. Long-context models lost 30-55% of their accuracy; commercial memory did worse.
Context-Folding
Give an agent two tools, branch and return, so it can do token-heavy work in a side thread and keep only the result. Trained with RL, a 32K-context agent matched a 327K one. Mind which baseline.
LLMLingua-2
Compress prompts 2-5x with a small encoder that decides, word by word, what to keep, trained on GPT-4's deletions. Fast and faithful in domain; behind question-aware compression out of it.
Prompt Cache
Precompute the attention states of reusable prompt pieces once and splice them into later prompts: 5 to 10 times faster first tokens, with an approximation that can cost accuracy.
Context Language Models
Give the model write access to its own context, as a file it edits with Bash, and let it invent its own compaction. Beats fixed harnesses at lower compute in most tests, with caveats in the setup.
U-Fold
Keep the full history, and on every user message rebuild a small working context: a summary with a to-do list plus verbatim lines from past tool output. Small gains over ReAct, big ones under noise.
ContextBench (coding agents)
1,136 real issues with expert-traced gold code, so you can score what a coding agent read, not only whether its patch passed. The retrieval scores did not rank agents the way solve rates did.
Lost in Multi-Turn Conversation
Across 15 models and six tasks, splitting instructions across turns lowered scores by 39% on average. Early answers and persistent wrong assumptions made conversations less reliable.
ACON
Microsoft learns a context-compression prompt from the agent's own failures, then distills it into a small model. Peak tokens fell 26% to 54%, though tasks took more steps and more time.
Zep
Zep stores agent memory as a knowledge graph whose facts record when they were true and when they were learned. On LongMemEval it beat full context by 8 to 11 points with 1.6K tokens instead of 115K.
Hindsight
Hindsight keeps an agent's facts, experiences, opinions and entity summaries in separate networks. A 20B open model went from 39.0% to 83.6% on LongMemEval, though baselines came from other reports.
EverMemOS
EverMemOS turns conversation into episodes, facts and expiring foresight, clusters them into scenes, and rebuilds just enough context. 93.05% on LoCoMo, at real build cost.
Systems classics for agents
Older rules about failure, authority and latency that every agent with real actions inherits.End-to-End Arguments
If only the applications at the ends can verify a function completely, building it into lower layers buys performance at most, never correctness. The 1984 rule behind every agent action.
Sagas
Split a long transaction into steps that each commit on their own, each with a compensating step that undoes it in business terms. The 1987 pattern behind multi-step agent actions.
Protection of Information
The 1975 tutorial that named least privilege, fail-safe defaults and complete mediation. Eight design principles written for time-sharing systems that read like a checklist for giving an agent tools.
A Note on Distributed Computing
Remote calls are not local calls with more latency: partial failure and concurrency cannot be hidden, so they belong in the interface. Sun's 1994 warning applies to every tool an agent calls.
The Tail at Scale
When a request waits on many servers, rare slow responses become the common case. Google's ways to live with latency variability: hedged and tied requests, micro-partitions, good-enough answers.
Agent runtimes and long-running work
Durable state, interruption, recovery and the execution services that keep long agent jobs alive.OpenHands Software Agent SDK
OpenHands rebuilt its agent core around one event log and immutable configuration. In a 15-day side-by-side rollout, infrastructure-caused failures fell 61%.
AgentScope 1.0
Alibaba's agent framework makes interruption, parallel tool calls, tool groups and saved state first-class. A design reference with no measurements, so test its claims yourself.
Magentic-UI
Microsoft's open-source agent interface puts the user in the team: co-planning, take-over, action approvals, saved plans. A simulated helper lifted GAIA accuracy from 30.3% to 51.9%.
TRACE: corrections to runtime checks
Remembering a correction is not obeying it. TRACE turns a user's repeated corrections into rules with checks that block a coding agent until it complies.
Agent Lightning v1.0
Train an agent through the same harness it runs in at deployment. Microsoft's 3,500-line control plane, four traps in turning API calls into training data, and a 41.8% to 56.4% SWE-bench gain.
SWE-RL
Meta trained Llama 3.3 70B with RL on 273,000 merged pull requests, rewarding patches for looking like the real fix. 41.0% on SWE-bench Verified, after sampling 500 patches per issue.
R2E-Gym
8,135 executable coding tasks built from commits instead of issues, and a finding about picking patches: tests and a learned judge each plateau near 43%, while combined they reach 51%.
Scaling Agent Systems
260 controlled runs, five architectures, six agentic benchmarks: adding agents ranged from +80.8% to -70%, and the one robust finding is that strong single agents gain little.
Kimi K2.5 and Agent Swarm
Moonshot trained an orchestrator to spawn and schedule parallel sub-agents. WideSearch time to a target score fell 3 to 4.5 times; total compute for the swarm is not reported.
GLM-5
Zhipu's 744B open model report is most useful for its agent RL plumbing: exact-token logging, stale-sample filters, cache-affine routing, 10,000 built coding environments and simple context folding.
Mobile-Agent-v3
Alibaba's GUI agent splits each step into Manager, Worker, Reflector and Notetaker roles, all one open model. AndroidWorld 73.3, though the gain over end to end also swaps 7B for 32B.
Choosing and sizing models
What a model's architecture, data and inference modes mean for the hardware and workload you have.Gemma 4
Google DeepMind's open-weight family from 2.3B to 31B, with thinking mode, audio and vision. The report's memory table is what to read first: which model fits your hardware, and with how many users.
Phi-4-Mini and Phi-4-Multimodal
Microsoft's 3.8B model plus a multimodal version that adds vision and speech as LoRAs on a frozen base, so text quality is untouched. Leads OpenASR; weak at answering spoken questions.
Qwen3
One open model with a switch: think hard or answer fast, plus a token budget in between. The report shows where thinking pays (math, code) and where it doesn't (chat, long-context retrieval).
Nemotron 3 Nano
NVIDIA's 31.6B model activates 3.2B per token and has only 6 attention layers among 52, so its KV cache is tiny. Fast long generation and 1M context; weaker on knowledge and multilingual tests.
gpt-oss
OpenAI's open-weight reasoning models: 120b fits one 80 GB GPU, 20b fits 16 GB, with three effort levels and tool use. The card also shows the weak spots: factual recall and holding a system prompt.
SmolLM2
Hugging Face trains a 1.7B model on 11T tokens and documents every data decision: LLM-labeled quality filters, late math and code, mixes changed mid-run. A recipe for curating any training set.
Fine-tuning, distillation and compression
Training a smaller or cheaper model for one job, and checking what it lost on the way.DoRA
Split each weight into per-column magnitude and direction; train magnitude directly, direction with LoRA. Same serving cost as LoRA, 1 to 4 points better, and far steadier at low rank.
LESS
Pick the 5% of a fine-tuning pool whose gradients point the same way as a few examples of the skill you want. It often beats training on everything, and the gradients come from a cheap LoRA warmup.
Magpie
Give an open chat model only the opening of a user turn and it writes a user's question. 300K filtered pairs from Llama-3 rival its official instruct model, at about $1 per 1,000 pairs.
Distilling Step-by-Step
Ask a big model why, not just what, and train a small model to predict both. A 770M model beats 540B PaLM's few-shot accuracy on ANLI with 80% of the data, and rationales cost nothing at serving time.
DistiLLM-2
Distill by raising the student's probability on the teacher's text and lowering it on its own bad text, with a different loss for each. Better students and faster speculative-decoding drafters.
Minitron
NVIDIA prunes a trained 15B model to 8B and 4B and distills it back with under 3% of the original data. The 8B rivals models trained on trillions of tokens, and the recipe comes as ten rules.
NVFP4 quantization-aware distillation
Quantizing a small model to 4-bit NVFP4 loses accuracy; retraining it on labels can lose more. Training it to match the original model's outputs got near-full accuracy back, even from random tokens.
When self-distillation hurts reasoning
A teacher that can see the answer never says "wait". Students trained on its confident reasoning get shorter and much worse on unseen math: AIME24 fell from 54.8 to 20.2.
KV cache and long-context infrastructure
Storing, moving, compressing and reusing attention state when you run the inference fleet.Mooncake
Kimi's serving stack schedules around the KV cache: separate prefill and decode pools, a cache pool in idle DRAM and SSD, early rejection under overload. About 75% more requests on real traffic.
CacheBlend
Prefix caching only helps the first retrieved chunk. CacheBlend reuses every chunk's cached KV and recomputes about 15% of tokens to restore cross-chunk attention: 2.2-3.3x faster first tokens in RAG.
CacheGen
A long document's cached KV can be gigabytes, and fetching it can take longer than recomputing. CacheGen encodes KV like video: 3.5-4.3x smaller than quantized KV, 3.2-3.7x faster first tokens.
KIVI
Quantize keys per channel and values per token, keep the newest 128 tokens in full precision, and a 2-bit KV cache holds up: 2.6x less memory, 4x larger batches, with no calibration or fine-tuning.
S-LoRA
One base model, thousands of customer fine-tunes: keep LoRA adapters in host memory, page them into the KV cache's GPU pool, and batch across adapters. 2,000 adapters on one GPU at 95% throughput.
Helix Parallelism
Past 8 GPUs, tensor parallelism stops speeding up attention: each extra GPU copies the KV cache. Helix splits the sequence for attention and all GPUs for the FFN, for up to 32x the users at 1M tokens.
TENT
Mooncake's team rebuilt its KV transfer engine after running it on thousands of GPUs: declare what to move, and 64 KB slices go to whichever links are fastest right now. 1.36x serving throughput.
Cost control, routing and recovery
Spending model calls where they change the answer, and getting serving capacity back after a crash.FrugalGPT
Ask a cheap model first and escalate only when a small scorer doubts its answer. On a news task this matched GPT-4 at 2% of the cost, because cheap models are sometimes right where GPT-4 is wrong.
BEST-Route
Routers overuse the big model because one small-model answer rarely matches it. BEST-Route also picks how many small-model answers to sample: 60% cheaper than GPT-4o, under 1% quality loss.
Dynamo fast recovery
Most serving crashes kill the engine process, not the GPU, yet restarts rebuild everything for minutes. Dynamo keeps weights resident and a parked backup engine ready: recovery in under 7 seconds.
“Most cited” uses dated Semantic Scholar counts shown on each paper. Citation counts describe research influence, not a recommendation for your system.