How to Train Tool-Using AI Agents: SFT, Distillation, RL and Evaluation
A practical guide to training AI agents from verified traces: loss masks, on-policy distillation, GRPO rewards, OpenEnv environments and asynchronous rollout checks, with a connected Python learning path.
BY MEI LIN · FDEINTERVIEWS EDITORIAL · UPDATED OCTOBER 29, 2026 · 9 MIN READ
PRACTICE THIS:Work through the agent post-training module ·Understand agent training loss masks ·Practice the asynchronous training diagnosis
Training a tool-using AI agent means improving the model's decisions across a sequence of actions and observations. Supervised fine-tuning can teach verified demonstrations. Distillation can supply teacher feedback, including on histories the student generates. Reinforcement learning can use independently checked episode outcomes. In every case, the training data must preserve what the actor knew, what it did and what the environment actually returned.
For an FDE, the first question is whether changing weights addresses the customer's failure. A model cannot learn its way around an absent lookup tool, a missing approval check or a controller that forgets the identity of an uncertain write.
Hugging Face's October 2026 curriculum for post-training agents provides a useful set of entry points into SFT, distillation, RL and training infrastructure. This guide develops the engineering decisions around those methods using an original support scenario. The connected Production AI Agents module adds exercises, a private source companion and a checkpoint review.
Start with one decision that went wrong
Consider this synthetic case. A support agent is allowed to request a replacement after approval. It sends the request, but the service's reply is lost. The agent says the request failed and sends another one with a new identifier. The customer receives two replacements.
There are several possible repairs. The controller should retain the original identity. The service needs the agreed idempotency contract. The agent needs a way to look up the original request. The final customer-facing status must come from evidence, not an invented receipt.
Only after these contracts work does a model training question become clear: given an unknown outcome and an available lookup tool, can the model reliably choose lookup before claiming success or issuing another effect?
That is a behavior you can demonstrate, observe and test. “Make the agent smarter” is not.
| Failure you observe | First experiment | What training might contribute |
|---|---|---|
| The required tool is unavailable | Repair tool registration and scope | Nothing until the capability exists |
| Required evidence is absent from context | Repair retrieval or state projection | Better use of evidence once available |
| Wrong action despite adequate evidence | Test a clearer prompt and constrained action set | A learned decision or format pattern |
| The model reports success without a receipt | Enforce the evidence boundary | Fewer unsupported proposals |
| A candidate exploits the evaluation rule | Repair and protect the verifier | Learning against a less misleading objective |
Our existing agent gym guide and harness course explain the surrounding execution system. Post-training belongs inside that system; it does not take over its authority.
Training signals answer different questions
A demonstration says, “Here is a behavior worth imitating.” Teacher feedback says, “Here is how another model evaluates this prefix or continuation.” An outcome reward says, “This complete attempt scored this way under the task's verifier.”
These signals can be combined, but combining them does not remove their weaknesses. A demonstration can contain an unsafe step. A teacher can confidently prefer the wrong action. A reward can measure an easy-to-exploit proxy.
Keep a baseline and name the smallest hypothesis before choosing a method. If ten carefully reviewed recovery examples solve a narrow formatting problem, that is a useful result. You do not need to complete every training method to justify the deployment.
SFT: teach actions without teaching the model to invent the environment
An agent trace contains several authors. The user supplies the request, the assistant produces tool calls, and services produce observations. Those observations often contain the evidence needed for the next decision.
In an assistant-only imitation objective, keep the observations in context while excluding their positions from direct prediction loss. Supervise the assistant's call names, arguments and responses under the intended template. Otherwise, a flattened transcript can train the model to reproduce service messages as well as assistant behavior.
This is the distinction behind agent training loss masks. A mask selects targets; it does not necessarily remove input context or prevent gradients from later predictions flowing through that context.
Inspect a decoded batch after templating, tokenization, truncation and collation. Can you still see the original request identity? Is the receipt present before the target response that relies on it? Are the selected tokens assistant-authored? Are padding and causal shifting handled correctly?
The official SFTTrainer documentation describes assistant-only loss and compatible chat-template generation spans. The presence of a flag in a configuration file is not enough. The actual labels are the evidence.
Our trace and SFT lab runs the data contract first. Its optional script also executes two real CPU optimizer steps using TRL 1.16.0 and a tiny random model. That proves the training path is connected. It does not produce a useful support agent, and the article does not claim that it does.
For a real checkpoint, retain its own tokenizer and chat template, pin its revision and repeat the mask audit. The teaching byte representation in the lab is deliberately easy to inspect; substituting it for a pretrained vocabulary would break the experiment.
Distillation: let the student expose the states it needs help with
A clean demonstration may never contain the history a student creates after its first mistake. More copies of that clean path do not automatically teach recovery. SFT can learn recovery when the dataset includes it; the practical problem is discovering and covering the relevant states.
On-policy distillation collects continuations from the student and applies teacher feedback to the prefixes it actually visits. The Generalized Knowledge Distillation paper studies this distribution mismatch in language models. Applying the method to an enterprise agent adds a requirement: preserve the relationship between each action and its actual tool result.
If the teacher rewrites a request into a lookup, the old request's observation cannot simply be attached to the new action. Execute the revised action in a valid environment state or keep the correction as a separately identified demonstration.
Teacher supervision also has formats. A written correction, a scalar quality rating and a next-token probability distribution are different data. A text-only endpoint does not expose a full probability distribution just because it is called a teacher. Direct token-distribution matching needs compatible vocabularies; the documented DistillationTrainer interface makes that constraint explicit.
The on-policy agent distillation concept develops the mechanism. The course's teacher-packet exercise asks learners to detect a subtler failure: a teacher recommending success because it saw private verifier state that the deployed actor would never receive.
RL: a score needs a trustworthy meaning
An outcome reward can allow several legitimate solutions without prescribing every intermediate action. It can also reward exactly the wrong thing if the verifier is weak. For the support task, “the final answer contains accepted” is easy to satisfy. “One approved replacement exists, its receipt was observed, and no forbidden action occurred” requires evidence from the environment.
Suppose four independent attempts at the same synthetic task receive rewards 1, 0, 1, 0. The mean is 0.5, so unscaled centered rewards are 0.5, -0.5, 0.5, -0.5. This illustrates a relative comparison, not the full GRPO algorithm. The optimizer also needs its probability-ratio, clipping, masking and regularization definitions.
The GRPO documentation exposes multiple scaling and loss variants. Record the configuration instead of assuming the acronym uniquely specifies the update. The original DeepSeekMath paper is the primary reference for the method's introduction.
If every reward is zero, this centered signal is zero. Inspect the episodes before adding workers. The initial policy may never reach a successful path, the task budget may truncate all recoveries, or the verifier may be broken. Each explanation calls for a different experiment.
Do not collapse missing evidence into failure. A known unsuccessful episode can receive a failure reward. A crashed verifier has not established the outcome. Preserve that incident and follow a documented collection policy, while retaining all planned cases in release reporting.
Environments make the experiment repeatable
An agent gym supplies resettable task state, permitted observations, actions, stopping rules and verification. You can use it to evaluate a fixed policy or collect demonstrations without training anything. RL adds an optimization procedure that consumes the resulting reward signal.
Reset must include consumed faults, receipts, counters, terminal state and other state that can affect the next attempt. Separate instances should not share writable task history. A request key that deduplicates one effect must not silently deduplicate every case in the batch.
Hugging Face's OpenEnv integration guide describes connecting environment instances and tool interactions to a trainer. The important application contract is independent of the wrapper: a returned observation must belong to the correct episode and action, and the verifier must retain authority over its own evidence.
For generated code or browser actions, an ordinary Python object is not a sandbox. Use a separately enforced execution boundary with restricted credentials, network access, resources and lifetime. Keep grader files and hidden task answers outside the actor's writable scope. Our local gym executes trusted scripted policies and makes no isolation claim for hostile code.
Scale after you can explain one update
Asynchronous training overlaps generation and optimization. It can keep hardware busy while some episodes wait for tools, but the rollout arriving now may have been generated by older weights.
Record the generating policy, task, environment, grader, template and intended comparison group. Where required by the objective, retain behavior probabilities and sampling settings. Detect missing group members and mixed reward rules before consuming the group.
The experimental AsyncGRPOTrainer documentation describes separate rollout and training processes, weight synchronization and staleness controls. A version stamp helps diagnose policy lag; it is not an off-policy correction algorithm. Nor is a cutoff of two or three updates universally safe.
Watch task mix as well as queue depth. Short easy episodes may finish first while difficult recovery cases wait. Count scheduled, completed, admitted and trained work by task family. More rollouts per hour can conceal less useful evidence for the behavior you are trying to improve.
The asynchronous training interview exercise tests this diagnosis. The rollout-admission lab supplies a local manifest gate and a separate release packet whose aggregate score improves while a mandatory safety condition fails.
Release an agent configuration, not just a weight file
A candidate checkpoint was evaluated with a particular tokenizer, prompt, tool set, context policy, controller and serving configuration. Changing any of those can change the behavior. Retain the complete tested configuration and the previous version needed for rollback.
Use held-out whole-task outcomes, boundary cases, unresolved results, operator workload and resource measurements. Keep forbidden actions separate from average success. A faster model that creates duplicate replacements is not an improvement under the support contract.
The public Production AI Agents overview explains the course format and links to the free starting lessons. The Premium post-training module continues the same Calder implementation through data auditing, teacher supervision, reward construction and checkpoint review. The core exercises need no paid model API; live model experiments require their own execution record.
If you are starting tomorrow, bring one failing episode to the review. Mark what the actor could see, the action it chose and the independent evidence of the result. That record will tell you more about the next useful training experiment than a choice of trainer alone.
Turn it into offers. Work the real questions and concepts this maps to:
FAQ
Model post-training changes weights using demonstrations, teacher feedback or rewards. Agent engineering also changes prompts, tools, memory and runtime controls without necessarily changing weights. Evaluate the complete agent configuration so you know which change improved the task.
Start with the failure and available evidence. SFT is a useful first experiment when verified demonstrations show the desired behavior. RL becomes a candidate when you can collect meaningful alternative trajectories and independently score their outcomes. Neither replaces tool permissions or a held-out evaluation.
They usually belong in the context because later actions depend on them. For an assistant-only imitation objective, exclude tool-authored positions from direct prediction loss while supervising assistant calls and responses. Verify this in the actual tokenized batch.
The core training-data, reward and rollout-admission exercises use Python's standard library. An optional CPU smoke script runs two real SFT steps on a tiny random model with pinned dependencies. It validates pipeline execution and makes no claim about trained-agent quality.
No. A gym supplies resettable tasks, interactions and verification. You can use it to evaluate a fixed policy or collect demonstrations without updating weights. Reinforcement learning additionally uses reward signals in an optimization procedure.
Discussion (4)
A useful first review artifact is the decoded training row with its selected targets highlighted. It gives the application engineer and training engineer something concrete to disagree about before an expensive run.
And keep one row after truncation, not only the original export. A receipt can exist in the source trace and still disappear before the target response reaches the trainer.
A missing verifier result deserves its own collection metric. Treating it as reward zero teaches from evidence that does not exist; dropping it from the release report hides an operating problem.
The tiny optimizer example is deliberately separate from the release packet. Passing a data-contract check and improving a customer workflow are different claims, and both need their own evidence.