ACON: Optimizing Context Compression for Long-horizon LLM Agents

An agent working through a long task keeps every action and tool output in its context. On AppWorld, a benchmark of everyday app tasks that average over 40 API calls each, that context grows until it is expensive to process and full of stale detail. The obvious fixes are crude: keep only the last few steps, or ask a model to summarize. Both lose things like a file path or an account ID that a later step needs.

ACON (Agent Context Optimization) compresses two things: the history, when it passes a token threshold, and individual observations, when a tool returns something long. A compressor LLM does both, following a natural-language guideline.

The contribution is how the guideline is improved. ACON runs the agent on training tasks with and without compression and collects the tasks that succeed only without it. For each, a strong model (o3) compares the full and compressed contexts and writes feedback about what was lost: a variable, a precondition, a fact. That feedback is aggregated into a rewritten guideline, and several candidates are scored on the same tasks. This is the utility step. A second compression step looks at tasks that succeed with compression and asks what information was actually used, producing a shorter guideline.

Finally, the expensive compressor is distilled: a small model (Qwen3-14B, Qwen3-8B or Phi-4) is fine-tuned on the teacher's compressions, cutting the compressor's cost per example from $0.045 to $0.0004.

On AppWorld with gpt-4.1, ACON matched the uncompressed accuracy at 26% fewer peak tokens; on multi-objective QA, peak tokens fell 54%.

Every long-running agent you deploy will eventually hit its context limit or its budget. ACON is a practical template for compressing context without retraining the agent, and it works with API models.

Learn the guideline from failures. The useful part is the procedure, not a particular prompt: run tasks with and without compression, look at the ones compression broke, and write down what the compressor dropped. For a customer's workflow that might be "always keep ticket IDs, the customer's plan tier and any error code". Two or three rounds of this beat a generic "summarize the conversation".

Compress observations, not just history. Tool outputs are often the bulk: a 3,000-token API response where the agent needs four fields. Extracting what matters at arrival keeps the context small from the start.

Use a small compressor. A distilled 14B model did the job at about a hundredth of the cost of gpt-4.1, keeping over 95% of its accuracy.

Know the price. Compression saved money per task ($0.331 to $0.285 on AppWorld) but the agent took more steps (16.1 to 22.8) and longer (73 to 88 seconds median), because every compression is an extra model call and rewriting history invalidates the cached prefix. For latency-sensitive tasks, compress observations and leave history alone; for cost-sensitive batch work, compress both.