LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression

Prompts keep getting longer: retrieved documents, meeting transcripts, many examples. Prompt compression removes words a model does not need, saving cost and latency. Earlier methods such as LLMLingua and Selective-Context ranked tokens by information entropy, computed by a small causal model like LLaMA-2-7B: surprising tokens stay, predictable ones go. Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang and colleagues at Tsinghua and Microsoft argue this is the wrong signal. Entropy is not the same as importance for an LLM, and a causal model sees only the words before each token.

LLMLingua-2 learns compression instead. First, data distillation: GPT-4 compresses MeetingBank transcripts under strict rules, remove unimportant words only; do not reorder, change, abbreviate or add words. Long transcripts are cut into chunks of up to 512 tokens first. Second, annotation: an algorithm aligns the compressed text with the original, with fuzzy matching and a bidirectional search to cope with GPT-4's slips, and labels every original word preserve or discard. Third, quality control: examples where GPT-4 introduced new words (high variation rate) or where the labels poorly reproduce GPT-4's compression (alignment gap) are filtered out.

The compressor is a token classifier on a bidirectional encoder, XLM-RoBERTa-large, or multilingual BERT for a smaller version. At run time it scores every word's probability of being kept, in one pass, and keeps the top fraction for the target ratio, in original order. It never generates text, so it cannot hallucinate, and it does not see the question, so one compressed version serves any downstream task.

With GPT-3.5-Turbo as the target model, LLMLingua-2 nearly matched the full prompt on MeetingBank questions at about 3x compression, beat entropy-based compressors on out-of-domain benchmarks (LongBench, ZeroSCROLLS, GSM8K, BBH), and transferred to Mistral-7B. It ran 3 to 6 times faster than earlier compressors, cut end-to-end latency 1.6 to 2.9 times, and used 2.1 GB of GPU memory against 16.6 GB for LLMLingua.

Prompt compression is one of the few levers that cuts both cost and latency without changing the model, and customers with long-context workloads (meeting notes, call transcripts, long retrieved passages) ask for it.

Extractive compression is safe in one way. The output contains only words from the input, in order. There is no paraphrase to hallucinate, which matters for regulated content and for traceability.

It is fast enough for the hot path. Half a second on a V100 for a meeting transcript, with a model that fits in about 2 GB, is cheap compared with the LLM call it shortens.

Task-agnostic has a price. Without the question, the compressor keeps what is generally important. On LongBench, question-aware LongLLMLingua scored 48.0 against 39.1, and the uncompressed prompt 44.0. If you know the question at compression time, use it.

Test on the customer's content. The compressor was trained on meeting transcripts only. It generalized reasonably, but tasks needing exact strings, codes or counts suffered: on LongBench's synthetic tasks it scored 21.3 against 37.8 uncompressed.