LESS: Selecting Influential Data for Targeted Instruction Tuning

The setting is targeted instruction tuning: you have a big, mixed pool of instruction data (here about 270,000 examples from FLAN v2, CoT, Dolly and Open Assistant) and a handful of examples of the capability you want, and you want to pick the training subset that builds that capability.

LESS (Low-rank gradiEnt Similarity Search) uses a first-order idea from earlier influence work: if training on example z moves the model's weights in a direction that also lowers the loss on target example z′, then z helps. With SGD that is the dot product of the two gradients. LESS adapts it in three ways. It uses the Adam update direction for the training example, since instruction tuning uses Adam. It uses cosine instead of a dot product, because an example's gradient averages over its tokens and so shrinks for long examples, which made the dot product pick very short data. And it makes the computation cheap with LoRA gradients and a random projection to 8,192 dimensions.

The steps: warm up a LoRA model on a random 5% of the pool for 4 epochs; compute and store every example's projected gradient at each checkpoint; for each target subtask, average the target examples' gradients and score every candidate by cosine similarity, summed over checkpoints; keep the top 5% and train the target model on it.

The selected 5% beat a random 5% everywhere and beat the full pool in 5 of 9 cases across LLaMA-2-7B, LLaMA-2-13B and Mistral-7B. Data selected with LLaMA-2-7B worked for the other two models. Word-frequency, n-gram and hidden-state selectors did not beat random consistently.

Customers often arrive with a pile of data (old tickets, chat logs, generic instruction sets) and a short list of examples of what they want the model to do well. The usual moves are to fine-tune on everything or to filter by keyword or embedding similarity. This paper shows both can be worse than a principled selection, and gives a method that is affordable.

Less data can be better. Training on the selected 5% beat the full pool in 5 of 9 cells, because much of a generic pool is irrelevant or harmful for a specific skill. The authors also note that fine-tuning on this pool lowered MMLU below the untuned base model for LLaMA-2-13B and Mistral-7B, a reminder that fine-tuning can take capability away.

Similar-looking is not the same as useful. Selecting by words, n-grams or the model's hidden representations picked data on the same topic or in the same language, and scored about as well as random or worse. Gradients pick data that exercises the same reasoning.

The cost is front-loaded and reusable. About 54 A100 hours and 17.7 GB of storage for 270K examples, once. After that, each new target skill is a sub-minute query. A team serving several customers or skills from one data pool pays the cost once.