active learning

Budget covers maybe 5M of 100M LLM calls. Run a cheap first stage on everything, route only low-confidence items to the LLM, and use those LLM labels to distill a student that absorbs most of the volume. The cost-vs-coverage math that decides where the threshold goes.

The amateur move is to train a fresh deep net on 200 rows and overfit instantly. The senior move is to spend almost no labels on training: lean on a pretrained model, use the 200 as a validation set, and design a labeling loop that buys the most signal per label.

Random sampling is the baseline you have to beat, and honest candidates admit it sometimes wins. Active learning earns its keep by labeling where the model is least sure, but uncertainty alone gives you a batch of near-duplicates, so you also need diversity, and a cold-start plan.