dedup

All-pairs comparison is O(n^2) and dies long before 1TB. The senior move is MinHash plus LSH: hash documents so only likely-similar pairs ever land in the same bucket, then verify just those.

The naive pipeline gives you 10 million 'summarize this paragraph' pairs and a model that can only summarize paragraphs. The hard part is diversity and quality control at 100-billion-document scale, not the generation call.