spark

The naive pipeline gives you 10 million 'summarize this paragraph' pairs and a model that can only summarize paragraphs. The hard part is diversity and quality control at 100-billion-document scale, not the generation call.

The Databricks screen opener that decides whether the rest of the interview goes deep or stays remedial. Definitions are table stakes, the points are in explaining what a shuffle physically does and how to see one in the Spark UI.

The Databricks signature deep-dive. Everyone says 'salting', almost nobody can write the salt correctly or explain why AQE might make hand-salting unnecessary. The full playbook, in the order a senior engineer actually tries it.

The small-files problem is the #1 self-inflicted lakehouse wound FDEs find in the field. The causes are always the same three things, and the fix has a prevention half most candidates forget.

Basic salting assumes one side is a small dimension you can explode. When both sides are huge and skewed, exploding either one blows up the cluster. The staff move is to split hot keys out and handle them on a different plan entirely.