Idempotent Data Pipelines

Pipelines retry, get re-run, and get backfilled, and every one of those re-runs must produce the same result as running once. Idempotency is the property that makes that true: write by key with upsert or partition overwrite, never blind append, so a retry cannot double-count. It is the single property that makes a pipeline safe to operate, because the alternative is a 2 a.m. page where you cannot tell if it is safe to run the job again.