25Design a daily pipeline that survives reruns, late-arriving data, and a 3-year backfill, without double-counting.▼hard★ EssentialDatabricksPalantirSnowflake2 replies◆ premiumThe pipeline-design question that exposes whether you've ever been paged. One property, idempotency, solves reruns, late data, and backfills at once, and there are exactly three patterns that deliver it.Open full answer →
39Design a backfill for an aggregate table that takes late and out-of-order events without double-counting or holes.▼hardDatabricksSnowflakePalantir1 replies◆ premiumThe aggregate-restatement problem that breaks naive backfills: events arrive days late and out of order, so any partition you already published may be wrong. The staff answer is event-time partitioning plus bounded restatement, not a bigger watermark.Open full answer →