← 🗄️ Data & SQL Engineering
Core
Idempotent Data Pipelines
Pipelines retry, get re-run, and get backfilled, and every one of those re-runs must produce the same result as running once. Idempotency is the property that makes that true: write by key with upsert or partition overwrite, never blind append, so a retry cannot double-count. It is the single property that makes a pipeline safe to operate, because the alternative is a 2 a.m. page where you cannot tell if it is safe to run the job again.
a free account unlocks the core curriculum tier · no card
RELATED CONCEPTS
LESSONS THAT TEACH THIS
PRACTICE THIS IN REAL QUESTIONS
System Design & Production EngineeringWhat's the difference between at-least-once and exactly-once delivery, and why should an FDE care?→System Design & Production EngineeringA Python service starts double-processing messages under load. Why does this happen, and how do you fix it?→MLOps & ML EngineeringHow do you ensure reproducibility in ML workflows?→MLOps & ML EngineeringHow would you implement an MLOps pipeline on AWS using SageMaker, CodePipeline, and Lambda?→MLOps & ML EngineeringExplain MLOps on Azure, how do Azure DevOps and Azure ML Pipelines fit together?→MLOps & ML EngineeringDesign an automated retraining pipeline. What should trigger retraining?→