Airflow

When to Use

• ETL/ELT: Scheduling nightly data warehouse loads.
• ML Ops: Retraining models when new data arrives.
• Dependency Management: "Run Task B only if Task A succeeds".

Core Concepts

DAGs (Directed Acyclic Graphs)

Task SDK

Edge Executor

Best Practices (2025)

• Use the TaskFlow API:

  @task

  decorators are cleaner than

  PythonOperator

  .
• Use Datasets: Define data-aware scheduling (

  schedule=[Dataset("s3://bucket/file")]

  ).

• Don't put top-level code in DAG files: It runs every scheduler heartbeat.

References

• Airflow Documentation