DAG Authoring Skill

This skill guides you through creating and validating Airflow DAGs using best practices and MCP tools.

For testing and debugging DAGs, see the testing-dags skill which covers the full test → debug → fix → retest workflow.

⚠️ CRITICAL WARNING: Use MCP Tools, NOT CLI Commands ⚠️

STOP! Before running ANY Airflow-related command, read this.
You MUST use MCP tools for ALL Airflow interactions. CLI commands like
astro dev run
,
airflow dags
, or shell commands to read logs are FORBIDDEN.
Why? MCP tools provide structured, reliable output. CLI commands are fragile, produce unstructured text, and often fail silently.

CLI vs MCP Quick Reference

ALWAYS use Airflow MCP tools. NEVER use CLI commands.

❌ DO NOT USE	✅ USE INSTEAD
`astro dev run dags list`	`list_dags` MCP tool
`airflow dags list`	`list_dags` MCP tool
`astro dev run dags test`	`trigger_dag_and_wait` MCP tool
`airflow tasks test`	`trigger_dag_and_wait` MCP tool
`cat` / `grep` on Airflow logs	`get_task_logs` MCP tool
`find` in dags folder	`list_dags` or `explore_dag` MCP tool
Any `astro dev run ...`	Equivalent MCP tool
Any `airflow ...` CLI	Equivalent MCP tool
`ls` on `/usr/local/airflow/dags/`	`list_dags` or `explore_dag` MCP tool
`cat ... \| jq` to filter MCP results	Read the JSON directly from MCP response

Remember:

✅ Airflow is ALREADY running — the MCP server handles the connection
❌ Do NOT attempt to start, stop, or manage the Airflow environment
❌ Do NOT use shell commands to check DAG status, logs, or errors
❌ Do NOT use bash to parse or filter MCP tool results — read the JSON directly
❌ Do NOT use
```
ls
```
,
```
find
```
, or
```
cat
```
on Airflow container paths (
```
/usr/local/airflow/...
```
)
✅ ALWAYS use MCP tools — they return structured JSON you can read directly

Workflow Overview

┌─────────────────────────────────────┐
│ 1. DISCOVER                         │
│    Understand codebase & environment│
└─────────────────────────────────────┘
                 ↓
┌─────────────────────────────────────┐
│ 2. PLAN                             │
│    Propose structure, get approval  │
└─────────────────────────────────────┘
                 ↓
┌─────────────────────────────────────┐
│ 3. IMPLEMENT                        │
│    Write DAG following patterns     │
└─────────────────────────────────────┘
                 ↓
┌─────────────────────────────────────┐
│ 4. VALIDATE                         │
│    Check import errors, warnings    │
└─────────────────────────────────────┘
                 ↓
┌─────────────────────────────────────┐
│ 5. TEST (with user consent)         │
│    Trigger, monitor, check logs     │
└─────────────────────────────────────┘
                 ↓
┌─────────────────────────────────────┐
│ 6. ITERATE                          │
│    Fix issues, re-validate          │
└─────────────────────────────────────┘

Phase 1: Discover

Before writing code, understand the context.

Explore the Codebase

Use file tools to find existing patterns:

```
Glob
```
for
```
**/dags/**/*.py
```
to find existing DAGs
```
Read
```
similar DAGs to understand conventions
Check
```
requirements.txt
```
for available packages

Query the Airflow Environment

Use MCP tools to understand what's available:

Tool	Purpose
`list_connections`	What external systems are configured
`list_variables`	What configuration values exist
`list_providers`	What operator packages are installed
`get_airflow_version`	Version constraints and features
`list_dags`	Existing DAGs and naming conventions
`list_pools`	Resource pools for concurrency

Example discovery questions:

"Is there a Snowflake connection?" →
```
list_connections
```
"What Airflow version?" →
```
get_airflow_version
```
"Are S3 operators available?" →
```
list_providers
```

Phase 2: Plan

Based on discovery, propose:

DAG structure - Tasks, dependencies, schedule
Operators to use - Based on available providers
Connections needed - Existing or to be created
Variables needed - Existing or to be created
Packages needed - Additions to requirements.txt

Get user approval before implementing.

Phase 3: Implement

Write the DAG following best practices (see below). Key steps:

Create DAG file in appropriate location
Update
```
requirements.txt
```
if needed
Save the file

Phase 4: Validate

Use the Airflow MCP as a feedback loop. Do NOT use CLI commands.

Step 1: Check Import Errors

After saving, call the MCP tool (Airflow will have already parsed the file):

MCP tool:

list_import_errors

If your file appears → fix and retry
If no errors → continue

Common causes: missing imports, syntax errors, missing packages.

Step 2: Verify DAG Exists

MCP tool:

get_dag_details(dag_id="your_dag_id")

Check: DAG exists, schedule correct, tags set, paused status.

Step 3: Check Warnings

MCP tool:

list_dag_warnings

Look for deprecation warnings or configuration issues.

Step 4: Explore DAG Structure

MCP tool:

explore_dag(dag_id="your_dag_id")

Returns in one call: metadata, tasks, dependencies, source code.

Phase 5: Test

📘 See the testing-dags skill for comprehensive testing guidance.

Once validation passes, test the DAG using the workflow in the testing-dags skill:

Get user consent — Always ask before triggering

Trigger and wait — Use

trigger_dag_and_wait(dag_id, timeout=300)

Analyze results — Check success/failure status
Debug if needed — Use
```
diagnose_dag_run
```
and
```
get_task_logs
```

Quick Test (Minimal)

# Ask user first, then:
trigger_dag_and_wait(dag_id="your_dag_id", timeout=300)

For the full test → debug → fix → retest loop, see testing-dags.

Phase 6: Iterate

If issues found:

Fix the code
Check for import errors with
```
list_import_errors
```
MCP tool
Re-validate using MCP tools (Phase 4)
Re-test using the testing-dags skill workflow (Phase 5)

Never use CLI commands to check status or logs. Always use MCP tools.

MCP Tools Quick Reference

Phase	Tool	Purpose
Discover	`list_connections`	Available connections
Discover	`list_variables`	Configuration values
Discover	`list_providers`	Installed operators
Discover	`get_airflow_version`	Version info
Validate	`list_import_errors`	Parse errors (check first!)
Validate	`get_dag_details`	Verify DAG config
Validate	`list_dag_warnings`	Configuration warnings
Validate	`explore_dag`	Full DAG inspection

Testing tools — See the testing-dags skill for
trigger_dag_and_wait
,
diagnose_dag_run
,
get_task_logs
, etc.

Best Practices & Anti-Patterns

For detailed code examples and patterns, see reference/best-practices.md.

Key topics covered:

TaskFlow API usage
Credentials management (connections, variables)
Provider operators
Idempotency patterns
Data intervals
Task groups
Setup/Teardown patterns
Data quality checks
Anti-patterns to avoid

Related Skills

testing-dags: For testing DAGs, debugging failures, and the test → fix → retest loop
debugging-dags: For troubleshooting failed DAGs
migrating-airflow-2-to-3: For migrating DAGs to Airflow 3

authoring-dags

NPX Install

Tags

SKILL.md Content

DAG Authoring Skill

⚠️ CRITICAL WARNING: Use MCP Tools, NOT CLI Commands ⚠️

CLI vs MCP Quick Reference

Workflow Overview

Phase 1: Discover

Explore the Codebase

Query the Airflow Environment

Phase 2: Plan

Phase 3: Implement

Phase 4: Validate

Step 1: Check Import Errors

Step 2: Verify DAG Exists

Step 3: Check Warnings

Step 4: Explore DAG Structure

Phase 5: Test

Quick Test (Minimal)

Phase 6: Iterate

MCP Tools Quick Reference

Best Practices & Anti-Patterns

Related Skills