ETL Pipeline Designer

Designs robust ETL and ELT data pipelines with focus on idempotency, incremental loading, data lineage, and orchestration. Guides implementation with Apache Airflow, dbt, and modern data stack tools.

Gold

v1.0.10 activationsData & AnalyticsTechnologyadvanced

SupaScore

84.9

Research Quality (15%)

8.5

Prompt Engineering (25%)

8.5

Practical Utility (15%)

8.7

Completeness (10%)

8.5

User Satisfaction (20%)

8.3

Decision Usefulness (15%)

8.5

Best for

▸Designing incremental loading patterns with CDC and timestamp-based watermarks
▸Building idempotent Airflow DAGs with proper retry and failure handling
▸Implementing dbt data transformation workflows with incremental models
▸Creating data quality validation frameworks with Great Expectations
▸Architecting ELT pipelines for cloud data warehouses like Snowflake and BigQuery

What you'll get

●Complete Airflow DAG code with task dependencies, error handling, and SLA monitoring for a multi-source data integration
●dbt project structure with incremental models, tests, and documentation for dimensional modeling
●Detailed architecture diagram showing data flow, transformation layers, and quality checkpoints with specific tool recommendations

Not designed for ↓

×Real-time streaming analytics or Kafka event processing
×Machine learning model training pipelines or MLOps workflows
×Ad-hoc data analysis or exploratory data science
×Frontend data visualization or dashboard building

Expects

Clear requirements for data sources, target systems, SLA requirements, data volume/velocity, and business transformation logic.

Returns

Detailed pipeline architecture with DAG design, incremental loading strategy, data quality checks, orchestration setup, and implementation code examples.

Evidence Policy

Enabled: this skill cites sources and distinguishes evidence from opinion.

etleltdata-pipelineairflowdbtdata-transformationincremental-loadingcdcdata-lineageidempotentdata-qualityorchestration

Research Foundation: 8 sources (5 official docs, 1 books, 1 industry frameworks, 1 community practice)

This skill was developed through independent research and synthesis. SupaSkills is not affiliated with or endorsed by any cited author or organisation.