Design and manage data pipelines for reliable data delivery.
Data Pipeline Architect
Airflow, Kafka, dbt, Great Expectations
Best for
- ▸Designing Airflow DAGs with proper task dependencies and failure recovery
- ▸Implementing data quality validation using Great Expectations in production pipelines
- ▸Building streaming data architecture with Kafka and Spark for real-time analytics
- ▸Creating idempotent ELT pipelines with dbt for cloud data warehouses
What you'll get
- ▸Detailed architecture diagrams with tool recommendations, explaining why ELT over ETL for cloud warehouses with specific Airflow DAG patterns
- ▸Step-by-step implementation guide for Kafka + Spark Streaming architecture with monitoring and alerting setup
- ▸dbt project structure with data quality tests, lineage documentation, and CI/CD pipeline integration patterns
Clear requirements including data sources, volumes, latency needs, destinations, and SLAs for a production data pipeline.
Detailed technical architecture recommendations with specific tools, code patterns, monitoring strategies, and implementation guidance.
What's inside
“You are a Data Pipeline Architect. You design, build, and operate reliable, scalable, maintainable data pipelines that deliver high-quality data on time, at reasonable cost. - Prioritize simplicity, reliability, and operational excellence over architectural purity; ELT with dbt is the modern default...”
Covers
Not designed for ↓
- ×Writing SQL queries for ad-hoc analysis (this is analytical work, not pipeline architecture)
- ×Building machine learning models (this focuses on data movement, not model training)
- ×Setting up basic database connections or simple data exports
- ×Frontend data visualization or dashboard creation
SupaScore
89.65▼
Evidence Policy
Standard: no explicit evidence policy.
Research Foundation: 8 sources (5 official docs, 3 books)
This skill was developed through independent research and synthesis. SupaSkills is not affiliated with or endorsed by any cited author or organisation.
Version History
v5.5 final distill
Pipeline v4: rebuilt with 3 helper skills
Auto-versioned: masterfile quality gate passed (score: 85.5)
Initial release
Works well with
Need more depth?
Specialist skills that go deeper in areas this skill touches.
Common Workflows
Modern Analytics Stack Implementation
End-to-end implementation from raw data ingestion through transformation and quality validation to analytics-ready datasets
© 2026 Kill The Dragon GmbH. This skill and its system prompt are protected by copyright. Unauthorised redistribution is prohibited. Terms of Service · Legal Notice