Data Pipeline Architect
Designs robust data pipelines covering ETL/ELT patterns, orchestration (Airflow/Dagster/Prefect), data quality validation, lineage tracking, idempotency, streaming vs batch architecture, and data lakehouse design for production workloads.
SupaScore
84.75Best for
- ▸Designing Airflow DAGs with proper task dependencies and failure recovery
- ▸Implementing data quality validation using Great Expectations in production pipelines
- ▸Building streaming data architecture with Kafka and Spark for real-time analytics
- ▸Creating idempotent ELT pipelines with dbt for cloud data warehouses
- ▸Setting up data lineage tracking and impact analysis for regulatory compliance
What you'll get
- ●Detailed architecture diagrams with tool recommendations, explaining why ELT over ETL for cloud warehouses with specific Airflow DAG patterns
- ●Step-by-step implementation guide for Kafka + Spark Streaming architecture with monitoring and alerting setup
- ●dbt project structure with data quality tests, lineage documentation, and CI/CD pipeline integration patterns
Not designed for ↓
- ×Writing SQL queries for ad-hoc analysis (this is analytical work, not pipeline architecture)
- ×Building machine learning models (this focuses on data movement, not model training)
- ×Setting up basic database connections or simple data exports
- ×Frontend data visualization or dashboard creation
Clear requirements including data sources, volumes, latency needs, destinations, and SLAs for a production data pipeline.
Detailed technical architecture recommendations with specific tools, code patterns, monitoring strategies, and implementation guidance.
Evidence Policy
Enabled: this skill cites sources and distinguishes evidence from opinion.
Research Foundation: 8 sources (5 official docs, 3 books)
This skill was developed through independent research and synthesis. SupaSkills is not affiliated with or endorsed by any cited author or organisation.
Version History
Auto-versioned: masterfile quality gate passed (score: 85.5)
Initial release
Works well with
Need more depth?
Specialist skills that go deeper in areas this skill touches.
Common Workflows
Modern Analytics Stack Implementation
End-to-end implementation from raw data ingestion through transformation and quality validation to analytics-ready datasets
Activate this skill in Claude Code
Sign up for free to access the full system prompt via REST API or MCP.
Start Free to Activate This Skill© 2026 Kill The Dragon GmbH. This skill and its system prompt are protected by copyright. Unauthorised redistribution is prohibited. Terms of Service · Legal Notice