Design or optimize a Celery task queue for high-volume applications.
Celery Task Queue Architect
Celery, RabbitMQ, Redis, Python
Best for
- ▸Design scalable Celery task queue architecture for high-volume applications processing millions of tasks daily
- ▸Debug and optimize existing Celery deployments experiencing message loss, worker bottlenecks, or retry failures
- ▸Implement robust task retry strategies with exponential backoff and dead letter queues for production systems
- ▸Configure RabbitMQ or Redis message brokers with proper durability, routing, and high availability for Celery
What you'll get
- ▸Complete Celery configuration with broker-specific settings, retry decorators, and monitoring setup for a specific workload pattern
- ▸Queue topology design with routing keys, exchange configuration, and worker deployment strategy for multi-environment setup
- ▸Production-ready task implementation with idempotency patterns, error handling, and performance optimization techniques
Specific details about task volume, latency requirements, failure scenarios, current architecture constraints, and the types of workloads (CPU-bound, I/O-bound, scheduled batch jobs) being processed.
Detailed architectural recommendations with specific Celery configuration code, broker setup instructions, queue topology diagrams, and production-ready monitoring and retry strategies.
What's inside
“You are a distributed systems engineer. You design and operate Celery task queue deployments processing millions of tasks per day across real-time offloading, batch ETL, multi-tenant SaaS, and large-scale data pipelines. - **Distinguish transient from permanent failures** -- Route HTTP 5xx and tim...”
Covers
Not designed for ↓
- ×General Python async/await programming or basic task scheduling without distributed queue requirements
- ×Real-time communication systems like WebSockets or chat applications requiring immediate bidirectional messaging
- ×Simple cron-based job scheduling that doesn't require distributed processing or complex retry logic
SupaScore
89.35▼
Evidence Policy
Standard: no explicit evidence policy.
Research Foundation: 7 sources (3 official docs, 2 books, 1 web, 1 industry frameworks)
This skill was developed through independent research and synthesis. SupaSkills is not affiliated with or endorsed by any cited author or organisation.
Version History
v5.5 distilled from v2 via Claude Sonnet
Pipeline v4: rebuilt with 3 helper skills
Initial release
Prerequisites
Use these skills first for best results.
Works well with
Need more depth?
Specialist skills that go deeper in areas this skill touches.
Common Workflows
Distributed System Architecture Design
Complete workflow for designing, implementing, monitoring, and operating a distributed task processing system from API endpoints through background processing to production reliability.
© 2026 Kill The Dragon GmbH. This skill and its system prompt are protected by copyright. Unauthorised redistribution is prohibited. Terms of Service · Legal Notice