Data & AnalyticsTechnologyPlatinum

Optimizing large-scale Apache Spark data processing tasks.

Apache Spark Data Processing Expert

PySpark, Spark SQL, Delta Lake

advancedv5.0

Best for

▸Optimizing PySpark ETL pipelines processing TB-scale data lakes
▸Tuning Spark SQL queries with partition skew and broadcast join optimization
▸Implementing Delta Lake ACID transactions with merge upsert patterns
▸Designing streaming pipelines with exactly-once semantics and checkpointing

What you'll get

▸Complete PySpark code with specific configuration parameters, partition strategies, and join optimizations for multi-TB ETL workflows
▸Detailed cluster sizing recommendations with memory allocation, executor configuration, and adaptive query execution settings
▸Production streaming pipeline architecture with fault tolerance patterns, exactly-once processing guarantees, and monitoring setup

Expects

Specific details about data volume, format, processing patterns, current performance bottlenecks, and cluster configuration to provide targeted optimization recommendations.

Returns

Production-ready PySpark code with detailed optimization strategies, cluster configuration tuning, and performance monitoring approaches tailored to your specific workload.

What's inside

“You are a Senior Apache Spark Data Processing Expert. You architect and optimize distributed data pipelines at petabyte scale, combining deep Spark internals knowledge with battle-tested production patterns. 1. **Systematic Optimization, Not Guessing** , Evaluate workloads across six dimensions: da...”

Covers

What You Do DifferentlyMethodologyWatch For

Not designed for ↓

×Basic SQL query writing or small dataset analysis
×Setting up Hadoop clusters or low-level HDFS administration
×Machine learning model development (focuses on data processing, not ML algorithms)
×Real-time sub-second latency processing (Spark is micro-batch, not true streaming)

SupaScore

89.35▼

Research Quality (15%)

8.85

Prompt Engineering (25%)

9.2

Practical Utility (15%)

8.8

Completeness (10%)

8.9

User Satisfaction (20%)

Decision Usefulness (15%)

8.65

Evidence Policy

Standard: no explicit evidence policy.

apache-sparkpysparkspark-sqldistributed-computingdata-pipelinedelta-lakeperformance-tuningetldata-engineeringlakehousestructured-streamingcluster-optimizationbig-data

Research Foundation: 8 sources (3 official docs, 2 books, 2 paper, 1 community practice)

This skill was developed through independent research and synthesis. SupaSkills is not affiliated with or endorsed by any cited author or organisation.