Why is my Spark job slow when most tasks finish quickly?

A few tasks running far longer than the rest is the classic signature of data skew — one or a handful of partitions hold most of the rows (often because a join or groupBy key is dominated by a few values like null or a default). Those tasks become stragglers that hold up the whole stage. Open the Spark UI, find the slow stage, and look at the task duration distribution and shuffle read size per task; a large max-vs-median gap confirms skew.

How do I fix data skew in Spark?

Match the fix to the mechanism. Enable Adaptive Query Execution (AQE) so Spark can split skewed partitions at runtime. For skewed joins where one side is small, use a broadcast join to avoid the shuffle entirely. For skewed aggregations or large-large joins, salt the hot key (add a random suffix to spread it across partitions, then aggregate in two passes). Filtering out null/sentinel keys before the join often removes the dominant skew source.

What is shuffle spill and why does it slow Spark down?

Spill happens when a task's data does not fit in execution memory and Spark writes it to disk during a shuffle or aggregation, then reads it back — turning fast in-memory work into slow disk I/O. In the Spark UI, spill (memory and disk) shows per stage. Causes include too few shuffle partitions (each too big), skew concentrating data in a task, or under-sized executors. Increasing parallelism, fixing skew, or right-sizing memory reduces spill.

How do I read the Spark UI to find a bottleneck?

Start at the Stages tab and find the longest-running stage. Open it and look at the task summary metrics: if max task duration and shuffle read are far above the median, you have skew/stragglers. Large spill (memory/disk) means memory pressure. A huge number of tiny tasks or input files means a small-files problem. GC time dominating means memory tuning. Each pattern points to a different fix rather than blindly adding executors.

Spark Job Running Slow: Data Skew and Shuffle Troubleshooting

September 7, 2026

"The Spark job is slow" is where a lot of data engineers reach for more executors — and often make nothing faster while doubling the bill. Spark performance is a diagnosis problem before it is a tuning problem. The Spark UI already tells you whether you're fighting data skew, shuffle spill, a small-files explosion, or genuine under-provisioning. Each has a different fix, and applying the wrong one wastes cluster money. This guide is how to read the signal and match the remedy.

Start in the Spark UI, at the slowest stage

Don't guess. Open the Stages tab, sort by duration, and open the longest stage. The task-level summary metrics are the diagnosis:

What you see in the stage	Diagnosis
Max task duration » median (a few stragglers)	Data skew — a few partitions hold most rows
Large Spill (memory) / Spill (disk)	Memory pressure — data doesn't fit, going to disk
Thousands of tiny tasks / tiny input files	Small-files problem — overhead dominates
Huge shuffle read/write bytes	Shuffle-heavy plan — wide transformations moving too much
GC time a large fraction of task time	Executor memory undersized / caching pressure

The single most useful chart is the task duration distribution for the slow stage. Even task times → the stage is genuinely heavy (scale up or optimise the logic). A long tail where max is 20× the median → skew, and no amount of extra executors helps because the straggler task runs alone.

Data skew: the dominant cause

Spark parallelises by partition. Skew means one key value owns a disproportionate share of the rows, so after a shuffle (join or groupBy) one partition is enormous and its task becomes a straggler while every other core sits idle. The usual culprits:

A join/group key dominated by null, an empty string, a -1 sentinel, or "UNKNOWN".
A natural hot key — one mega-customer, one country, one popular product.
A default timestamp/date bucket that swallows unparseable rows.

Skew fixes, matched to the case

Enable Adaptive Query Execution (AQE). Modern Spark can detect and split skewed partitions at runtime and coalesce shuffle partitions. Confirm spark.sql.adaptive.enabled=true and spark.sql.adaptive.skewJoin.enabled=true. This is the first, cheapest lever and often enough on its own.
Broadcast the small side. If a skewed join has one side small enough to fit in executor memory, a broadcast (map-side) join avoids the shuffle entirely — no shuffle, no skew. AQE can promote joins to broadcast automatically; you can also hint it.
Filter the sentinel first. If null/-1 keys carry no business meaning in the join, drop them before the join. Often this single filter removes the entire skew.
Salt the hot key. For large–large joins or aggregations where the hot key is legitimate, add a random salt (e.g. key + a suffix 0..N) to spread it across N partitions, aggregate, then combine. More code, but the standard remedy when AQE and broadcast don't apply.

Shuffle spill and memory pressure

Spill is Spark writing intermediate shuffle/aggregation data to disk because it doesn't fit in execution memory, then reading it back — converting fast memory work into slow disk I/O. If the slow stage shows large spill:

Too few shuffle partitions makes each partition huge. Historically the default 200 is wrong for large data; let AQE coalesce, or set spark.sql.shuffle.partitions so each partition is a sane size (a few hundred MB, not many GB).
Skew concentrates data into one task's memory — fix the skew and the spill often disappears.
Under-sized executors — genuinely too little memory per core. Right-size executor memory and cores together; more, smaller executors sometimes beats a few giant ones for shuffle.

The small-files problem

Reading a million tiny files (or writing them) makes per-task and metadata overhead dominate actual work — you see a stage with a vast number of trivially short tasks. Fixes: compact input on ingestion, use columnar formats (Parquet) with reasonable file sizes, coalesce/repartition before writing, and on lakehouse tables use table optimisation (e.g. Delta OPTIMIZE/compaction) to keep file sizes healthy.

Reduce the shuffle, not just tune it

The fastest shuffle is the one you don't do. Before tuning a wide stage, ask whether the plan can avoid it:

Filter and project early so less data reaches the shuffle (predicate/column pruning; check the physical plan).
Pre-aggregate before a join where possible.
Prefer broadcast joins for dimension tables.
Persist/cache a DataFrame only when it's reused across actions — needless caching causes its own memory pressure.

A troubleshooting order that saves money

Find the slow stage and read the task distribution.
Stragglers → treat as skew (AQE → broadcast → filter sentinel → salt).
Spill/GC → memory/partition sizing.
Tiny tasks → small files.
Only after ruling the above out, scale the cluster.

Common wrong approaches

Adding executors to fix skew. The straggler task still runs on one core; you've paid for idle machines.
Blindly raising spark.sql.shuffle.partitions. Too many partitions creates scheduling overhead and tiny tasks — the opposite problem.
Caching everything. Cache pressure evicts real work and causes spill/GC.
Leaving AQE off. On modern Spark it fixes a large share of skew and partition-sizing issues for free.

Related resources

Why RAG retrieval quality drops in production — for the data pipelines feeding AI systems.
Golden Signals dashboard for SRE — observing batch/stream job health.
Data Engineering job support guide and MLOps job support guide.

If a Spark job is blowing its SLA in a live pipeline and you need someone senior to read the UI with you and decide skew-vs-spill fast, that's exactly what real-time proxy job support is for. And walking an interviewer through this diagnosis — UI to root cause to the matching fix — is the kind of depth data engineering / data science interview proxy support prepares you to demonstrate.

Last reviewed: September 2026.