# Spark Tuning Guide

_Version 1.0, July 2, 2026_

MadMatcher runs on your Spark session. It does **not** create a cluster, size any
memory, or set executor, shuffle, or result-size options for you. When you call a
MadMatcher operation, the library reuses your existing `SparkSession`
(`SparkSession.builder.getOrCreate()`), and the **only** Spark setting it ever
changes is turning on Arrow (`spark.sql.execution.arrow.pyspark.enabled=true`) so
data moves between the JVM and Python efficiently.

So the memory, result-size, batch, spill, and executor settings that govern a run
are the ones on your own Spark session. Tuning them is a normal part of running any
Spark workload at scale, and this guide maps the error you actually see to the
standard Spark setting that addresses it.

If instead you see a MadMatcher error message (a missing `[semantic]` dependency, a
corrupt checkpoint, a data-shape or column problem), that is covered in the
[Troubleshooting guide](madmatcher-troubleshooting.md), not here.

---

## How to set these

Every setting below is a standard Spark configuration key. Set it in one of the
usual ways:

- **On the session builder**, before `getOrCreate()`:

  ```python
  from pyspark.sql import SparkSession

  spark = (
      SparkSession.builder
          .master("local[*]")
          .appName("MadMatcher")
          .config("spark.sql.shuffle.partitions", "64")
          .config("spark.driver.maxResultSize", "4g")
          .getOrCreate()
  )
  ```

- **On the command line** with `spark-submit`, for a cluster:

  ```bash
  spark-submit --conf spark.executor.memory=8g --conf spark.executor.cores=4 your_script.py
  ```

- **In `spark-defaults.conf`**, to apply to every job on a cluster.

One caveat that causes a lot of confusion: **driver and executor memory and cores
(and `spark.driver.maxResultSize`) must be set before the session starts.** They
size the JVMs at launch and cannot be changed on a session that is already running,
so a `spark.conf.set(...)` after `getOrCreate()` silently has no effect for those
keys. The SQL runtime settings (`spark.sql.shuffle.partitions`,
`spark.sql.execution.arrow.maxRecordsPerBatch`) can be changed on a live session.

`spark.driver.memory` in **local mode** (`local[*]`) is a sharper case of the same
trap: the driver JVM is already running by the time `SparkSession.builder` executes,
so `.config("spark.driver.memory", ...)` on the builder silently does nothing. In
local mode set driver memory with `--driver-memory` (on `spark-submit`), the
`SPARK_DRIVER_MEMORY` environment variable, or `spark-defaults.conf` — not the
builder.

The failure modes below are ordered from most to least commonly hit.

---

## Slow or uneven jobs: cores and parallelism

**Symptom.** No error — the job just runs far slower than the machine should allow.
Cores sit idle, or the Spark UI shows a stage with only a handful of tasks (or one
giant long-running task) while everything else waits, or thousands of tiny tasks
with more scheduling overhead than work.

**Cause.** Spark runs one task per core and cuts your data into partitions, one task
per partition. If there are fewer partitions than cores, cores sit idle. If there
are far more partitions than cores, you pay scheduling overhead on each. Two things
decide this: how many tasks can run at once (`spark.executor.cores`,
`spark.driver.cores`) and how many partitions the data is split into
(`spark.default.parallelism`, `spark.sql.shuffle.partitions`).

**Fix.**

- `spark.executor.cores` — tasks per executor, e.g. `4`. (On `local[*]` the driver
  does the work and already uses all cores.)
- `spark.driver.cores` — e.g. `4`.
- `spark.sql.shuffle.partitions` — partitions after a DataFrame shuffle, e.g. `64`
  (default `200`). The blocking and fusion joins go through this one.
- `spark.default.parallelism` — default partitions for lower-level (RDD) operations,
  e.g. `64`. Set it alongside `shuffle.partitions`.

A common starting point is 2–3× your total core count for the partition counts.

**Watch.** Too few partitions leaves cores idle and makes each task hold more data
(raising OOM risk); too many makes scheduling overhead dominate. Read the task
counts in the Spark UI and adjust.

## Result too large: `spark.driver.maxResultSize`

**Symptom.**

```
Total size of serialized results of N tasks (X GB) is bigger than spark.driver.maxResultSize (1024.0 MB)
```

**Cause.** An action pulled more data back to the driver than its cap allows —
usually a large `collect()` or `toPandas()`. The driver limits how many bytes of
task results it will gather so a big pull cannot exhaust its heap.

**Fix.** `spark.driver.maxResultSize`, e.g. `4g` (default `1g`; `0` means unlimited,
which is not recommended).

**Watch.** This is a guardrail, so raise `spark.driver.memory` to match — more data
on the driver needs more driver heap. If you hit this from inside a MadMatcher
operation rather than your own `collect()`, it can point to a data-shape problem;
see the [Troubleshooting guide](madmatcher-troubleshooting.md).

## Out of memory / heap errors

**Symptom.**

```
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: GC overhead limit exceeded
```

or an executor killed with `Container killed ... for exceeding memory limits` /
`Exit code is 137`.

**Cause.** A task needed more memory than its JVM had — a large partition, a wide
feature or embedding vector, or too few partitions so each task holds too much.
`spark.executor.memory` and `spark.driver.memory` size the JVM heap;
`spark.executor.memoryOverhead` covers **off-heap** memory (Arrow buffers, the
Python workers, native libraries) that lives outside that heap.

**Fix.**

- `spark.executor.memory` — JVM heap per executor, e.g. `8g`.
- `spark.driver.memory` — e.g. `8g` (on `local[*]` the driver does the work, so this
  is usually the one that matters). In local mode it must come from `--driver-memory`,
  the `SPARK_DRIVER_MEMORY` environment variable, or `spark-defaults.conf`, **not** the
  builder `.config()` (see "How to set these").
- `spark.executor.memoryOverhead` — off-heap, e.g. `2g`. Raise this one
  specifically for the "exceeding memory limits" / exit-137 kill, which is an
  off-heap overrun, not a heap overrun.

**Watch.** These are launch-time settings (see "How to set these"). More memory per
executor usually means fewer executors fit on a node. If more memory does not help,
increase the partition count (above) so each task handles less data.

## Arrow batch size: ENOBUFS and large-batch OOM

**Symptom.** A socket error — `No buffer space available` (ENOBUFS; the errno number
varies by platform, `[Errno 105]` on Linux, `[Errno 55]` on macOS/BSD) — or a
`Connection reset`, or an out-of-memory, during a step that moves data to Python:
featurization, applying the matcher, or semantic search.

**Cause.** This is the one part of Spark's data path MadMatcher touches: the library
enables Arrow to move rows between the JVM and Python. Arrow ships rows in batches,
and `spark.sql.execution.arrow.maxRecordsPerBatch` (default `10000`) caps the batch
size. A very wide row — a large feature vector or a high-dimensional embedding —
times 10,000 rows can produce a batch big enough to overrun a worker's socket buffer
or its memory.

**Fix.** `spark.sql.execution.arrow.maxRecordsPerBatch`, e.g. `2000` — lower it until
the batch fits the row width (default `10000`).

**Watch.** Smaller batches use less memory but add per-batch overhead, so lower it
only as far as you need. One case this Spark knob does **not** cover: in semantic
search, an overrun on a skewed (hot) index cell is bounded by the library's own
`max_cell_queries` and `max_cell_docs` settings on `SemanticSearcher`, not by this
Arrow setting — if you hit ENOBUFS there, see the [Troubleshooting
guide](madmatcher-troubleshooting.md).

## Executor loss and heartbeat timeouts

**Symptom.**

```
ExecutorLostFailure (executor 3 exited caused by one of the running tasks)
Executor heartbeat timed out after 130000 ms
```

or repeated `Removing executor` messages and task retries.

**Cause.** An executor stopped answering the driver within the allowed window —
usually because it was stuck in a long garbage-collection pause under memory
pressure, or was killed outright. The driver waits `spark.network.timeout` for a
heartbeat before it gives the executor up.

**Fix.**

- `spark.network.timeout`, e.g. `300s` (default `120s`) — tolerate longer pauses.
- `spark.executor.heartbeatInterval`, e.g. `30s` — must stay well below the network
  timeout.

**Watch.** Raising the timeout only hides the symptom. If executors keep dying,
treat it as memory pressure first (the memory and partition settings above); a
timeout bump helps only when the executor is alive but briefly too busy to respond.

## "No space left on device" during shuffle spill

**Symptom.**

```
java.io.IOException: No space left on device
```

in the middle of a shuffle or sort — commonly misread as "my dataset is too big"
even when the input data itself fits fine.

**Cause.** When a shuffle (a join or group-by — the fusion RRF and wide blocking
joins are the usual ones) does not fit in memory, Spark spills temporary files to
local disk under `spark.local.dir` (default `/tmp`). If that volume fills up, the
job fails even though nothing is wrong with your data.

**Fix.** `spark.local.dir`, e.g. `/mnt/large-disk/spark-tmp` — a path on a volume
with room. Comma-separate several directories to spread the load across disks.

**Watch.** Point it at your largest (and ideally fastest) local disk, not a small
root partition. Reducing the shuffle itself — more partitions, fewer wide joins —
also lowers how much gets spilled. Under YARN, `spark.local.dir` is ignored: the node
manager's local directories (`yarn.nodemanager.local-dirs`) decide where spill goes,
so on a YARN cluster set the spill location there, at the cluster level.

## "Too many open files"

**Symptom.**

```
java.io.IOException: Too many open files
```

during a shuffle-heavy stage — often misread as a data problem when it is really a
process limit.

**Cause.** A shuffle opens many files at once (roughly map-tasks × reduce-partitions),
plus sockets and index files. At scale this can exceed the operating system's
per-process open-file limit (`ulimit -n`, often 1024 by default), and the job fails
even though the data is fine. This is an OS limit, **not** a Spark setting.

**Fix.** Raise the open-file limit for the user that runs Spark — e.g. `ulimit -n
65536` in the launching shell, or a `nofile` entry in `/etc/security/limits.conf` (on
a cluster, on the worker nodes). Fewer shuffle partitions also lowers the file count.

**Watch.** This is an environment limit, so it is set on the host/cluster, not through
`SparkSession`; a change needs a new shell (or login) to take effect.

## xgboost + JVM crash (SIGSEGV / exit 139)

This one is **not** a Spark setting — it is a native-library conflict — but it looks
alarming enough to belong here.

**Symptom.** The whole process dies with no Python traceback: a segmentation fault
(`SIGSEGV`, signal 11), surfacing as exit code `139` (128 + 11) or an
`hs_err_pid*.log` file dropped in the working directory, when the xgboost matcher
runs in the same process as the PyLucene / Spark JVM (and/or faiss).

**Cause.** xgboost, faiss, and the JVM each bring their own copy of the OpenMP
runtime (`libomp`). Loading more than one copy into a single process is unsafe and
can crash it.

**Fix.** This is an environment and process fix, not a config key:

- Set `KMP_DUPLICATE_LIB_OK=TRUE` in the environment before the process starts
  (e.g. `export KMP_DUPLICATE_LIB_OK=TRUE`), which lets the duplicate runtimes
  coexist.
- Native-thread pileup makes the collision more likely, but MadMatcher already pins
  xgboost/BLAS to a single native thread during **prediction** — `SKLearnModel` caps
  the OpenMP and BLAS pools (via `threadpoolctl`) before it predicts — so the
  `apply_matcher` step needs nothing from you. **Training** is not wrapped that way,
  so if you fit the matcher in the same process as the JVM/faiss, build it
  single-threaded: with `SKLearnModel`, pass the estimator **class** and the argument
  together, `SKLearnModel(XGBClassifier, n_jobs=1)`, so it is forwarded when the
  estimator is constructed.
- Where you can, run the faiss / xgboost work and the JVM (Lucene / Spark) work in
  separate processes.

**Watch.** `KMP_DUPLICATE_LIB_OK=TRUE` is a workaround, not a guarantee; the durable
fix is keeping the native libraries in separate processes. Fuller detail is in the
[Troubleshooting guide](madmatcher-troubleshooting.md).

---

## A note on scope

This is not a complete list of Spark errors or fixes. It covers the common ones we
have seen so far, each mapped to the standard Spark setting that addresses it. If you
hit something not here, the general [Spark tuning](https://spark.apache.org/docs/latest/tuning.html)
and [configuration](https://spark.apache.org/docs/latest/configuration.html) docs
apply, and the Spark UI is the place to start diagnosing it. If the cause still
isn't clear, contact support.

For MadMatcher's own error messages and data-shape problems, see the [Troubleshooting
guide](madmatcher-troubleshooting.md). Support: **support@madmatcher.ai**.