Spark Tuning Guide

Version 1.0, July 2, 2026

MadMatcher runs on your Spark session. It does not create a cluster, size any memory, or set executor, shuffle, or result-size options for you. When you call a MadMatcher operation, the library reuses your existing SparkSession (SparkSession.builder.getOrCreate()), and the only Spark setting it ever changes is turning on Arrow (spark.sql.execution.arrow.pyspark.enabled=true) so data moves between the JVM and Python efficiently.

So the memory, result-size, batch, spill, and executor settings that govern a run are the ones on your own Spark session. Tuning them is a normal part of running any Spark workload at scale, and this guide maps the error you actually see to the standard Spark setting that addresses it.

If instead you see a MadMatcher error message (a missing [semantic] dependency, a corrupt checkpoint, a data-shape or column problem), that is covered in the Troubleshooting guide, not here.

How to set these

Every setting below is a standard Spark configuration key. Set it in one of the usual ways:

  • On the session builder, before getOrCreate():

    from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
        .master("local[*]")
        .appName("MadMatcher")
        .config("spark.sql.shuffle.partitions", "64")
        .config("spark.driver.maxResultSize", "4g")
        .getOrCreate()
)

  • On the command line with spark-submit, for a cluster:

    spark-submit --conf spark.executor.memory=8g --conf spark.executor.cores=4 your_script.py

  • In spark-defaults.conf, to apply to every job on a cluster.

One caveat that causes a lot of confusion: driver and executor memory and cores (and spark.driver.maxResultSize) must be set before the session starts. They size the JVMs at launch and cannot be changed on a session that is already running, so a spark.conf.set(...) after getOrCreate() silently has no effect for those keys. The SQL runtime settings (spark.sql.shuffle.partitions, spark.sql.execution.arrow.maxRecordsPerBatch) can be changed on a live session.

spark.driver.memory in local mode (local[*]) is a sharper case of the same trap: the driver JVM is already running by the time SparkSession.builder executes, so .config("spark.driver.memory", ...) on the builder silently does nothing. In local mode set driver memory with --driver-memory (on spark-submit), the SPARK_DRIVER_MEMORY environment variable, or spark-defaults.conf — not the builder.

The failure modes below are ordered from most to least commonly hit.

Slow or uneven jobs: cores and parallelism

Symptom. No error — the job just runs far slower than the machine should allow. Cores sit idle, or the Spark UI shows a stage with only a handful of tasks (or one giant long-running task) while everything else waits, or thousands of tiny tasks with more scheduling overhead than work.

Cause. Spark runs one task per core and cuts your data into partitions, one task per partition. If there are fewer partitions than cores, cores sit idle. If there are far more partitions than cores, you pay scheduling overhead on each. Two things decide this: how many tasks can run at once (spark.executor.cores, spark.driver.cores) and how many partitions the data is split into (spark.default.parallelism, spark.sql.shuffle.partitions).

Fix.

  • spark.executor.cores — tasks per executor, e.g. 4. (On local[*] the driver does the work and already uses all cores.)

  • spark.driver.cores — e.g. 4.

  • spark.sql.shuffle.partitions — partitions after a DataFrame shuffle, e.g. 64 (default 200). The blocking and fusion joins go through this one.

  • spark.default.parallelism — default partitions for lower-level (RDD) operations, e.g. 64. Set it alongside shuffle.partitions.

A common starting point is 2–3× your total core count for the partition counts.

Watch. Too few partitions leaves cores idle and makes each task hold more data (raising OOM risk); too many makes scheduling overhead dominate. Read the task counts in the Spark UI and adjust.

Result too large: spark.driver.maxResultSize

Symptom.

Total size of serialized results of N tasks (X GB) is bigger than spark.driver.maxResultSize (1024.0 MB)

Cause. An action pulled more data back to the driver than its cap allows — usually a large collect() or toPandas(). The driver limits how many bytes of task results it will gather so a big pull cannot exhaust its heap.

Fix. spark.driver.maxResultSize, e.g. 4g (default 1g; 0 means unlimited, which is not recommended).

Watch. This is a guardrail, so raise spark.driver.memory to match — more data on the driver needs more driver heap. If you hit this from inside a MadMatcher operation rather than your own collect(), it can point to a data-shape problem; see the Troubleshooting guide.

Out of memory / heap errors

Symptom.

java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: GC overhead limit exceeded

or an executor killed with Container killed ... for exceeding memory limits / Exit code is 137.

Cause. A task needed more memory than its JVM had — a large partition, a wide feature or embedding vector, or too few partitions so each task holds too much. spark.executor.memory and spark.driver.memory size the JVM heap; spark.executor.memoryOverhead covers off-heap memory (Arrow buffers, the Python workers, native libraries) that lives outside that heap.

Fix.

  • spark.executor.memory — JVM heap per executor, e.g. 8g.

  • spark.driver.memory — e.g. 8g (on local[*] the driver does the work, so this is usually the one that matters). In local mode it must come from --driver-memory, the SPARK_DRIVER_MEMORY environment variable, or spark-defaults.conf, not the builder .config() (see “How to set these”).

  • spark.executor.memoryOverhead — off-heap, e.g. 2g. Raise this one specifically for the “exceeding memory limits” / exit-137 kill, which is an off-heap overrun, not a heap overrun.

Watch. These are launch-time settings (see “How to set these”). More memory per executor usually means fewer executors fit on a node. If more memory does not help, increase the partition count (above) so each task handles less data.

Arrow batch size: ENOBUFS and large-batch OOM

Symptom. A socket error — No buffer space available (ENOBUFS; the errno number varies by platform, [Errno 105] on Linux, [Errno 55] on macOS/BSD) — or a Connection reset, or an out-of-memory, during a step that moves data to Python: featurization, applying the matcher, or semantic search.

Cause. This is the one part of Spark’s data path MadMatcher touches: the library enables Arrow to move rows between the JVM and Python. Arrow ships rows in batches, and spark.sql.execution.arrow.maxRecordsPerBatch (default 10000) caps the batch size. A very wide row — a large feature vector or a high-dimensional embedding — times 10,000 rows can produce a batch big enough to overrun a worker’s socket buffer or its memory.

Fix. spark.sql.execution.arrow.maxRecordsPerBatch, e.g. 2000 — lower it until the batch fits the row width (default 10000).

Watch. Smaller batches use less memory but add per-batch overhead, so lower it only as far as you need. One case this Spark knob does not cover: in semantic search, an overrun on a skewed (hot) index cell is bounded by the library’s own max_cell_queries and max_cell_docs settings on SemanticSearcher, not by this Arrow setting — if you hit ENOBUFS there, see the Troubleshooting guide.

Executor loss and heartbeat timeouts

Symptom.

ExecutorLostFailure (executor 3 exited caused by one of the running tasks)
Executor heartbeat timed out after 130000 ms

or repeated Removing executor messages and task retries.

Cause. An executor stopped answering the driver within the allowed window — usually because it was stuck in a long garbage-collection pause under memory pressure, or was killed outright. The driver waits spark.network.timeout for a heartbeat before it gives the executor up.

Fix.

  • spark.network.timeout, e.g. 300s (default 120s) — tolerate longer pauses.

  • spark.executor.heartbeatInterval, e.g. 30s — must stay well below the network timeout.

Watch. Raising the timeout only hides the symptom. If executors keep dying, treat it as memory pressure first (the memory and partition settings above); a timeout bump helps only when the executor is alive but briefly too busy to respond.

“No space left on device” during shuffle spill

Symptom.

java.io.IOException: No space left on device

in the middle of a shuffle or sort — commonly misread as “my dataset is too big” even when the input data itself fits fine.

Cause. When a shuffle (a join or group-by — the fusion RRF and wide blocking joins are the usual ones) does not fit in memory, Spark spills temporary files to local disk under spark.local.dir (default /tmp). If that volume fills up, the job fails even though nothing is wrong with your data.

Fix. spark.local.dir, e.g. /mnt/large-disk/spark-tmp — a path on a volume with room. Comma-separate several directories to spread the load across disks.

Watch. Point it at your largest (and ideally fastest) local disk, not a small root partition. Reducing the shuffle itself — more partitions, fewer wide joins — also lowers how much gets spilled. Under YARN, spark.local.dir is ignored: the node manager’s local directories (yarn.nodemanager.local-dirs) decide where spill goes, so on a YARN cluster set the spill location there, at the cluster level.

“Too many open files”

Symptom.

java.io.IOException: Too many open files

during a shuffle-heavy stage — often misread as a data problem when it is really a process limit.

Cause. A shuffle opens many files at once (roughly map-tasks × reduce-partitions), plus sockets and index files. At scale this can exceed the operating system’s per-process open-file limit (ulimit -n, often 1024 by default), and the job fails even though the data is fine. This is an OS limit, not a Spark setting.

Fix. Raise the open-file limit for the user that runs Spark — e.g. ulimit -n 65536 in the launching shell, or a nofile entry in /etc/security/limits.conf (on a cluster, on the worker nodes). Fewer shuffle partitions also lowers the file count.

Watch. This is an environment limit, so it is set on the host/cluster, not through SparkSession; a change needs a new shell (or login) to take effect.

xgboost + JVM crash (SIGSEGV / exit 139)

This one is not a Spark setting — it is a native-library conflict — but it looks alarming enough to belong here.

Symptom. The whole process dies with no Python traceback: a segmentation fault (SIGSEGV, signal 11), surfacing as exit code 139 (128 + 11) or an hs_err_pid*.log file dropped in the working directory, when the xgboost matcher runs in the same process as the PyLucene / Spark JVM (and/or faiss).

Cause. xgboost, faiss, and the JVM each bring their own copy of the OpenMP runtime (libomp). Loading more than one copy into a single process is unsafe and can crash it.

Fix. This is an environment and process fix, not a config key:

  • Set KMP_DUPLICATE_LIB_OK=TRUE in the environment before the process starts (e.g. export KMP_DUPLICATE_LIB_OK=TRUE), which lets the duplicate runtimes coexist.

  • Native-thread pileup makes the collision more likely, but MadMatcher already pins xgboost/BLAS to a single native thread during prediction — SKLearnModel caps the OpenMP and BLAS pools (via threadpoolctl) before it predicts — so the apply_matcher step needs nothing from you. Training is not wrapped that way, so if you fit the matcher in the same process as the JVM/faiss, build it single-threaded: with SKLearnModel, pass the estimator class and the argument together, SKLearnModel(XGBClassifier, n_jobs=1), so it is forwarded when the estimator is constructed.

  • Where you can, run the faiss / xgboost work and the JVM (Lucene / Spark) work in separate processes.

Watch. KMP_DUPLICATE_LIB_OK=TRUE is a workaround, not a guarantee; the durable fix is keeping the native libraries in separate processes. Fuller detail is in the Troubleshooting guide.

A note on scope

This is not a complete list of Spark errors or fixes. It covers the common ones we have seen so far, each mapped to the standard Spark setting that addresses it. If you hit something not here, the general Spark tuning and configuration docs apply, and the Spark UI is the place to start diagnosing it. If the cause still isn’t clear, contact support.

For MadMatcher’s own error messages and data-shape problems, see the Troubleshooting guide. Support: support@madmatcher.ai.