Troubleshooting Guide

Version 1.0, July 2, 2026

This guide covers MadMatcher’s own failures: install and dependency problems, its error messages, data-shape and schema problems, and checkpoint/recovery problems. For each, it gives the exact message you see, what causes it, and the fix.

MadMatcher errors vs Spark errors

There are two families of failure, and they live in two guides:

  • A MadMatcher or setup error — a missing dependency, a bad id column, a checkpoint mismatch, a native-library crash. Those are here.

  • A Spark resource error — heap or out-of-memory, “result too large”, shuffle spill, executor loss, Arrow batch overrun. MadMatcher runs on your Spark session and does not set memory or resource options, so those are tuned on your session. They live in the Spark tuning guide.

If you are not sure which one you have, the quick test: a Python Exception / ValueError / ImportError with a MadMatcher message is here; a Java/JVM stack trace (java.lang.OutOfMemoryError, ExecutorLostFailure, and the like) is the Spark guide. Two failures this guide owns break that rule: a PyLucene JVM-init failure is a Java error that belongs here (it is a setup problem, not a resource one), and the xgboost/libomp crash is a hard process death that is neither a clean Python exception nor a JVM stack trace — both are covered below.

Package (wheel) will not install

This is the most upstream failure — the pip install step (run for you by the bundled installer) fails before any MadMatcher code runs. The bundled installer is the supported install path: PyLucene, which sparkly and delex require, has no pip wheel, so a bare pip install madmatcher-pro cannot produce a working install on its own.

Symptom. One of:

ERROR: Package 'madmatcher-pro' requires a different Python: 3.11.9 not in '>=3.12'

or, for a compiled dependency, a Could not build wheels for ... / No matching distribution found for ... naming pyarrow, torch, faiss-cpu, xgboost, or numpy.

Cause. madmatcher-pro requires Python >= 3.12 (its requires-python), so an older interpreter is refused outright. The madmatcher-pro wheel itself is pure Python and carries no platform tag, so it is never the source of an OS or CPU-architecture mismatch; those come from its compiled dependencies (pyarrow, torch, faiss-cpu, xgboost, numpy), which ship platform-specific wheels and fall back to a source build — which then fails — when none matches your Python, OS, or CPU.

Fix. Install into a Python 3.12+ environment (check with python --version). If a dependency has no wheel for your platform, use the bundled installer or the Docker image, which pin a known-good Python and dependency set; see the installation/ scripts and installation/README.md. PyLucene is a separate case, below.

PyLucene or Java not installed

This is the most common fresh-install failure once the package is installed, because PyLucene is the one hard dependency with no pip wheel — it is a Java/JCC build. Sparkly and delex both need it.

Symptom. On the first import of a blocking engine:

ModuleNotFoundError: No module named 'lucene'

or, if PyLucene is present but its JVM cannot start (wrong or missing JDK), a JVM initialization error when the first Lucene call runs. The exact JVM error text is platform- and JDK-specific, so match on the behavior — import lucene fails, or the first index/search call fails inside the JVM — rather than a single string.

Cause. PyLucene is not installed into the active environment, or it was built against a JDK that is not on the path now. PyLucene 9.12.0 is verified against Eclipse Temurin JDK 17 (the release within pyspark 3.5.x’s supported range); a different or absent JDK is the usual second cause.

Fix. Install PyLucene with the bundled installer for your platform rather than by hand — it builds PyLucene/JCC against the correct JDK and installs it into your environment. See the installation/ scripts and installation/README.md. After it runs, verify with:

python -c "import lucene; print('PyLucene', lucene.VERSION)"

If that prints a version, the JVM is wired up correctly.

Missing [semantic] extra (optional dependencies)

The semantic engine’s heavy backends (faiss, sentence-transformers/torch, fastembed, openai) are optional extras, imported the first time they are used, so the base package installs without them.

Symptom. When a backend is missing, a message that names the package and the install command, for example:

dense (semantic) index build needs the optional 'faiss' package, which is not installed.
Install the semantic extras:
    pip install "madmatcher-pro[semantic]"
See the semantic embeddings guide / troubleshooting doc for setup.

The package name changes with the backend: sentence_transformers for SentenceTransformerProvider, fastembed for FastEmbedProvider, openai for OpenAIEmbeddingProvider, faiss for the index build. (Because sentence_transformers pulls in torch, a missing torch also surfaces through the sentence_transformers message.) A few advanced code paths import their backend directly, so you may instead see a raw ModuleNotFoundError: No module named 'faiss' — the fix is the same.

Cause. The [semantic] extra is not installed.

Fix.

pip install "madmatcher-pro[semantic]"

This can surface deep inside a Spark worker (the message reads like a product defect when it does); it is only the missing extra.

Data-shape and schema problems

Blocking assumes each table has a well-formed integer id column. Both check_tables_manual and check_tables_auto validate the id columns and raise a ValueError; run one right after loading, before you spend time embedding or indexing. check_tables_auto additionally requires table B’s columns to be a superset of table A’s; check_tables_manual does not — which is why semantic blocking, where A and B need not share columns, uses the manual check. The messages below are what they raise — the table_a / table_b prefix names which table failed, and '_id' is whatever id column you passed.

Symptoms (each a ValueError):

  • Missing id column:

    table_a: missing id column '_id'. Available columns: [...]

  • Empty table:

    table_a: empty dataframe

  • Nulls in the id column:

    table_a: nulls are present in the id column '_id'

  • Non-integer id column (Spark tables):

    table_a: id column '_id' must be an integer type

    and, for pandas tables, with the dtype named:

    table_a: id column '_id' must be int32/int64 or Int32/Int64 (got object)

  • Non-unique id column:

    table_a: id column '_id' must be unique

  • Table B is not a superset of table A’s columns (check_tables_auto only):

    table_b columns: [...] must be a superset of table_a columns: [...]

Cause. The id column is absent, non-integer, non-unique, or contains nulls; or, under check_tables_auto only, table B’s columns are not a superset of table A’s.

Fix. Give each table a unique, non-null, 32- or 64-bit integer id column and pass its name as id_col. If your ids are strings or UUIDs, add an integer id (for example with zipWithIndex or a monotonically increasing id you then persist). Embedding fields may differ between A and B, but the id columns must both be valid; for the fields, embed the same kind of information on each side (see the semantic tutorial, Step 3).

Checkpoint and recovery problems

Crash recovery pins the run’s configuration and input size in the checkpoint’s manifest.json and refuses to resume if either changed, so a resume can never mix work from two different runs. For the conceptual model — how groups, markers, and resumes work — see Reference: Crash Recovery and Progress Tracking in the semantic tutorial (each engine tutorial has the same section). The errors:

Changed configuration. A pinned manifest key (id dtype, grouping, feature/program fingerprint, n_groups, and so on) differs from the current run:

checkpoint at <dir> has <key>=<old> but this run uses <key>=<new>; refusing to resume. Use a fresh checkpoint_dir or matching parameters.

Changed input. The input row count differs from when the checkpoint was created (a CheckpointMismatchError):

input row count changed since checkpoint <dir> was created (100 -> 120); a resumable run requires a stable input, because a record inserted into (or deleted from) an already-committed group would never be processed. Use a fresh checkpoint_dir for a changed input, or pass validate_input=False to skip this check.

Corrupt or incompatible checkpoint. A manifest that does not pin n_groups:

checkpoint manifest at <dir> does not pin n_groups (corrupt or incompatible checkpoint); refusing to resume

Changed centroids (semantic build). Resuming a semantic index build with a different injected centroid set:

injected centroids differ from the set a prior run committed; a resume must reuse the same cell space (do not re-train centroids between runs)

Corrupt model checkpoint. Loading a matcher whose meta.json is not valid JSON (a ModelStoreError):

model checkpoint at <dir> is corrupt: meta.json is not valid JSON (<detail>). Remove the checkpoint dir to retrain, or restore a good copy.

Corrupt dashboard checkpoint. The dashboard reader on a bad manifest.json:

checkpoint at <dir> has a corrupt manifest.json (<detail>); cannot display it

Cause and fix. For the config/input/centroid mismatches, this is the guardrail working: you changed a parameter or the input between runs. Either run with the original configuration and input to resume, or point at a fresh checkpoint_dir to start over. validate_input=False skips only the row-count check (use it when you know the input is stable but its count legitimately changed). For the corrupt-file cases, the checkpoint was interrupted mid-write or damaged — remove that checkpoint directory and re-run, or restore a known-good copy.

Dashboard does not appear

Symptom. Your job runs to completion, but the progress dashboard never comes up: session.url is None, the URL you printed does not load, or no browser opens.

Cause. The dashboard is best-effort by design — if the server cannot start, the failure is swallowed so your job is never affected. The usual reasons are a busy port (every port in the scan range starting at dashboard_port is taken), a headless host where the server cannot bind or open a browser, or a remote driver whose bind address you cannot reach from your machine.

Fix.

  • Busy port: choose a different starting port — MadMatcherSession.builder.port(4060), the dashboard_port argument on an operation, or --port on the read-only viewer. The server scans upward from there for a free port.

  • Headless host / no browser: expected; the job and dashboard still run, there is just no browser to open. Reach it by URL (session.url) or use the read-only viewer.

  • Remote driver: the dashboard runs in the Spark driver. Reach it with an SSH tunnel (ssh -L 4050:localhost:4050 user@driver-host), or bind it wider with .host("0.0.0.0") and secure it at the network layer (it has no built-in authentication).

  • Fallback that always works: point the read-only viewer at the checkpoint directory from any machine that can read it — python -m madmatcher_pro.reliability.dashboard <checkpoint_dir>.

Because the dashboard is best-effort, none of these affect your results or recovery. For the full reference (builder options, remote access, the viewer’s flags), see “Viewing progress on the dashboard” in any engine tutorial.

xgboost / libomp native crash (SIGSEGV)

The Spark tuning guide defers the full detail here.

Symptom. The whole process dies with no Python traceback: a segmentation fault (SIGSEGV, signal 11), surfacing as exit code 139 (128 + 11) or an hs_err_pid*.log file dropped in the working directory, when the xgboost matcher runs in the same process as the PyLucene / Spark JVM (and/or faiss).

Cause. xgboost, faiss, and the JVM each load their own copy of the OpenMP runtime (libomp). More than one copy in a single process is unsafe and can crash it.

Fix.

  • Set KMP_DUPLICATE_LIB_OK=TRUE in the environment before the process starts (export KMP_DUPLICATE_LIB_OK=TRUE), which lets the duplicate runtimes coexist.

  • MadMatcher already pins xgboost/BLAS to a single native thread during prediction — SKLearnModel caps the OpenMP and BLAS pools (via threadpoolctl) before it predicts — so apply_matcher needs nothing from you. Training is not wrapped that way, so if you fit the matcher in the same process as the JVM/faiss, build it single-threaded: with SKLearnModel, pass the estimator class and the argument together, SKLearnModel(XGBClassifier, n_jobs=1).

  • The durable fix is process separation: run the faiss / xgboost work and the JVM (Lucene / Spark) work in separate processes where you can.

KMP_DUPLICATE_LIB_OK=TRUE is a workaround, not a guarantee; process separation is what removes the conflict.

Semantic search: hot-cell overrun (ENOBUFS)

The Spark guide’s ENOBUFS section defers the semantic-specific case here.

Symptom. A No buffer space available (ENOBUFS) socket error or an out-of-memory during SemanticSearcher.search, concentrated on one task while the rest finish — the signature of a single oversized group rather than broad memory pressure.

Cause. Semantic search groups query and document vectors by index cell and scores each cell as one matrix. If the index has a skewed (hot) cell — far more vectors in it than the others — that one group’s score matrix and its Arrow payload can overrun a worker’s socket buffer or memory. This is bounded by the searcher’s own knobs, not by a Spark setting.

Fix. Lower max_cell_queries and/or max_cell_docs on SemanticSearcher (both default 2048). They cap how many queries and documents a single scored group holds by sub-partitioning a hot cell, so a smaller value bounds the per-task matrix and payload:

searcher = SemanticSearcher(index=index, max_cell_queries=512, max_cell_docs=512)

Lower them until the overrun clears; smaller groups add some overhead but keep any one task bounded. The result is unchanged — sub-partitioning a cell does not change which candidates are returned.

maxResultSize from inside a MadMatcher operation

The Spark guide covers spark.driver.maxResultSize as a Spark setting. This is the case where it points here instead.

Symptom. The maxResultSize error (see the Spark guide for the exact string) raised from inside a MadMatcher call, not from your own collect() or toPandas().

Cause. A MadMatcher operation pulled an unexpectedly large result to the driver. The usual reason is a data-shape problem upstream — a non-unique or mis-typed id producing a many-to-many blowup in a join, or far more candidates per record than expected — so the amount coming back to the driver is much larger than the data should produce.

Fix. First rule out the data shape: run check_tables_manual on your inputs (see “Data-shape and schema problems” above) and check your candidate counts per record. If the shape is correct and the volume is genuinely that large, raise spark.driver.maxResultSize as described in the Spark tuning guide.

License check-in fails when creating a session

madmatcher-pro is licensed software (all-or-nothing: the whole package). Importing the package does nothing license-related; the single license gate is a mandatory online check-in that runs when you create a MadMatcherSession. It resolves your API key from the first of three sources that yields one – the MADMATCHER_LICENSE_KEY environment variable (raw key), the MADMATCHER_LICENSE_KEY_FILE environment variable (a path to a file containing the key), then the config file ~/.config/madmatcher/license.key – and, only if there is no valid cached token, POSTs your API key to the license server over HTTPS; on success it caches the returned signed token at ~/.cache/madmatcher/license.jwt (honoring XDG_CACHE_HOME) so later sessions start offline. A cached token stays usable for a 30-day grace period past its expiry, and a valid cache lets a session start even when the server is unreachable. (The open-source engines are a separate GitHub project and are not gated.)

Symptom. Creating the session raises a LicenseError with one of three messages:

MadMatcher license check-in denied: this API key is not licensed. Contact support@madmatcher.ai to activate or renew your license.

Cannot reach the MadMatcher license server at https://license.madmatcher.ai/checkin (...); no valid cached license is available to fall back on. Check your network connection and retry, or contact support@madmatcher.ai.

MadMatcher license key not found. Provide it via one of (first wins): the MADMATCHER_LICENSE_KEY environment variable (the raw key); the MADMATCHER_LICENSE_KEY_FILE environment variable (a path to a file containing the key); or the config file ~/.config/madmatcher/license.key. Contact support@madmatcher.ai to obtain a key.

Cause and fix, per message:

  • “not licensed” (server returned 403). The check-in reached the server and it refused this API key (not active, revoked, or unknown). Contact support@madmatcher.ai to activate or renew (licenses are issued by hand for now), then create the session again.

  • “Cannot reach the MadMatcher license server”. The server was unreachable (timeout or connection error) and there is no valid cached token to fall back on. Check your network / firewall and retry. Once you have checked in successfully at least once, the cached token carries you through later outages for the grace period.

  • “license key not found”. None of the three key sources yielded a key (and there is no valid cache). Provide the key MadMatcher gave you via, first-wins:

    • MADMATCHER_LICENSE_KEY – the raw key, for clusters where a fixed home-dir path is unreliable. Note a raw key here is visible in the process environment, so prefer the file-by-path option below in shared or logged environments.

    • MADMATCHER_LICENSE_KEY_FILE – a path to a file containing the key, for secret-mount setups (orchestrators mount the secret as a file and pass its path), keeping the raw secret out of the environment.

    • ~/.config/madmatcher/license.key – the single-machine convenience.

    Contact support@madmatcher.ai to obtain a key.

The check-in sends only your API key to the license server. No dataset, table, query, or result ever leaves your infrastructure. The gate runs once, at session creation, so a job that starts after a successful check-in runs to completion.

A note on scope

This is not a complete list of every possible failure. It covers the common ones we have seen so far, each with the message the code actually emits. For Spark resource and tuning issues, see the Spark tuning guide. For anything not covered here, contact support@madmatcher.ai.