Semantic (dense / embedding blocking)
=====================================

Dense, embedding-based blocking. The pipeline is ``create_embeddings`` →
``SemanticIndex`` → ``SemanticSearcher`` → ``reciprocal_rank_fusion``; the providers
and serializers control how records are embedded. The **advanced** section at the
bottom lists the optional knobs for tuning a ``SemanticIndex`` beyond its defaults
and the bounded-memory build for very large inputs.

Core pipeline
-------------

.. autofunction:: madmatcher_pro.semantic.create_embeddings

.. autoclass:: madmatcher_pro.semantic.SemanticIndex
   :members: upsert_docs, load
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.SemanticSearcher
   :members: search
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.SemanticQuerySpec
   :show-inheritance:

.. autofunction:: madmatcher_pro.semantic.reciprocal_rank_fusion

.. autoclass:: madmatcher_pro.semantic.CandidateSource
   :show-inheritance:

.. autofunction:: madmatcher_pro.semantic.shrink_mrl

Embedding providers
-------------------

Pass one as the ``provider`` to ``create_embeddings``.

.. autoclass:: madmatcher_pro.semantic.EmbeddingProvider
   :members:
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.SentenceTransformerProvider
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.FastEmbedProvider
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.OpenAIEmbeddingProvider
   :show-inheritance:

Serializers
-----------

Control how a record's fields become the one string that is embedded. Pass one as
the ``serializer`` to ``create_embeddings``.

.. autoclass:: madmatcher_pro.semantic.Serializer
   :members:
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.DittoSerializer
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.PlainConcatSerializer
   :show-inheritance:

Advanced
--------

Optional. Most users do not need these.

Index compression codecs
~~~~~~~~~~~~~~~~~~~~~~~~~~

Pass an instance as ``SemanticIndex(codec=...)`` to store compressed posting
vectors (the default keeps full vectors).

.. autoclass:: madmatcher_pro.semantic.FlatCodec
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.SQ8Codec
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.PQCodec
   :show-inheritance:

Vector stores
~~~~~~~~~~~~~

Pass an instance as ``SemanticIndex(store=...)`` to control where full vectors are
kept for the rerank step (the default reads them from Parquet).

.. autoclass:: madmatcher_pro.semantic.FileSystemVectorStore
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.MemmapVectorStore
   :show-inheritance:

Bounded-memory build (very large inputs)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``run_chunked_search`` builds and searches in bounded memory when the embeddings are
too large to index at once. Provide the A/B data as a ``ChunkSource``.

.. autofunction:: madmatcher_pro.semantic.run_chunked_search

.. autoclass:: madmatcher_pro.semantic.EmbeddedSource
   :show-inheritance:

.. autoclass:: madmatcher_pro.semantic.LazyEmbedSource
   :show-inheritance: