MatchFlow (ML matching + active learning)¶

ML-based matching with active learning. You call the pipeline functions (create_features → featurize → create_seeds / label_data → train_matcher → apply_matcher), construct a model wrapper and a labeler and pass them in, and optionally validate your tables. The advanced base classes at the bottom are only for building custom pieces.

Pipeline functions¶

MatchFlow - A toolkit for entity matching.

This package provides tools for creating, training, and applying entity matchers using various tokenization, featurization, and machine learning techniques.

madmatcher_pro.matchflow.create_features(A, B, a_cols, b_cols, sim_functions=None, tokenizers=None, null_threshold=0.5)¶

creates the features which will be used to featurize your tuple pairs

Parameters:

A (Union[pd.DataFrame, SparkDataFrame]) – the records of table A
B (Union[pd.DataFrame, SparkDataFrame]) – the records of table B
a_cols (list) – The names of the columns for DataFrame A that should have features generated
b_cols (list) – The names of the columns for DataFrame B that should have features generated
sim_functions (list of callables, optional) – similarity functions to apply (default: None)
tokenizers (list of callables, optional) – tokenizers to use (default: None)
null_threshold (float) – the portion of values that must be null in order for the column to be dropped and not considered for feature generation

Returns:

a list containing initialized feature objects for columns in A, B

Return type:

List[Callable]

madmatcher_pro.matchflow.get_base_sim_functions()¶

get the base similarity functions

Returns:: a list of similarity functions, currently includes: - TFIDFFeature - JaccardFeature - SIFFeature - OverlapCoeffFeature - CosineFeature
Return type:: list

madmatcher_pro.matchflow.get_base_tokenizers()¶

get the base tokenizers

Returns:: a list of tokenizers, currently includes: - StrippedWhiteSpaceTokenizer - NumericTokenizer - QGramTokenizer(3)
Return type:: list

madmatcher_pro.matchflow.get_extra_tokenizers()¶

get the extra tokenizers

Returns:: a list of extratokenizers, currently includes: - AlphaNumericTokenizer - QGramTokenizer(5) - StrippedQGramTokenizer(3) - StrippedQGramTokenizer(5)
Return type:: list

madmatcher_pro.matchflow.featurize(features, A, B, candidates, output_col='feature_vectors', fill_na=0.0, *, enable_crash_recovery=False, checkpoint_dir=None, n_groups=10, rows_per_group=None, show_progress=False, validate_input=True, a_embeddings=None, b_embeddings=None, cosine_scores_col=None, embedding_id_col='_id', embedding_col='embedding', cosine_in_score=True, cosine_available_bytes=None, cosine_n_chunks=None, cosine_n_a=None, cosine_n_b=None, cosine_dim=None, dashboard=None, dashboard_port=None, open_browser=None)¶

applies the featurizer to the record pairs in candidates

Semantic cosine (matching) feature, optional. Beyond the features you pass, featurize can append a semantic cosine – cosine(emb[id1], emb[id2]) – as the LAST element of every pair’s vector. It turns on in any of three ways, in order of precedence: (1) automatically, when the candidates already carry the canonical mm_sem_scores column that semantic / fusion blocking emit (no arguments needed); (2) cosine_scores_col – reuse a per-pair cosine column the candidates already carry; or (3) pass both a_embeddings and b_embeddings (id-keyed embedding tables) to compute it. It stays off if none of these apply, so the output is unchanged for lexical-only blocking.

Parameters:

features (List[Callable]) – a list containing initialized feature objects for columns in A, B
A (Union[pd.DataFrame, SparkDataFrame]) – the records of table A
B (Union[pd.DataFrame, SparkDataFrame]) – the records of table B
candidates (Union[pd.DataFrame, SparkDataFrame]) – id pairs of A and B that are potential matches candidates with the following columns: - id2: id from table B - id1_list: list of candidate ids from table A
output_col (str) – the name of the column for the resulting feature vectors, default feature_vectors
fill_na (float) – value to fill in for missing data, default 0.0
enable_crash_recovery (bool) – When True, featurization is resumable: candidates are split into n_groups deterministic xxhash64(id2) groups, each group’s feature vectors written with a commit marker, and a re-run with the same checkpoint_dir skips committed groups and unions the rest. checkpoint_dir is then required and the arguments below take effect. When False (default) a single-job featurize runs. The crash-recovery path returns a Spark DataFrame unless A and B were pandas (then the union is collected to pandas).
checkpoint_dir (str, optional) – required when enable_crash_recovery=True: where the per-group feature vectors and commit markers are stored. May be local/hdfs:///s3a://.
n_groups (int) – number of groups to split the candidates into (enable_crash_recovery=True)
rows_per_group (int, optional) – if set, derive n_groups as ceil(num_candidates / rows_per_group)
show_progress (bool) – show a per-group progress bar (enable_crash_recovery=True)
validate_input (bool) – when enable_crash_recovery=True (default True): record the candidate row count and refuse to resume if it changed (one extra pass over candidates). The feature fingerprint, output_col and fill_na are also pinned, so a changed feature set / output column / fill value refuses a stale checkpoint. Set False only for known-immutable candidates.
a_embeddings (DataFrame, optional) – Opt-in semantic cosine feature. Pass this AND b_embeddings (both id-keyed embedding tables with columns embedding_id_col, embedding_col) to append cosine(emb[id1], emb[id2]) as the LAST element of every pair’s vector. Off by default (output unchanged). Pass both or neither. This is the A-side (id1) embedding table.
b_embeddings (DataFrame, optional) – The B-side (id2) embedding table for the cosine feature. Pass together with a_embeddings (both or neither).
cosine_scores_col (str, optional) – Reuse a per-pair cosine the candidates already carry (the semantic blocker’s scores) instead of computing it. This asserts the column is RAW cosine; it is NOT validated (that would scan the table), so do not point it at a fused RRF / lexical score. With embeddings also given, reuse where present and compute only the rest.
embedding_id_col (str) – Name of the id column in the embedding tables, default ‘_id’.
embedding_col (str) – Name of the vector column in the embedding tables, default ‘embedding’.
cosine_in_score (bool) – Add the cosine into the score column (default True).
cosine_available_bytes (int, optional) – Per-chunk disk budget for chunk-A/stream-B; sizes the number of A-chunks only when cosine_n_a is also given. No count() is ever forced.
cosine_n_chunks (int, optional) – Explicit number of A-chunks (K_A) for chunk-A/stream-B (the count-free way to chunk huge embeddings); None (default) is a single join.
cosine_n_a (int, optional) – Row count of A (id1) embeddings, if known. Supplied (not counted) so cosine_available_bytes can size the number of A-chunks.
cosine_n_b (int, optional) – Row count of B (id2) embeddings, if known. Used only for the cost log.
cosine_dim (int, optional) – Embedding vector width. None (default) probes one row rather than scanning; pass it to skip even that probe.
dashboard (bool, optional) – opt-in progress dashboard for this call. None (default) defers to the active MadMatcherSession or the env/default; True/False overrides. The global opt-out MADMATCHER_DASHBOARD=0 turns it off. See madmatcher_pro.MadMatcherSession.
dashboard_port (int, optional) – port for the dashboard server; scans upward if busy.
open_browser (bool, optional) – open the dashboard in a browser on launch.

Returns:

DataFrame with feature vectors created with the following schema: (id2, id1, output_col, score, other columns from candidates). Returns pandas DataFrame if inputs A and B are pandas DataFrames, otherwise returns Spark DataFrame.

Return type:

Union[pd.DataFrame, SparkDataFrame]

madmatcher_pro.matchflow.down_sample(fvs, percent, search_id_column, score_column='score', bucket_size=1000, seed=None)¶

down sample by score_column to produce approximately percent * fvs.count() rows

This is an approximate, score-stratified sampler: it buckets by a hash of search_id_column and takes a score-spread slice from each bucket. The output size is close to percent * fvs.count() for inputs at least bucket_size large, but can differ by a few rows for smaller inputs, where a single bucket and integer rounding dominate. It is not an exact-count sampler by design.

Parameters:

fvs (Union[pd.DataFrame, SparkDataFrame]) – the feature vectors to be downsampled
percent (float) – the portion of the vectors to be output, (0.0, 1.0]
search_id_column (str) – the name of the column containing unique identifiers for each record
score_column (str) – the column that scored the vectors, should be positively correlated with the probability of the pair being a match
bucket_size (int = 1000) – the size of the buckets for partitioning, default 1000
seed (int, optional) – seed for the within-bucket random tie-break that picks which rows fill each bucket’s quota. None (default) leaves the selection unseeded (the prior behaviour, nondeterministic across runs); set it for a reproducible sample.

Returns:

the down sampled dataset with approximately percent * fvs.count() rows and the same schema as fvs

Return type:

Union[pd.DataFrame, SparkDataFrame]

madmatcher_pro.matchflow.create_seeds(fvs, nseeds, labeler, score_column='score', parquet_file_path='active-matcher-training-data.parquet', *, dashboard=None, dashboard_port=None, open_browser=None)¶

Create labeled seed examples for active learning.

Parameters:

fvs (Union[pd.DataFrame, SparkDataFrame]) – DataFrame containing feature vectors with scores
nseeds (int) – the number of seeds you want to use to train an initial model
labeler (Labeler) – the labeler object you want to use to assign labels to rows
score_column (str, default='score') – the name of the score column in your fvs DataFrame
parquet_file_path (str, default='active-matcher-training-data.parquet') – The path to save the labeled data to
dashboard (bool, optional) – opt-in progress dashboard for this call. None (default) defers to the active MadMatcherSession or the env/default; True/False overrides. The global opt-out MADMATCHER_DASHBOARD=0 turns it off. See madmatcher_pro.MadMatcherSession.
dashboard_port (int, optional) – port for the dashboard server; scans upward if busy.
open_browser (bool, optional) – open the dashboard in a browser on launch.

Returns:

A DataFrame with labeled seeds, schema is (previous schema of fvs, label) where the values in label are either 0.0 or 1.0

Return type:

Union[pd.DataFrame, SparkDataFrame]

madmatcher_pro.matchflow.train_matcher(model, labeled_data, feature_col='feature_vectors', label_col='label')¶

Train a matcher model on labeled data.

Parameters:

model (MLModel) – An MLModel instance to train
labeled_data (pandas DataFrame) – DataFrame containing the labeled data
feature_col (str, default="feature_vectors") – Name of the column containing feature vectors
label_col (str, default="label") – Name of the column containing labels

Returns:

The trained model

Return type:

MLModel

madmatcher_pro.matchflow.apply_matcher(model, df, feature_col, prediction_col, confidence_col=None, *, enable_crash_recovery=False, checkpoint_dir=None, id_col='_id', n_groups=10, rows_per_group=None, show_progress=False, validate_input=True, dashboard=None, dashboard_port=None, open_browser=None)¶

Apply a trained model to make predictions.

Parameters:

model (MLModel) – A trained MLModel instance
df (pandas DataFrame) – The DataFrame to make predictions on
feature_col (str) – Name of the column containing feature vectors
prediction_col (str) – Name of the column to store predictions in
confidence_col (str, optional) – Name of the column to store confidence scores in. If provided, both predictions and confidence scores will be computed efficiently in a single pass.
enable_crash_recovery (bool) – When True, prediction is resumable: rows are split into n_groups deterministic xxhash64(id_col) groups, each group’s predictions written with a commit marker, and a re-run with the same checkpoint_dir skips committed groups and unions the rest. checkpoint_dir is then required and the arguments below take effect. When False (default) a single-job predict runs. The crash-recovery path operates on Spark and returns a Spark DataFrame (pandas input is converted first, the union collected back).
checkpoint_dir (str, optional) – required when enable_crash_recovery=True: where per-group predictions and commit markers are stored. May be local/hdfs:///s3a://.
id_col (str) – the stable per-row id to group on (enable_crash_recovery=True, default _id, the key featurize assigns each pair).
n_groups (int) – number of groups to split the rows into (enable_crash_recovery=True)
rows_per_group (int, optional) – if set, derive n_groups as ceil(num_rows / rows_per_group)
show_progress (bool) – show a per-group progress bar (enable_crash_recovery=True)
validate_input (bool) – when enable_crash_recovery=True (default True): record the row count and refuse to resume if it changed. The trained-model fingerprint and the prediction columns are also pinned, so applying a different model / column layout refuses a stale checkpoint.
dashboard (bool, optional) – opt-in progress dashboard for this call. None (default) defers to the active MadMatcherSession or the env/default; True/False overrides. The global opt-out MADMATCHER_DASHBOARD=0 turns it off. See madmatcher_pro.MadMatcherSession.
dashboard_port (int, optional) – port for the dashboard server; scans upward if busy.
open_browser (bool, optional) – open the dashboard in a browser on launch.

Returns:

The input DataFrame with predictions added (and confidence scores if requested)

Return type:

Union[pd.DataFrame, SparkDataFrame]

madmatcher_pro.matchflow.label_data(model, mode, labeler, fvs, seeds=None, parquet_file_path='active-matcher-training-data.parquet', **learner_kwargs)¶

Generate labeled data using active learning.

Parameters:

model (MLModel) – An MLModel instance
mode (Literal["batch", "continuous"]) – Whether to use batch or continuous active learning
labeler (Labeler) – A Labeler instance
fvs (pandas DataFrame) – The data that needs to be labeled
seeds (Union[pandas DataFrame, SparkDataFrame], optional) – Initial labeled examples to start with
parquet_file_path (str, default='active-matcher-training-data.parquet') – The path to save the labeled data to
**learner_kwargs – Additional keyword arguments to pass to the active learner constructor. For batch mode, see EntropyActiveLearner (e.g. batch_size, max_iter). For continuous mode, see ContinuousEntropyActiveLearner (e.g. queue_size, max_labeled, on_demand_stop).

Returns:

DataFrame with ids of potential matches and the corresponding label

Return type:

Union[pd.DataFrame, SparkDataFrame]

madmatcher_pro.matchflow.label_pairs(labeler, pairs)¶

Label pairs without active learning.

Parameters:

labeler (Labeler) – A Labeler instance
pairs (Union[pd.DataFrame, SparkDataFrame]) – The pairs to label

Returns:

DataFrame with labeled pairs

Return type:

Union[pd.DataFrame, SparkDataFrame]

madmatcher_pro.matchflow.save_features(features, path)¶

Save a list of feature objects to disk using pickle serialization.

Parameters:

features (List[Callable]) – List of feature objects to save
path (str) – Path where to save the features file

Return type:

None

madmatcher_pro.matchflow.load_features(path)¶

Load a list of feature objects from disk using pickle deserialization.

Parameters:: path (str) – Path to the saved features file
Returns:: List of loaded feature objects
Return type:: List[Callable]

madmatcher_pro.matchflow.save_dataframe(dataframe, path)¶

Save a DataFrame to disk, automatically detecting whether it’s a pandas or Spark DataFrame.

Parameters:

dataframe (Union[pd.DataFrame, pyspark.sql.DataFrame]) – DataFrame to save (pandas or Spark)
path (str) – Path where to save the DataFrame

Return type:

None

madmatcher_pro.matchflow.load_dataframe(path, df_type)¶

Load a DataFrame from disk based on the specified type.

Parameters:

path (str) – Path to the saved DataFrame file
df_type (str) – Type of DataFrame to load (‘pandas’ or ‘sparkdf’)

Returns:

Loaded DataFrame

Return type:

Union[pd.DataFrame, pyspark.sql.DataFrame]

madmatcher_pro.matchflow.check_tables(table_a, table_b)¶

Check that both table_a and table_b have the column ‘_id’. Check that both id columns are unique.

Parameters:

table_a (Union[pd.DataFrame, SparkDataFrame]) – table A; must have a unique ‘_id’ column
table_b (Union[pd.DataFrame, SparkDataFrame]) – table B; must have a unique ‘_id’ column. Must be the same kind of DataFrame (pandas or Spark) as table_a.

madmatcher_pro.matchflow.check_candidates(candidates, table_a, table_b)¶

Check that the candidates have the column ‘id2’ and ‘id1_list’. Check that the id2 column is unique. Check that the id1_list column is a list of ids. Check that the ids in the id1_list column are present in the table_a id column. Check that the ids in the id2 column are present in the table_b id column.

Parameters:

candidates (Union[pd.DataFrame, SparkDataFrame]) – the candidate pairs to validate; must have columns ‘id2’ (unique) and ‘id1_list’ (an array of ids). Any parallel ‘scores’/’mm_sem_scores’ array column must be the same length as ‘id1_list’.
table_a (Union[pd.DataFrame, SparkDataFrame]) – table A; every id in ‘id1_list’ must be present in its ‘_id’ column
table_b (Union[pd.DataFrame, SparkDataFrame]) – table B; every id in ‘id2’ must be present in its ‘_id’ column

madmatcher_pro.matchflow.check_labeled_data(labeled_data, table_a, table_b, label_column_name)¶

Check that the labeled_data have the column ‘id2’, ‘id1_list’, and label_column_name. Check that the label_column_name column is a list of floats. Check that the id2 column is unique. Check that the id1_list column is a list of ids. Check that the ids in the id1_list column are present in the table_a id column. Check that the ids in the id2 column are present in the table_b id column. Check that the label_column list is the same length as the id1_list column.

Parameters:

labeled_data (Union[pd.DataFrame, SparkDataFrame]) – the labeled candidate pairs to validate; must have columns ‘id2’ (unique), ‘id1_list’ (an array of ids), and label_column_name (a list of labels the same length as ‘id1_list’)
table_a (Union[pd.DataFrame, SparkDataFrame]) – table A; every id in ‘id1_list’ must be present in its ‘_id’ column
table_b (Union[pd.DataFrame, SparkDataFrame]) – table B; every id in ‘id2’ must be present in its ‘_id’ column
label_column_name (str) – name of the per-pair label list column, which must be index-aligned with (and the same length as) ‘id1_list’

madmatcher_pro.matchflow.check_gold_data(gold_data, table_a, table_b)¶

Gold data must have the columns ‘id1’ and ‘id2’. Check that the ids in the id1 column are present in the table_a ‘_id’ column. Check that the ids in the id2 column are present in the table_b ‘_id’ column.

Parameters:

gold_data (Union[pd.DataFrame, SparkDataFrame]) – the gold matches to validate; must have columns ‘id1’ and ‘id2’
table_a (Union[pd.DataFrame, SparkDataFrame]) – table A; every id in ‘id1’ must be present in its ‘_id’ column
table_b (Union[pd.DataFrame, SparkDataFrame]) – table B; every id in ‘id2’ must be present in its ‘_id’ column

Model wrappers¶

Wrap your estimator and pass it to train_matcher (then apply_matcher); you do not call the wrapper’s own methods. Use SKLearnModel for an sklearn / xgboost estimator or SparkMLModel for a Spark ML estimator.

class madmatcher_pro.matchflow.SKLearnModel(model, nan_fill=None, use_floats=True, **model_args)¶

Bases: MLModel

Scikit-learn model wrapper.

This class wraps scikit-learn models to provide a consistent interface with PySpark ML models. It handles conversion between pandas and PySpark DataFrames, and manages model training and prediction.

Parameters:

model (sklearn.base.BaseEstimator or type) – The scikit-learn model class or instance to use
nan_fill (float or None, optional) – Value to use for filling NaN values
use_floats (bool, optional) – Whether to use float32 (True) or float64 (False) precision
**model_args (dict) – Additional arguments to pass to the model constructor

class madmatcher_pro.matchflow.SparkMLModel(model, nan_fill=0.0, **model_args)¶

Bases: MLModel

PySpark ML model wrapper.

Wraps a PySpark ML estimator (or a fitted Transformer) to provide the common MLModel interface.

Parameters:

model (pyspark.ml.Estimator, pyspark.ml.Transformer, or type) – The PySpark ML estimator class (or an already-fitted Transformer, which is treated as the trained model) to use
nan_fill (float or None, optional) – Value to use for filling NaN values in feature vectors, default 0.0
**model_args (dict) – Additional arguments to pass to the model constructor

Labelers¶

Construct one and pass it to label_data (or create_seeds) to label pairs during active learning.

class madmatcher_pro.matchflow.GoldLabeler(gold)¶

Bases: Labeler

Gold labeler for labeling pairs of records.

Parameters:: gold (Union[pd.DataFrame, SparkDataFrame]) – the gold dataframe, should contain columns ‘id1’ and ‘id2’

class madmatcher_pro.matchflow.CLILabeler(a_df, b_df, id_col='_id')¶

Bases: Labeler

CLI for labeling pairs of records.

Parameters:

a_df (Union[pd.DataFrame, SparkDataFrame]) – the first dataframe
b_df (Union[pd.DataFrame, SparkDataFrame]) – the second dataframe
id_col (str, default '_id') – the column name of the id column

class madmatcher_pro.matchflow.WebUILabeler(a_df, b_df, id_col='_id', flask_port=5005, streamlit_port=8501, flask_host='127.0.0.1')¶

Bases: Labeler

Web interface for labeling pairs of records.

Parameters:

a_df (Union[pd.DataFrame, SparkDataFrame]) – the first dataframe; its column order is preserved in the UI
b_df (Union[pd.DataFrame, SparkDataFrame]) – the second dataframe
id_col (str, default '_id') – the column name of the id column
flask_port (int, default 5005) – port for the internal Flask backend server
streamlit_port (int, default 8501) – port for the Streamlit UI subprocess
flask_host (str, default '127.0.0.1') – host the Flask backend binds to

class madmatcher_pro.matchflow.HybridCosineLabeler(delegate, cosine_by_pair, *, hi, lo)¶

Bases: Labeler

Auto-label the confident cosine tails; defer the uncertain band to a real labeler.

For each pair the cosine is read from a precomputed {(id1, id2): cosine} map. If cosine >= hi the pair is auto-labeled a match (1.0); if cosine <= lo it is auto-labeled a non-match (0.0); otherwise – the uncertain middle band, or any pair not in the map – the wrapped delegate labeler (gold / CLI / Web / custom) is asked. This spends the expensive (human/gold) labeling budget only where cosine is not decisive.

Why a per-PAIR cosine map and not the embedding vectors: the labeler is a driver-side per-pair callback, and it only ever needs the cosine of the CANDIDATE pairs (one float each), never the full embedding corpus (N x dim floats, which would OOM the driver). So the map is built once, bounded by the candidate set: from_candidates reuses the per-pair cosine already carried through blocking (mm_sem_scores), and from_embeddings computes it for the candidate pairs via the chunked, bounded cosine_for_pairs (chunk-A / stream-B – never collects whole embedding tables).

Why not a single threshold: cosine is also the matcher’s strongest feature (mm_sem_scores), so labeling purely by a cosine cutoff is circular and is most error-prone exactly in the band active learning cares about. Auto-labeling only the confident tails sidesteps both: the tails are where cosine is reliable, the band is delegated.

The delegate’s return value is passed through verbatim for band pairs, so its stop signal (-1.0) still terminates labeling.

Parameters:

delegate (Labeler) – The real labeler asked for the uncertain band (and any pair absent from the map).
cosine_by_pair (dict[tuple[int, int], float]) – Precomputed {(id1, id2): cosine} map over the candidate pairs. Usually built via the from_candidates / from_embeddings classmethods rather than by hand.
hi (float (required)) – Cosine at or above which a pair is auto-labeled a match. Required (no default): the right cutoff is dataset-dependent, so it must be a deliberate choice – calibrate it on a small labeled sample, do not guess. In [-1, 1]; set hi=1.0 to never auto-match.
lo (float (required)) – Cosine at or below which a pair is auto-labeled a non-match. Required, same rationale. In [-1, 1] with lo <= hi; set lo=-1.0 to never auto-reject. Widening the band (lo, hi) sends more pairs to the delegate; narrowing it auto-labels more.

Advanced: extension base classes¶

Subclass these only to build a custom piece (a custom feature, tokenizer, vectorizer, model wrapper, or labeler). Most users use the built-ins above.

class madmatcher_pro.matchflow.Feature(a_attr, b_attr)¶

Bases: ABC

Base class for a feature computed from a pair of records.

Parameters:

a_attr (str) – name of the attribute from table A used to generate this feature. Must be a str (a TypeError is raised otherwise).
b_attr (str) – name of the attribute from table B used to generate this feature. Must be a str (a TypeError is raised otherwise).

build(A, B, cache)¶: Guarenteed to be called before the features preprocessing is done. this method should generate and store all of the metadata required to compute the features over A and B, NOTE B may be None

property a_attr¶: the name of the attribute from table a used to generate this feature

property b_attr¶: the name of the attribute from table a used to generate this feature

preprocess_output_column(for_table_a)¶

get the name of the preprocessing output column for table A or B

Parameters:: for_table_a (bool)

preprocess(data, is_table_a)¶: preprocess the data, adding the output column to data

class madmatcher_pro.matchflow.Tokenizer¶

Bases: ABC

tokenize_spark(input_col)¶

return a column expression that gives the same output as the tokenize method. required for effeciency when building metadata for certain methods

Parameters:: input_col (Column)

abstractmethod tokenize(s)¶: convert the string into a BAG of tokens (tokens should not be deduped)

out_col_name(input_col)¶: the name of the output column from the tokenizer e.g. for a 3gram tokenizer, the tokens from the name columns could be “3gram(name)”

tokenize_set(s)¶: tokenize the string and return a set or None if the tokenize returns None

class madmatcher_pro.matchflow.Vectorizer¶

Bases: object

Base class for all vectorizers.

class madmatcher_pro.matchflow.MLModel¶

Bases: ABC

Abstract base class for machine learning models.

This class defines the interface that all machine learning models must implement, whether they are scikit-learn models or PySpark ML models. It provides methods for training, prediction, confidence estimation, and entropy calculation.

nan_fill¶

Value to use for filling NaN values in feature vectors

Type:: float or None

use_vectors¶

Whether the model expects feature vectors in vector format

Type:: bool

use_floats¶

Whether the model uses float32 (True) or float64 (False) precision

Type:: bool

abstract property nan_fill: float | None¶

Value to use for filling NaN values in feature vectors.

Returns:: The value to use for filling NaN values, or None if no filling is needed
Return type:: float or None

abstract property use_vectors: bool¶

Whether the model expects feature vectors in vector format.

Returns:: True if the model expects vectors, False if it expects arrays
Return type:: bool

abstract property use_floats: bool¶

Whether the model uses float32 or float64 precision.

Returns:: True if the model uses float32, False if it uses float64
Return type:: bool

abstract property trained_model¶

The trained ML Model object

Returns:: The trained ML Model object
Return type:: MLModel

abstractmethod predict(df, vector_col, output_col)¶

Make predictions using the trained model.

Parameters:

df (pandas.DataFrame or pyspark.sql.DataFrame) – The DataFrame containing the feature vectors to predict on
vector_col (str) – Name of the column containing feature vectors
output_col (str) – Name of the column to store predictions in

Returns:

The input DataFrame with predictions added in the output_col

Return type:

pandas.DataFrame or pyspark.sql.DataFrame

abstractmethod predict_with_confidence(df, vector_col, prediction_col, confidence_col)¶

Make predictions and confidence scores using the trained model.

This method is more efficient than calling predict() and prediction_conf() separately as it computes both in a single pass when possible.

Parameters:

df (pandas.DataFrame or pyspark.sql.DataFrame) – The DataFrame containing the feature vectors to predict on
vector_col (str) – Name of the column containing feature vectors
prediction_col (str) – Name of the column to store predictions in
confidence_col (str) – Name of the column to store confidence scores in

Returns:

The input DataFrame with predictions and confidence scores added

Return type:

pandas.DataFrame or pyspark.sql.DataFrame

abstractmethod prediction_conf(df, vector_col, label_column)¶

Calculate prediction confidence scores.

Parameters:

df (pandas.DataFrame or pyspark.sql.DataFrame) – The DataFrame containing the feature vectors
vector_col (str) – Name of the column containing feature vectors
label_column (str) – Name of the column containing true labels

Returns:

The input DataFrame with confidence scores added

Return type:

pandas.DataFrame or pyspark.sql.DataFrame

abstractmethod entropy(df, vector_col, output_col)¶

Calculate entropy of predictions.

Parameters:

df (pandas.DataFrame or pyspark.sql.DataFrame) – The DataFrame containing the feature vectors
vector_col (str) – Name of the column containing feature vectors
output_col (str) – Name of the column to store entropy values in

Returns:

The input DataFrame with entropy values added in the output_col

Return type:

pandas.DataFrame or pyspark.sql.DataFrame

abstractmethod train(df, vector_col, label_column)¶

Train the model on the given data.

Parameters:

df (pandas.DataFrame or pyspark.sql.DataFrame) – The DataFrame containing training data
vector_col (str) – Name of the column containing feature vectors
label_column (str) – Name of the column containing labels

Returns:

The trained model (self)

Return type:

MLModel

abstractmethod params_dict()¶

Get a dictionary of model parameters.

Returns:: Dictionary containing model parameters and configuration
Return type:: dict

fingerprint()¶

A stable identifier of this (trained) model.

Pinned in the crash-recovery checkpoint manifest so applying a different model to the same checkpoint_dir is refused rather than unioning mismatched predictions. The base captures class + hyperparameters (params_dict); subclasses add a digest of the trained weights so a retrain with the same config is also distinguished. Returns a JSON-serialisable dict.

Return type:: dict

prep_fvs(fvs, feature_col='feature_vectors')¶

Prepare feature vectors for model input.

This method handles NaN filling and conversion between vector and array formats based on the model’s requirements.

Parameters:

fvs (pandas.DataFrame or pyspark.sql.DataFrame) – DataFrame containing feature vectors
feature_col (str, optional) – Name of the column containing feature vectors

Returns:

DataFrame with prepared feature vectors

Return type:

pandas.DataFrame or pyspark.sql.DataFrame

class madmatcher_pro.matchflow.Labeler¶

Bases: ABC

Base class for labelers used by active learning and label_pairs.

Subclass this and implement __call__() to label a candidate pair. The labeler is called with the two record ids and returns the pair’s label as a float: 1.0 for a match, 0.0 for a non-match, 2.0 for unsure (not added to the training set), or -1.0 to stop labeling early. Override finish() for an optional hook run once labeling completes.

finish(n_labeled, stopped=False)¶

Called by label_pairs when labeling is complete.

Parameters:

n_labeled (int)
stopped (bool)

class madmatcher_pro.matchflow.CustomLabeler(a_df, b_df, id_col='_id')¶

Bases: Labeler

Custom labeler for labeling pairs of records.

Parameters:

a_df (Union[pd.DataFrame, SparkDataFrame]) – the first dataframe
b_df (Union[pd.DataFrame, SparkDataFrame]) – the second dataframe
id_col (str, default '_id') – the column name of the id column

abstractmethod label_pair(row1, row2)¶

label the pair (id1, id2)

Returns:: float
Return type:: the label for the pair