Similarity Functions and Tokenizers in MatchFlow¶

In MatchFlow, we create a set of features, then use them to convert each tuple pair into a feature vector. Roughly speaking, a feature is a way to compare two values and determine how similar they are. Think of it as answering questions like:

How similar are these two names?
How different are these two ages?

Features are created through two main components:

Tokenizers
What they are: Tools that break text into pieces
Example:
```
Input: "John Smith"
Tokenizer output: ["john", "smith"]

Input: "123 Main Street"
Tokenizer output: ["123", "main", "street"]
```
Breaking text into tokens helps handle:
- Different word orders
- Extra/missing words
- Punctuation
- Typos
- Variations in formatting
- etc.
Similarity Functions
What they are: Methods to compute how similar two sets of tokens are
Available similarity functions in MatchFlow (as of July 31, 2025):
- TF-IDF: Term frequency-inverse document frequency similarity
- Jaccard: Set-based similarity using intersection over union
- SIF: Smooth inverse frequency similarity
- Overlap Coefficient: Set overlap measure
- Cosine: Vector space similarity between token vectors

Understanding Your Similiarity Functions¶

When using MatchFlow, it’s crucial to understand how your chosen similarity functions work:

Why This Matters:

Different similarity functions interpret “high” and “low” scores differently
Some functions return higher scores for more similar items
Some functions return higher scores for less similar items
Some functions have different score ranges (0-1, 0-100, etc.)

Common Patterns:

Set-based functions (Jaccard, Overlap): Higher scores = more similar
Distance functions (Edit Distance): Lower scores = more similar
Vector functions (Cosine, TF-IDF): Higher scores = more similar
Custom functions: You need to test and understand them yourself

Best Practice: Always test your similarity functions with examples to see how the score relates to the probability of a match