delex.storage package

Subpackages

Submodules

delex.storage.memmap_arr module

class delex.storage.memmap_arr.MemmapArray(arr)

Bases: SparkDistributable

deinit()

deinitialize the object, closing resources (e.g. file handles)

delete()
init()

initialize the object to be used on in a spark worker

property shape
size_in_bytes()
to_spark()

send the obj to the spark cluster to be used on spark workers

property values

delex.storage.memmap_seqs module

class delex.storage.memmap_seqs.MemmapSeqs

Bases: SparkDistributable

a class to hold arbitrary sequences of elements e.g. strings, arrays of ints, etc.

classmethod build(df: DataFrame, seq_col: str, dtype: type, id_col: str = '_id')

create a MemmapSeqs instance from a spark dataframe

Parameters:

  • df (pyspark.sql.DataFrame) – the dataframe containing the sequences and ids

  • seq_col (str) – the name of the column in df that contains the sequences, e.g. strings, arrays

  • dtype (type) – the dtype of the elements in seq_col

  • id_col (str) – the name of the column in df that contains the ids for retrieving the sequences

Return type:

MemmapSeqs

deinit()

deinitialize the object, closing resources (e.g. file handles)

delete()
fetch(i: int, /) → ndarray | None

retrieve the sequence associated with i

Return type:

np.ndarray if i is found, else None

init()

initialize the object to be used on in a spark worker

size_in_bytes() → int

return the size in bytes on disk

to_spark()

send the obj to the spark cluster to be used on spark workers

delex.storage.packed_memmap_arrs module

class delex.storage.packed_memmap_arrs.PackedMemmapArrays(arrs)

Bases: SparkDistributable

a container for many MemmapArrays. used to store many MemmapArrays in a single file

deinit()

deinitialize the object, closing resources (e.g. file handles)

delete()
init()

initialize the object to be used on in a spark worker

size_in_bytes() → int
to_spark()

send the obj to the spark cluster to be used on spark workers

unpack() → List[ndarray]

read all of the memmap arrays and return as a list

delex.storage.sorted_set module

class delex.storage.sorted_set.MemmapSortedSets

Bases: MemmapSeqs

a class for storing sorted sets of token ids (as arrays)

class CacheKey(index_col: str, search_col: str | None, tokenizer_type: str)

Bases: CachedObjectKey

index_col: str
search_col: str | None
tokenizer_type: str
classmethod build(df: DataFrame, col: str, id_col: str = '_id')

Create a new MemmapSortedSets over tokens in df[col] and writing to disk

cosine(query: ndarray, ids: ndarray) → ndarray

compute cosine score between query and the sequences referenced by ids

Parameters:

  • query (np.ndarray) – a sorted unique array of token ids

  • ids (np.ndarray) – an array of ids of token sets in self

Returns:

scores[i] = cosine(query, token_sets[ids[i]]) if ids[i] is in token_sets else scores[i] = np.nan

Return type:

an array of scores where

jaccard(query: ndarray, ids: ndarray) → ndarray

compute jaccard score between query and the sequences referenced by ids

Parameters:

  • query (np.ndarray) – a sorted unique array of token ids

  • ids (np.ndarray) – an array of ids of token sets in self

Returns:

scores[i] = jaccard(query, token_sets[ids[i]]) if ids[i] is in token_sets else scores[i] = np.nan

Return type:

an array of scores where

overlap_coeff(query: ndarray, ids: ndarray) → ndarray

compute overlap_coefficient score between query and the sequences referenced by ids

Parameters:

  • query (np.ndarray) – a sorted unique array of token ids

  • ids (np.ndarray) – an array of ids of token sets in self

Returns:

scores[i] = overlap_coefficient(query, token_sets[ids[i]]) if ids[i] is in token_sets else scores[i] = np.nan

Return type:

an array of scores where

delex.storage.span_map module

delex.storage.span_map.create_span_map(keys, offsets, lengths, load_factor=0.75)

create a new span map of for keys, offsets, and lengths

Return type:

np.ndarray

delex.storage.span_map.span_map_get_key(arr, key)

get the entry from the span map, return the offset and length as a tuple

delex.storage.span_map.span_map_insert_key(arr, key, offset, length)

insert a single key into the span_map arr

delex.storage.span_map.span_map_insert_keys(arr, keys, offsets, lengths)

insert many keys into the span_map arr

delex.storage.string_store module

class delex.storage.string_store.MemmapStrings

Bases: MemmapSeqs

class CacheKey(index_col: str)

Bases: CachedObjectKey

index_col: str
classmethod build(df, col, id_col='_id')

create a MemmapSeqs instance from a spark dataframe

Parameters:

  • df (pyspark.sql.DataFrame) – the dataframe containing the sequences and ids

  • seq_col (str) – the name of the column in df that contains the sequences, e.g. strings, arrays

  • dtype (type) – the dtype of the elements in seq_col

  • id_col (str) – the name of the column in df that contains the ids for retrieving the sequences

Return type:

MemmapSeqs

fetch(i)

retrieve the sequence associated with i

Return type:

np.ndarray if i is found, else None

fetch_bytes(i)

delex.storage.vector_store module

class delex.storage.vector_store.MemmapVectorStore

Bases: MemmapSeqs

a class for storing sorted sets of token ids (as arrays)

class CacheKey(index_col: str, search_col: str | None, tokenizer_type: str)

Bases: CachedObjectKey

index_col: str
search_col: str | None
tokenizer_type: str
static arrays_to_encoded_sparse_vector(ind: ndarray, val: ndarray) → bytes
classmethod build(df: DataFrame, seq_col: str, id_col: str = '_id')

create a MemmapSeqs instance from a spark dataframe

Parameters:

  • df (pyspark.sql.DataFrame) – the dataframe containing the sequences and ids

  • seq_col (str) – the name of the column in df that contains the sequences, e.g. strings, arrays

  • dtype (type) – the dtype of the elements in seq_col

  • id_col (str) – the name of the column in df that contains the ids for retrieving the sequences

Return type:

MemmapSeqs

static decode_sparse_vector(bin: bytes) → ndarray
dot(query: ndarray, ids: ndarray) → ndarray

compute cosine score between query and the sequences referenced by ids

Parameters:

  • query (np.ndarray) – a sorted unique array of token ids

  • ids (np.ndarray) – an array of ids of token sets in self

Returns:

scores[i] = cosine(query, token_sets[ids[i]]) if ids[i] is in token_sets else scores[i] = np.nan

Return type:

an array of scores where

fetch(i: int, /) → ndarray | None

retrieve the sequence associated with i

Return type:

np.ndarray if i is found, else None

vector_dtype = dtype([('ind', '<i4'), ('val', '<f4')])
delex.storage.vector_store.iter_spark_rows(df, prefetch_size: int)

Module contents