delex.graph package¶
Submodules¶
delex.graph.algorithms module¶
- delex.graph.algorithms.clone_graph(nodes)¶
- delex.graph.algorithms.find_all_nodes(node)¶
- delex.graph.algorithms.find_sink(node)¶
- delex.graph.algorithms.topological_sort(sink_node)¶
delex.graph.node module¶
- class delex.graph.node.IntersectNode¶
Bases:
SetOpNodeintersect the output of all incoming edges and return a single set
- validate()¶
perform validation for this node, e.g. ensure that UnionNodes have multiple inputs, MinusNodes have two inputs, etc.
- Raises:
ValueError – if validation fails
- class delex.graph.node.MinusNode(left, right)¶
Bases:
SetOpNodecompute the set minus of two sets left - right
- property left¶
- property right¶
- validate()¶
perform validation for this node, e.g. ensure that UnionNodes have multiple inputs, MinusNodes have two inputs, etc.
- Raises:
ValueError – if validation fails
- class delex.graph.node.Node¶
Bases:
ABCabstract base class for all graph nodes
- add_in_edge(other)¶
add an edge between other -> self
- add_out_edge(other)¶
add an edge between self -> other
- ancestors() set¶
get all ancestors of this node
- Returns:
all the ancestors of this node
- Return type:
Set[Node]
- abstractmethod build(index_table: DataFrame, id_col: str, cache: BuildCache | None = None)¶
build this node over index_table using id_col as the unique id, optionally with cache
- Parameters:
index_table (pyspark.sql.DataFrame) – the dataframe that will be preprocessed / indexed
id_col (str) – the name of the unique id column in index_table
cache (Optional[BuildCache] = None) – the cache for built indexes and hash tables
- equivalent(other) bool¶
check self is equivalent to other, this does a recursive check and can be used to compare two graphs if self and other are both sinks
- Parameters:
other (Node) – the node to be compared to
- Return type:
True if equivalent else False
- abstractmethod execute(stream)¶
execute the operation of this node over a DataFrameStream and return a new DataFrameStream
- property id_string: str¶
a string and identifies this node for graph comparison without accounting for edges
- property in_degree: int¶
- insert_after(node)¶
insert node after this node, e.g.
self -> x becomes self -> node -> x
- Parameters:
node (Node) – the node to be inserted
- insert_before(node)¶
insert node before this node, e.g.
x -> self becomes x -> node -> self
- Parameters:
node (Node) – the node to be inserted
- property is_sink: bool¶
- property is_source: bool¶
- iter_dependencies() Iterator¶
return an iterator over the dependencies of this node
- iter_in() Iterator¶
return an iterator of nodes for the incoming edges
- iter_out() Iterator¶
return an iterator of nodes for the outgoing edges
- property out_degree: int¶
- property output_col: str¶
- pop()¶
remove this node from the graph and reconnect edges between in and out
- Return type:
self
- Raises:
RuntimeError – if self.out_degree > 1 and self.in_degree > 1
- remove_in_edge(other)¶
remove other -> self
- Raises:
KeyError – if other -> self doesn’t exist
- remove_in_edges()¶
remove all incoming edges, x -> self
- remove_out_edge(other)¶
remove self -> other
- Raises:
KeyError – if self -> other doesn’t exist
- remove_out_edges()¶
remove all outgoing edges, self -> x
- abstract property streamable: bool¶
True if the operation at this node can be streamed, else False
- abstractmethod validate()¶
perform validation for this node, e.g. ensure that UnionNodes have multiple inputs, MinusNodes have two inputs, etc.
- Raises:
ValueError – if validation fails
- abstractmethod working_set_size() dict¶
return the working set size of each component use for this node, dict values are None if self.build has not been called yet
- class delex.graph.node.PredicateNode(predicate)¶
Bases:
Nodea node that execute a Predicate
- build(index_table, id_col, cache=None)¶
build this node over index_table using id_col as the unique id, optionally with cache
- Parameters:
index_table (pyspark.sql.DataFrame) – the dataframe that will be preprocessed / indexed
id_col (str) – the name of the unique id column in index_table
cache (Optional[BuildCache] = None) – the cache for built indexes and hash tables
- execute(stream)¶
execute the operation of this node over a DataFrameStream and return a new DataFrameStream
- init()¶
- iter_dependencies()¶
return an iterator over the dependencies of this node
- property predicate¶
- property streamable¶
True if the operation at this node can be streamed, else False
- validate()¶
perform validation for this node, e.g. ensure that UnionNodes have multiple inputs, MinusNodes have two inputs, etc.
- Raises:
ValueError – if validation fails
- working_set_size() dict¶
return the working set size of each component use for this node, dict values are None if self.build has not been called yet
- class delex.graph.node.SetOpNode¶
Bases:
NodeBase Class for all set operations nodes
- build(index_table, id_col, cache=None)¶
build this node over index_table using id_col as the unique id, optionally with cache
- Parameters:
index_table (pyspark.sql.DataFrame) – the dataframe that will be preprocessed / indexed
id_col (str) – the name of the unique id column in index_table
cache (Optional[BuildCache] = None) – the cache for built indexes and hash tables
- execute(stream)¶
execute the operation of this node over a DataFrameStream and return a new DataFrameStream
- init()¶
- property streamable¶
True if the operation at this node can be streamed, else False
- working_set_size() dict¶
return the working set size of each component use for this node, dict values are None if self.build has not been called yet