vgi.table_filter_pushdown

Module overview

Filter pushdown AST classes for table functions.

This module provides:

• Filter AST classes for representing pushdown filter predicates
• ColumnBounds for extracting numeric bounds from filters
• PushdownFilters container with evaluation and helper methods
• Deserialization from Arrow IPC format

Filter types

ConstantFilter: Comparison with a constant value (=, !=, >, >=, <, <=) IsNullFilter: IS NULL check IsNotNullFilter: IS NOT NULL check InFilter: Set membership (IN clause) AndFilter: Conjunction of child filters OrFilter: Disjunction of child filters StructFilter: Nested struct field filter

class AndFilter

source

Bases: Filter

Description

Conjunction of child filters.

All child filters must pass for a row to pass.

Attributes

attribute children

tuple[Filter, …]

The child filters that are ANDed together.

Methods

method evaluate

source

evaluate(batch: pa.RecordBatch) -> pa.BooleanArray

Evaluate AND of all child filters.

Parameters

batch

RecordBatch to evaluate the filter against.

Returns

Boolean array with True for rows passing every child filter.

Inherited members (2)

• column_name attribute · from Filter
• column_index attribute · from Filter

class ColumnBounds

source

Description

Numeric/comparable bounds for a column extracted from filters.

Use case: Partition pruning, index range scans, bounded data fetches.

Attributes

attribute min_value

pa.Scalar[Any] | None

Minimum bound value, or None if unbounded below.

attribute min_inclusive

bool

True if min_value is inclusive (>=), False if exclusive (>).

attribute max_value

pa.Scalar[Any] | None

Maximum bound value, or None if unbounded above.

attribute max_inclusive

bool

True if max_value is inclusive (<=), False if exclusive (<).

Methods

method contains

source

contains(value: Any) -> bool

Check if a value satisfies these bounds.

Parameters

value

Value to check against bounds.

Returns

True if value is within bounds, False otherwise.

class ColumnRefNode

source

Bases: ExpressionNode

Description

Column reference node.

Note: In v1, all column refs in an expression filter refer to the same column (the filter column). The index is stored for future multi-column support but to_sql() always uses the filter’s column_name.

Attributes

attribute index

int

Column position; reserved for future multi-column support.

Methods

method to_sql

source

to_sql(column_name: str) -> str

Return quoted column name with double-quote escaping.

Parameters

column_name

Name of the filter column to render.

Returns

The column name as a double-quoted SQL identifier.

Inherited members (1)

• expr_type attribute · from ExpressionNode

class ComparisonNode

source

Bases: ExpressionNode

Description

Comparison node (left op right).

Attributes

attribute op

ComparisonOp

The comparison operator applied between left and right.

attribute left

ExpressionNode

The left-hand operand expression node.

attribute right

ExpressionNode

The right-hand operand expression node.

Methods

method to_sql

source

to_sql(column_name: str) -> str

Format as (left op right).

Parameters

column_name

Name of the filter column to substitute for column refs.

Returns

The comparison rendered as a parenthesized SQL fragment.

Inherited members (1)

• expr_type attribute · from ExpressionNode

class ComparisonOp

source

Bases: Enum

Description

Comparison operators for constant filters.

Attributes

attribute EQ

Equality (=).

attribute NE

Inequality (!=).

attribute GT

Greater than (>).

attribute GE

Greater than or equal (>=).

attribute LT

Less than (<).

attribute LE

Less than or equal (<=).

attribute symbol

str

Return the SQL symbol for this operator.

class ConjunctionNode

source

Bases: ExpressionNode

Description

AND/OR conjunction node.

Attributes

attribute conjunction_type

str

Either "and" or "or", selecting how the children combine.

attribute children

tuple[ExpressionNode, …]

The child expression nodes being combined.

Methods

method to_sql

source

to_sql(column_name: str) -> str

Format as (child1 AND/OR child2 AND/OR …).

Parameters

column_name

Name of the filter column to substitute for column refs.

Returns

The conjunction rendered as a parenthesized SQL fragment.

Inherited members (1)

• expr_type attribute · from ExpressionNode

class ConstantFilter

source

Bases: Filter

Description

Comparison filter: column <op> value.

Attributes

attribute op

ComparisonOp

The comparison operator applied between the column and value.

attribute value

pa.Scalar[Any]

The constant scalar the column is compared against.

Methods

method evaluate

source

evaluate(batch: pa.RecordBatch) -> pa.BooleanArray

Evaluate comparison against batch column.

Parameters

batch

RecordBatch to evaluate the filter against.

Returns

Boolean array with True for rows that pass the comparison.

Inherited members (2)

• column_name attribute · from Filter
• column_index attribute · from Filter

class ConstantNode

source

Bases: ExpressionNode

Description

Constant value node.

Attributes

attribute value

pa.Scalar[Any]

The constant scalar literal.

attribute field

pa.Field[Any] | None

Arrow field carrying extension metadata for value (if available), used to render extension types correctly in SQL.

Methods

method to_sql

source

to_sql(column_name: str) -> str

Format Arrow scalar as SQL literal, using field metadata for extension types.

Parameters

column_name

Name of the filter column (unused for constants).

Returns

The constant rendered as a SQL literal.

Inherited members (1)

• expr_type attribute · from ExpressionNode

function deserialize_filters

source

deserialize_filters(
batch: pa.RecordBatch,
join_keys: list[pa.RecordBatch] | None = None,
) -> PushdownFilters

Deserialize Arrow IPC bytes to typed AST.

Parameters

batch

Arrow RecordBatch containing the serialized filters.

join_keys

Optional list of single-column Arrow RecordBatches, one per IN filter column. Each batch may have a different row count. Referenced by join_keys filter type entries in the filter spec.

Returns

PushdownFilters container with parsed filter AST.

Raises

FilterDeserializationError

If parsing fails.

FilterVersionError

If version is unsupported.

class ExpressionFilter

source

Bases: Filter

Description

Expression tree filter pushed from DuckDB.

Contains a recursive expression tree that the worker evaluates using DuckDB. Typical use: spatial predicates like geom && box.

Attributes

attribute expr

ExpressionNode

Root of the recursive expression tree to evaluate.

Methods

method evaluate

source

evaluate(batch: pa.RecordBatch) -> pa.BooleanArray

Evaluate expression tree against batch using DuckDB.

Uses a cached per-process DuckDB connection with spatial extension pre-loaded (if available). The engine is imported lazily via :mod:vgi._duckdb (haybarn preferred, duckdb fallback) — workers that don’t use expression filters don’t need either installed.

Parameters

batch

RecordBatch to evaluate the expression against.

Returns

Boolean array with True for rows passing the expression.

Inherited members (2)

• column_name attribute · from Filter
• column_index attribute · from Filter

class ExpressionNode

source

Description

Base class for expression tree nodes.

Subclasses must set expr_type to match their class. This field is used for serialization round-tripping (JSON expr_type key).

Attributes

attribute expr_type

ExpressionNodeType

Discriminator identifying the concrete node kind.

Methods

method to_sql

source

to_sql(column_name: str) -> str

Convert node to SQL string. Override in subclasses.

Parameters

column_name

Name of the filter column to substitute for column refs.

Returns

The node rendered as a SQL fragment.

class ExpressionNodeType

source

Bases: Enum

Description

Expression node type identifiers matching the JSON protocol.

Attributes

attribute COLUMN_REF

Reference to a column in the filtered table.

attribute CONSTANT

Constant literal value.

attribute FUNCTION

Function call or infix operator.

attribute COMPARISON

Binary comparison (left op right).

attribute CONJUNCTION

AND/OR conjunction of child nodes.

class Filter

source

Description

Base class for all filter types.

Attributes

attribute column_name

str

Name of the column this filter applies to.

attribute column_index

int

Index of the column in the output schema.

Methods

method evaluate

source

evaluate(batch: pa.RecordBatch) -> pa.BooleanArray

Evaluate filter against batch using PyArrow compute.

Parameters

batch

RecordBatch to evaluate filter against.

Returns

Boolean array with True for rows that pass the filter.

Raises

NotImplementedError

Base class does not implement evaluation.

class FilterDeserializationError

source

Bases: FilterError

Description

Failed to parse filter IPC bytes.

class FilterError

source

Bases: Exception

Description

Base exception for filter pushdown errors.

class FilterType

source

Bases: Enum

Description

Filter type identifiers matching the JSON protocol.

Attributes

attribute CONSTANT

Comparison against a constant value (=, !=, <, …).

attribute IS_NULL

IS NULL check.

attribute IS_NOT_NULL

IS NOT NULL check.

attribute IN

Set membership (IN clause).

attribute JOIN_KEYS

Membership against join-key values pushed as a separate batch.

attribute AND

Conjunction of child filters.

attribute OR

Disjunction of child filters.

attribute STRUCT

Filter on a nested field within a struct column.

attribute EXPRESSION

Arbitrary expression tree evaluated by DuckDB.

class FilterVersionError

source

Bases: FilterError

Description

Unsupported filter protocol version.

class FunctionNode

source

Bases: ExpressionNode

Description

Function call node.

Attributes

attribute function_name

str

Name of the function or infix operator to invoke.

attribute children

tuple[ExpressionNode, …]

The argument expression nodes.

Methods

method to_sql

source

to_sql(column_name: str) -> str

Format as function_name(args…) or infix for operators like &&.

Parameters

column_name

Name of the filter column to substitute for column refs.

Returns

The function call rendered as a SQL fragment.

Inherited members (1)

• expr_type attribute · from ExpressionNode

class InFilter

source

Bases: Filter

Description

IN (v1, v2, …) set membership filter.

Attributes

attribute values

pa.Array[Any]

The candidate values as an Arrow array (the contents of the list column); a row passes if its column value is in this set.

Methods

method evaluate

source

evaluate(batch: pa.RecordBatch) -> pa.BooleanArray

Evaluate IN membership against batch column.

Parameters

batch

RecordBatch to evaluate the filter against.

Returns

Boolean array with True for rows whose value is in the set.

Inherited members (2)

• column_name attribute · from Filter
• column_index attribute · from Filter

class IsNotNullFilter

source

Bases: Filter

Description

IS NOT NULL check filter.

Methods

method evaluate

source

evaluate(batch: pa.RecordBatch) -> pa.BooleanArray

Evaluate IS NOT NULL check against batch column.

Parameters

batch

RecordBatch to evaluate the filter against.

Returns

Boolean array with True for rows where the column is non-null.

Inherited members (2)

• column_name attribute · from Filter
• column_index attribute · from Filter

class IsNullFilter

source

Bases: Filter

Description

IS NULL check filter.

Methods

method evaluate

source

evaluate(batch: pa.RecordBatch) -> pa.BooleanArray

Evaluate IS NULL check against batch column.

Parameters

batch

RecordBatch to evaluate the filter against.

Returns

Boolean array with True for rows where the column is null.

Inherited members (2)

• column_name attribute · from Filter
• column_index attribute · from Filter

class OrFilter

source

Bases: Filter

Description

Disjunction of child filters.

At least one child filter must pass for a row to pass.

Attributes

attribute children

tuple[Filter, …]

The child filters that are ORed together.

Methods

method evaluate

source

evaluate(batch: pa.RecordBatch) -> pa.BooleanArray

Evaluate OR of all child filters.

Parameters

batch

RecordBatch to evaluate the filter against.

Returns

Boolean array with True for rows passing any child filter.

Inherited members (2)

• column_name attribute · from Filter
• column_index attribute · from Filter

class PushdownFilters

source

Description

Container for pushdown filters with evaluation and query helpers.

The top-level filters array represents a conjunction (AND). Each filter in the array must be satisfied for a row to pass. Individual filters may themselves be AND/OR compound filters for more complex expressions.

Provides:

• evaluate(batch) / apply(batch) - Apply filters using PyArrow compute
• get_column_bounds(name) - Extract numeric bounds for partition pruning
• get_column_constant(name) - Get equality constant for a column
• get_column_in_values(name) - Get IN list values
• get_column_filters(name) - Get all filters for a column
• to_sql() - Generate SQL WHERE clause

Attributes

attribute filters

tuple[Filter, …]

The top-level filters, combined with AND.

attribute version

str

Filter protocol version the filters were deserialized from.

attribute join_keys_batches

list[pa.RecordBatch] | None

Optional single-column batches of join-key values, one per IN/join-keys filter column, or None when none were pushed.

attribute filtered_columns

frozenset[str]

Set of column names that have filters applied.

Use case: Quick check of which columns are constrained.

Methods

method get_join_keys_batch

source

get_join_keys_batch() -> pa.RecordBatch | None

Return a merged join keys batch for temp table registration.

When all join key batches have the same row count (the semi-join case), returns a single RecordBatch with all columns merged. When batches have different row counts (independent IN filters), they cannot be merged, so this returns None.

For individual column access, use :meth:get_join_keys_batches or :meth:get_column_in_values.

Example:

keys = params.current_pushdown_filters.get_join_keys_batch()
if keys is not None:
  conn.register("join_keys", keys)
  result = conn.sql(
      "SELECT d.* FROM my_data d JOIN join_keys USING (id)"
  )

Returns

Merged RecordBatch when all batches have equal row counts, or None.

method get_join_keys_batches

source

get_join_keys_batches() -> list[pa.RecordBatch] | None

Return all join key batches (one per IN filter column).

Each batch is a single-column RecordBatch. Different batches may have different row counts. Returns None if no join keys were pushed.

Returns

The list of single-column join-key batches, or None.

method evaluate

source

evaluate(batch: pa.RecordBatch) -> pa.BooleanArray

Evaluate all filters, returning boolean mask.

Filters are combined with AND at the top level - a row passes only if ALL filters evaluate to true for that row.

Parameters

batch

RecordBatch to evaluate filters against.

Returns

Boolean array with True for rows that pass all filters.

method apply

source

apply(batch: pa.RecordBatch) -> pa.RecordBatch

Apply all filters to batch, returning filtered batch.

Parameters

batch

RecordBatch to filter.

Returns

Filtered RecordBatch containing only rows that pass all filters.

method get_column_filters

source

get_column_filters(column_name: str) -> list[Filter]

Get all top-level filters for a specific column.

Use case: Inspect what constraints apply to a column.

Parameters

column_name

Name of the column to get filters for.

Returns

List of filters that apply to the column.

method has_filter_for_column

source

has_filter_for_column(column_name: str) -> bool

Check if any filter constrains the given column.

Parameters

column_name

Name of the column to check.

Returns

True if at least one filter applies to the column.

method get_column_constant

source

get_column_constant(column_name: str) -> pa.Scalar[Any] | None

Get constant value if column has an equality filter.

Use case: Partition key lookup, exact match optimization.

Parameters

column_name

Name of the column to check.

Returns

The constant value if an equality filter exists, None otherwise.

method get_column_in_values

source

get_column_in_values(column_name: str) -> pa.Array[Any] | None

Get IN list values if column has an IN filter.

Use case: Multi-key lookup, batch fetching.

Parameters

column_name

Name of the column to check.

Returns

Arrow array of IN values if an IN filter exists, None otherwise.

method get_column_values

source

get_column_values(column_name: str) -> pa.Array[Any] | None

Get all distinct values a column could have based on filters.

Returns values from equality (=) or IN filters as an Arrow array. Useful for partition pruning when partitions are keyed by specific values.

Use case: Partition key lookup, directory-based partitioning.

Parameters

column_name

Name of the column to check.

Returns

Arrow array of discrete values if available, None otherwise.

method get_column_bounds

source

get_column_bounds(column_name: str) -> ColumnBounds | None

Extract numeric bounds from comparison filters.

Analyzes gt/ge/lt/le filters to determine value range.

Use case: Range scans, partition pruning, bounded iteration.

Parameters

column_name

Name of the column to extract bounds for.

Returns

ColumnBounds with min/max values if bounds exist, None otherwise.

method to_sql

source

to_sql(
quote_identifier: Callable[[str], str] | None = None,
placeholder: str = ‘?’,
) -> tuple[str, list[Any]]

Convert filters to SQL WHERE clause with parameters.

Parameters

quote_identifier

Function to quote column names (default: double quotes)

placeholder

Parameter placeholder style (“?”, “%s”, “:name”)

Returns

Tuple of (where_clause, params) - clause excludes “WHERE” keyword.

method empty

source

empty() -> PushdownFilters

Create an empty PushdownFilters instance (no filters).

Returns

A PushdownFilters with no filters.

class StructFilter

source

Bases: Filter

Description

Nested struct field filter.

Filters on a nested field within a struct column. Example: address.city = ‘Seattle’

Attributes

attribute child_index

int

Position of the nested field within the struct column.

attribute child_name

str

Name of the nested field within the struct column.

attribute child_filter

Filter

The filter applied to the nested field’s values.

Methods

method evaluate

source

evaluate(batch: pa.RecordBatch) -> pa.BooleanArray

Evaluate filter on nested struct field.

Parameters

batch

RecordBatch to evaluate the filter against.

Returns

Boolean array with True for rows passing the nested-field filter.

Inherited members (2)

• column_name attribute · from Filter
• column_index attribute · from Filter