Filter pushdown

On this page

Receiving the predicates DuckDB pushed toward the scan, and applying them.

enum ComparisonOperator

source

public enum ComparisonOperator

Description

The six comparison operators a PushdownFilter.Constant can carry. Wire tokens (eq, ne, …) come from the C++ extension’s filter-spec JSON.

Members

String wireToken()

The wire token for this operator (e.g. "eq").

String symbol()

SQL-like symbol used by the diagnostic format helpers.

ComparisonOperator fromWire(String token)

Resolve a wire token to its operator.

boolean test(int cmp)

Apply this operator to a Comparable#compareTo result.

boolean testEquality(boolean equal)

Apply this operator when only equality is known (unordered operands). Only EQ/NE yield a definite answer; ordered comparisons against unordered operands are treated as non-matching.

class FilterApplier

source

public final class FilterApplier

Description

Helper for table-function fixtures that opt into filter pushdown (FunctionMetadata.filterPushdown=true, autoApplyFilters=true).

Decoded once per init via #from and reused across emits — each #apply evaluates the pre-parsed filter AST against the batch and returns a compacted batch (closing the original on row drop).

Members

FilterApplier from(byte[] filterBytes, List<byte[]> joinKeysIpc)

Create an applier over the init-time pushdown payload; decoding is deferred to first #apply.

List<String> expressionPredicates()

The rendered SQL predicates of any pushed expression filters (spatial &&, list_contains, …). #apply only handles column filters; expression filters are pass-through there and must be applied separately via the worker’s expression evaluator.

VectorSchemaRoot apply(VectorSchemaRoot src)

Compact src to only rows that pass the parsed filters. Returns src unchanged when no filters were pushed; closes src and returns a new root when rows are dropped.

VectorSchemaRoot compact(VectorSchemaRoot src, boolean[] mask)

Compact src to only the rows whose mask entry is true. Returns src unchanged when every row is kept; otherwise builds a new root and closes src (ownership transfer). Shared by the column-filter path here and the worker’s expression-filter evaluator.

interface PushdownFilter

source

public sealed interface PushdownFilter permits PushdownFilter.Constant, PushdownFilter.IsNull, PushdownFilter.IsNotNull, PushdownFilter.In, PushdownFilter.And, PushdownFilter.Or, PushdownFilter.Struct, PushdownFilter.Expression

Description

Sealed AST for VGI filter-pushdown predicates. Mirrors vgi-go’s Filter hierarchy. The wire format is a JSON array of filter specs encoded as a single string column inside an Arrow batch; sibling columns carry the actual constant values referenced by value_ref indices.

Currently the AST is read-only — fixtures inspect filters and DuckDB applies them post-emit (auto-apply). A future pass will add per-filter Evaluate for fixtures that opt out of auto-apply.

enum PushdownFilterType

source

public enum PushdownFilterType

Description

Discriminator tag for a filter spec entry in the pushdown wire JSON.

Members

String wireToken()

The wire token for this filter type (e.g. "constant").

PushdownFilterType fromWire(String token)

Resolve a wire token, or null if it isn’t a recognised type.

record PushdownFilters

source

public record PushdownFilters(List<PushdownFilter> filters, String version)

Description

Container for the parsed top-level filter list (implicit AND).

Members

PushdownFilters empty()

An empty filter set at the current supported version.

String formatInline()

Format the filters as a human-readable AND-joined SQL-like string with values inlined. Used by diagnostic fixtures like filter_echo; matches vgi-go’s formatFiltersInline.

String formatRepr()

Python-repr-style format that wraps each leaf in its filter class name (e.g. ConstantFilter(n < 9500), IsNullFilter(c IS NULL)). The dynamic-filter integration test asserts LIKE '%ConstantFilter(n <%' on this representation to confirm the C++ extension extracted the dynamic bound from inside a ConjunctionAndFilter.

List<String> expressionPredicates()

The rendered SQL predicates of the top-level expression filters (e.g. ("geom" && ST_MakeEnvelope(...))). These can’t be evaluated row-at-a-time — #evaluate treats them as pass-through; the worker applies them via an embedded engine (see vgi-example-worker’s ExpressionFilterEvaluator).

java.util.Set<String> filteredColumns()

The set of column names referenced by the top-level filters (each filter’s column_name). Mirrors vgi-python’s PushdownFilters.filtered_columns.

boolean hasFilterForColumn(String name)

Whether any top-level filter constrains column name. Mirrors vgi-python’s PushdownFilters.has_filter_for_column.

List<PushdownFilter> filtersForColumn(String name)

Return the top-level filters that directly target column name. “Top-level” means an entry in #filters() — does not descend into nested And/Or/Struct, since those don’t admit a simple per-column extraction. Useful for fixtures that want to know “what bounds did DuckDB push down on column X?” without re-implementing the AST walk.

List<Object> directEqualityValues(String name)

Extract the set of values that name is definitely constrained to equal at the top level: collects eq-Constant values and In value lists. Returns an empty list when no such constraint exists; callers can treat that as “no direct equality push-down”.

java.util.Optional<List<Object>> getColumnValues(String name)

The discrete set of values name could take, when the pushed filters pin it to an enumerable set — an equality, an IN list, or an OR whose every branch pins name to discrete values (their union). Descends one level into a top-level And (see #collectColumnFilters). Returns empty when not enumerable (no filter, a bare range, an OR with a range/other-column branch, or deeper nesting) — the caller must then fall back to a full scan rather than an unsafe subset. Mirrors vgi-python’s get_column_values.

boolean[] evaluate(VectorSchemaRoot root)

Evaluate every filter against root and return a boolean mask (one entry per row) — true means the row passes all filters (top-level AND). The mask is a boolean[] so callers can drive fixture-side row filtering without an Arrow allocation.

class PushdownFiltersDecoder

source

public final class PushdownFiltersDecoder

Description

Decode pushdown-filter IPC bytes into a PushdownFilters AST.

Wire shape: a single Arrow record batch with column 0 holding a single UTF-8 string row containing the JSON-encoded filter spec list. Siblings carry the actual constant values referenced by value_ref indices (so the JSON stays type-agnostic).

Members

PushdownFilters decode(byte[] data)

Decode pushdown-filter IPC bytes with no join-key inputs.

PushdownFilters decode(byte[] data, List<byte[]> joinKeysIpc)

Decode pushdown-filter IPC bytes, resolving join_keys filters against the supplied out-of-band key batches.