Skip to content
Query.Farm
Talk with Us

Filter pushdown

On this page

Receiving and evaluating pushed-down WHERE predicates.

source
type AndFilter struct {
columnName string
columnIndex int
Children []Filter
}

Description

AndFilter is a conjunction of child filters. All children must pass.

Methods

source
func (f *AndFilter) ColumnIndex() int

ColumnIndex returns the index of the column this filter applies to.

source
func (f *AndFilter) ColumnName() string

ColumnName returns the name of the column this filter applies to.

source
func (f *AndFilter) Evaluate(ctx context.Context, batch arrow.RecordBatch) (arrow.Array, error)

Evaluate combines its child filters with a Kleene logical AND, returning a boolean array that is true only where every child passes.

source
func (f *AndFilter) Type() FilterType

Type returns the filter type identifier, FilterAnd.

source
type ColumnBounds struct {
// MinValue is the minimum bound value, or nil if unbounded below.
MinValue scalar.Scalar
// MinInclusive is true if MinValue is inclusive (>=), false if exclusive (>).
MinInclusive bool
// MaxValue is the maximum bound value, or nil if unbounded above.
MaxValue scalar.Scalar
// MaxInclusive is true if MaxValue is inclusive (<=), false if exclusive (<).
MaxInclusive bool
}

Description

ColumnBounds represents numeric/comparable bounds for a column extracted from comparison filters.

source
type ComparisonOp string

Description

ComparisonOp identifies a comparison operator used in constant filters.

Methods

source
func (op ComparisonOp) Symbol() string

Symbol returns the SQL operator string for this comparison op.

source
func (op ComparisonOp) computeFuncName() string

computeFuncName returns the arrow compute function name for this operator.

source
type ConstantFilter struct {
columnName string
columnIndex int
Op ComparisonOp
Value scalar.Scalar
}

Description

ConstantFilter compares a column against a constant value.

Methods

source
func (f *ConstantFilter) ColumnIndex() int

ColumnIndex returns the index of the column this filter applies to.

source
func (f *ConstantFilter) ColumnName() string

ColumnName returns the name of the column this filter applies to.

source
func (f *ConstantFilter) Evaluate(ctx context.Context, batch arrow.RecordBatch) (arrow.Array, error)

Evaluate compares the column against the constant value using the configured comparison operator, returning a boolean array of matches.

source
func (f *ConstantFilter) Type() FilterType

Type returns the filter type identifier, FilterConstant.

source
type ExpressionFilter struct {
columnName string
columnIndex int
Expr *exprNode
// Values holds the scalars referenced by constant-node value_ref indices.
// Populated at parse time from the filter batch's value-ref columns.
Values []scalarValueRef
}

Description

ExpressionFilter is a recursive expression tree pushed from DuckDB. The worker evaluates it via an embedded DuckDB connection (which can load the spatial extension for geometry predicates). Mirrors vgi-python’s ExpressionFilter in table_filter_pushdown.py.

Methods

source
func (f *ExpressionFilter) ColumnIndex() int

ColumnIndex returns the index of the column the filter applies to.

source
func (f *ExpressionFilter) ColumnName() string

ColumnName returns the name of the column the filter applies to.

source
func (f *ExpressionFilter) Evaluate(ctx context.Context, batch arrow.RecordBatch) (arrow.Array, error)

Evaluate renders the expression tree to SQL, feeds the input batch to a local DuckDB, and runs SELECT (<expr>)::BOOLEAN. Returns a Boolean array of length batch.NumRows().

source
func (f *ExpressionFilter) Type() FilterType

Type returns the filter type, which is always FilterExpression.

source
type Filter interface {
// ColumnName returns the name of the column this filter applies to.
ColumnName() string
// ColumnIndex returns the index of the column in the output schema.
ColumnIndex() int
// Type returns the filter type identifier.
Type() FilterType
// Evaluate evaluates the filter against a record batch, returning a
// boolean array where true indicates the row passes the filter.
Evaluate(ctx context.Context, batch arrow.RecordBatch) (arrow.Array, error)
}

Description

Filter is the interface all pushdown filter types implement.

Methods

source
func parseFilter(spec filterSpec, getValue func(int) (scalar.Scalar, error), getJoinKey func(string) arrow.Array) (Filter, error)
source
func parseFilterWithBatch(spec filterSpec, getValue func(int) (scalar.Scalar, error), getJoinKey func(string) arrow.Array, batch arrow.RecordBatch) (Filter, error)

parseFilterWithBatch is like parseFilter but also provides access to the raw filter batch so expression filters can harvest their value-ref columns (including Arrow extension metadata such as geoarrow.wkb).

source
func withColumnIndex(f Filter, idx int) Filter

withColumnIndex returns a new filter with the column index adjusted. This is used by StructFilter to evaluate child filters at index 0.

source
type FilterPrimitive interface {
~string | ~int64 | ~int32 | ~int16 | ~int8 |
~uint64 | ~uint32 | ~uint16 | ~uint8 |
~float64 | ~float32 | ~bool
}

Description

FilterPrimitive constrains the value type that EqualOrInValues / EqualValue can extract. Covers the scalar types DuckDB serializes through the pushdown filter wire protocol.

source
type FilterType string

Description

FilterType identifies the kind of pushdown filter.

source
type InFilter struct {
columnName string
columnIndex int
Values arrow.Array
}

Description

InFilter checks whether a column value is in a set of values.

Methods

source
func (f *InFilter) ColumnIndex() int

ColumnIndex returns the index of the column this filter applies to.

source
func (f *InFilter) ColumnName() string

ColumnName returns the name of the column this filter applies to.

source
func (f *InFilter) Evaluate(ctx context.Context, batch arrow.RecordBatch) (arrow.Array, error)

Evaluate tests whether each column value is a member of the filter’s value set, returning a boolean array that is true for matching rows.

source
func (f *InFilter) Type() FilterType

Type returns the filter type identifier, FilterIn.

source
type IsNotNullFilter struct {
columnName string
columnIndex int
}

Description

IsNotNullFilter checks whether a column value is not null.

Methods

source
func (f *IsNotNullFilter) ColumnIndex() int

ColumnIndex returns the index of the column this filter applies to.

source
func (f *IsNotNullFilter) ColumnName() string

ColumnName returns the name of the column this filter applies to.

source
func (f *IsNotNullFilter) Evaluate(ctx context.Context, batch arrow.RecordBatch) (arrow.Array, error)

Evaluate tests each column value for non-null, returning a boolean array that is true where the value is not null.

source
func (f *IsNotNullFilter) Type() FilterType

Type returns the filter type identifier, FilterIsNotNull.

source
type IsNullFilter struct {
columnName string
columnIndex int
}

Description

IsNullFilter checks whether a column value is null.

Methods

source
func (f *IsNullFilter) ColumnIndex() int

ColumnIndex returns the index of the column this filter applies to.

source
func (f *IsNullFilter) ColumnName() string

ColumnName returns the name of the column this filter applies to.

source
func (f *IsNullFilter) Evaluate(ctx context.Context, batch arrow.RecordBatch) (arrow.Array, error)

Evaluate tests each column value for null, returning a boolean array that is true where the value is null.

source
func (f *IsNullFilter) Type() FilterType

Type returns the filter type identifier, FilterIsNull.

source
type OrFilter struct {
columnName string
columnIndex int
Children []Filter
}

Description

OrFilter is a disjunction of child filters. At least one child must pass.

Methods

source
func (f *OrFilter) ColumnIndex() int

ColumnIndex returns the index of the column this filter applies to.

source
func (f *OrFilter) ColumnName() string

ColumnName returns the name of the column this filter applies to.

source
func (f *OrFilter) Evaluate(ctx context.Context, batch arrow.RecordBatch) (arrow.Array, error)

Evaluate combines its child filters with a Kleene logical OR, returning a boolean array that is true where at least one child passes.

source
func (f *OrFilter) Type() FilterType

Type returns the filter type identifier, FilterOr.

source
type PushdownFilters struct {
Filters []Filter
Version string
}

Description

PushdownFilters holds the deserialized pushdown filters for a function call.

Methods

source
func (pf *PushdownFilters) Apply(ctx context.Context, batch arrow.RecordBatch) (arrow.RecordBatch, error)

Apply applies all filters to the batch, returning a filtered batch.

source
func DeserializeFilters(batch arrow.RecordBatch, joinKeys …map[string]arrow.Array) (*PushdownFilters, error)

DeserializeFilters deserializes a pushdown filters record batch into a PushdownFilters container with a typed filter AST. Join-keys InFilters are resolved by name against joinKeys (name -> column) when provided.

source
func (pf *PushdownFilters) Evaluate(ctx context.Context, batch arrow.RecordBatch) (arrow.Array, error)

Evaluate evaluates all filters against the batch, returning a boolean mask. Filters are combined with AND at the top level.

source
func (pf *PushdownFilters) FilteredColumns() map[string]struct{}

FilteredColumns returns the set of column names that have filters applied.

source
func (pf *PushdownFilters) GetColumnBounds(name string) *ColumnBounds

GetColumnBounds extracts numeric bounds from comparison filters on the named column. Returns nil if no bounds can be determined.

source
func (pf *PushdownFilters) GetColumnConstant(name string) scalar.Scalar

GetColumnConstant returns the constant value if the column has an equality filter, or nil if no equality filter exists.

source
func (pf *PushdownFilters) GetColumnFilters(name string) []Filter

GetColumnFilters returns all top-level filters for a specific column.

source
func (pf *PushdownFilters) GetColumnInValues(name string) arrow.Array

GetColumnInValues returns the IN filter values for a column, or nil if no IN filter exists.

source
func (pf *PushdownFilters) GetColumnValues(name string) arrow.Array

GetColumnValues returns discrete values a column could have based on equality or IN filters. For EQ, wraps the value in a 1-element array. Returns nil if no discrete values can be determined.

Descends one level into AndFilter children (via collectColumnFilters), consistent with GetColumnBounds: DuckDB pushes col = v / col IN (...) conjoined with derived range bounds as a single AndFilter (e.g. a semi-join emits col IN (...) AND col >= min AND col <= max). Without the descent the discrete-value fast path silently misses those and pruning callers fall back to scanning every partition.

An OrFilter resolves to the UNION of its branches, but only when every branch pins this column to discrete values; if any branch is a range/IS NULL or constrains a different column the set is not enumerable and we return nil.

source
func (pf *PushdownFilters) HasFilterForColumn(name string) bool

HasFilterForColumn returns true if any filter constrains the given column.

source
func (pf *PushdownFilters) Repr() string

Repr returns a Python-style repr of the filter set, matching the vgi-python PushdownFilters.repr output. Useful for diagnostic display (e.g., the dynamic_filter_echo example) and stable regression testing. Format:

PushdownFilters([]) when empty
PushdownFilters([Filter1, Filter2, ...]) otherwise
source
func (pf *PushdownFilters) ToSQL(quoteIdentifier func(string) string, placeholder string) (string, []interface{})

ToSQL converts filters to a SQL WHERE clause with parameters. The quoteIdentifier function is used to quote column names (default: double quotes). The placeholder is the parameter placeholder style (“?”, “%s”, etc.). Returns the clause (excluding “WHERE” keyword) and parameter values.

source
func (pf *PushdownFilters) collectColumnFilters(name string) []Filter

collectColumnFilters collects filters for a column from top-level and direct AND children.

source
type StructFilter struct {
columnName string
columnIndex int
ChildIndex int
ChildName string
ChildFilter Filter
}

Description

StructFilter filters on a nested field within a struct column.

Methods

source
func (f *StructFilter) ColumnIndex() int

ColumnIndex returns the index of the column this filter applies to.

source
func (f *StructFilter) ColumnName() string

ColumnName returns the name of the column this filter applies to.

source
func (f *StructFilter) Evaluate(ctx context.Context, batch arrow.RecordBatch) (arrow.Array, error)

Evaluate applies the child filter to the nested struct field, returning the boolean array produced by evaluating that field.

source
func (f *StructFilter) Type() FilterType

Type returns the filter type identifier, FilterStruct.

source
type driver_value = driver.Value
source
type exprNode struct {
Type string `json:"expr_type"`
Index int `json:"index,omitempty"`
ValueRef *int `json:"value_ref,omitempty"`
FunctionName string `json:"function_name,omitempty"`
Children []exprNode `json:"children,omitempty"`
Op string `json:"op,omitempty"`
Left *exprNode `json:"left,omitempty"`
Right *exprNode `json:"right,omitempty"`
Conjunction string `json:"conjunction_type,omitempty"`
}

Methods

source
func (n *exprNode) toSQL(columnName string, schema *arrow.Schema, values []scalarValueRef) (string, error)

toSQL converts the expression tree to a SQL string. Column references resolve to the (double-quoted) filter column name. Constants are rendered from the scalar-values array (resolved at filter-parse time).

source
type filterSpec struct {
Type string `json:"type"`
ColumnName string `json:"column_name"`
ColumnIndex int `json:"column_index"`
Op string `json:"op,omitempty"`
ValueRef *int `json:"value_ref,omitempty"`
KeysColumn string `json:"keys_column,omitempty"`
Children []filterSpec `json:"children,omitempty"`
ChildIndex int `json:"child_index,omitempty"`
ChildName string `json:"child_name,omitempty"`
ChildFilter *filterSpec `json:"child_filter,omitempty"`
Expr *json.RawMessage `json:"expr,omitempty"`
}
source
type scalarValueRef struct {
value any // Go scalar (int64, float64, string, []byte, nil)
hex string // hex string for binary values
wkb bool // true if the value comes from a geoarrow.wkb column
}

Description

scalarValueRef captures one scalar value + its Arrow field metadata (so geoarrow.wkb constants can wrap into ST_GeomFromHEXWKB).

Methods

source
func resolveScalarValueRef(batch arrow.RecordBatch, ref int) (scalarValueRef, error)

resolveScalarValueRef extracts one scalar from the filter batch at column index ref+1 (column 0 holds the JSON specs).

source
func EqualOrInValues[T FilterPrimitive](pf *PushdownFilters, column string) ([]T, bool)

EqualOrInValues returns the values from a single column = X ConstantFilter, a single column IN (...) InFilter, or an AND-conjunction of such filters (all on the same column). Returns ok=false if the column has any other filter shape (range comparisons, IS NULL/NOT NULL, OR, nested struct, etc.) — callers should fall back to PushdownFilters.Apply for those.

The fast path eliminates the per-worker type-switch over Filter.(type) and the repeated scalar-to-Go-value conversion code that workers otherwise write to support filter pushdown on string / integer key columns.

Returns a deduplicated, order-preserving slice of values.

source
func EqualValue[T FilterPrimitive](pf *PushdownFilters, column string) (T, bool)

EqualValue is the single-value variant: returns the value of a single column = X ConstantFilter, or ok=false otherwise.

source
func HasOnlyEqualOrInOn(pf *PushdownFilters, columns …string) bool

HasOnlyEqualOrInOn returns true when every filter in pf is an eq/in (or AND-of-those) on one of the named columns. Useful for “either I can fast- path everything, or fall back to a slow path entirely” decisions.

source
func appenderForConn(ctx context.Context, conn *sql.Conn, tableName string) (*duckdb.Appender, error)
source
func arrayToGoSlice(arr arrow.Array) []interface{}
source
func arrowToDriverValue(col arrow.Array, i int, field arrow.Field) (interface{}, error)
source
func arrowValueAtForList(col arrow.Array, i int) (interface{}, error)

arrowValueAtForList extracts a Go scalar from a list’s child array.

source
func batchHasDictionary(batch arrow.RecordBatch) bool

batchHasDictionary reports whether any column of batch is dictionary-encoded.

source
func buildCreateTableSQL(name string, schema *arrow.Schema) (string, error)
source
func collectEqOrIn[T FilterPrimitive](f Filter, out *[]T, seen map[T]struct{}) bool

collectEqOrIn walks one filter, appending typed values to *out. Returns false if the filter (or any AND child) is not an eq/in. AND of eq/in is treated as the intersection-by-listing; callers usually want a single column filter so this composes naturally.

source
func collectExprValueRefs(n *exprNode, out map[int]bool)

collectExprValueRefs walks the expression tree collecting every value_ref index referenced by constant nodes.

source
func duckDBTypeFor(f arrow.Field) (string, error)
source
func ensureEvalDB() (*sql.DB, error)
source
func escapeIdent(s string) string
source
func evalExpressionAgainstBatch(ctx context.Context, batch arrow.RecordBatch, sqlExpr string) (arrow.Array, error)

evalExpressionAgainstBatch feeds a RecordBatch to embedded DuckDB as a temporary table, runs SELECT (<sqlExpr>)::BOOLEAN, and returns the result mask as an Arrow Boolean array.

The batch is staged via a one-row-at-a-time INSERT to avoid depending on duckdb’s Arrow integration (which requires a binary build matching the driver). This is slower than the Python version’s from_arrow but correctness-first; a future optimisation could use the appender API.

source
func filterArrayTo[T FilterPrimitive](a arrow.Array) ([]T, bool)

filterArrayTo extracts all non-null values from an Arrow array as []T. Returns ok=false on type mismatch with the target Go type.

source
func filterColumn(ctx context.Context, col, mask arrow.Array, opts compute.FilterOptions) (arrow.Array, error)

filterColumn filters a single column by mask. Dictionary columns filter their primitive index array (supported by the take kernel) and rebuild with the same dictionary values, preserving the dictionary encoding.

source
func filterIsEqOrInOn(f Filter, allowed map[string]struct{}) bool
source
func filterRecordBatchDictAware(ctx context.Context, batch arrow.RecordBatch, mask arrow.Array) (arrow.RecordBatch, error)

filterRecordBatchDictAware filters batch by mask column-by-column, handling dictionary columns the Go arrow fork’s array_take kernel cannot take.

source
func filterScalarTo[T FilterPrimitive](s scalar.Scalar) (T, bool)

filterScalarTo converts an Arrow scalar.Scalar to a concrete Go value of type T. Returns ok=false for type mismatches or invalid scalars.

source
func filterToSQL(f Filter, quote func(string) string, placeholder string) (string, []interface{})
source
func insertBatchToDuckDB(ctx context.Context, conn *sql.Conn, name string, batch arrow.RecordBatch) error
source
func isOperatorName(name string) bool
source
func isWKBField(f arrow.Field) bool

isWKBField returns true if the Arrow field is flagged with the geoarrow.wkb extension name.

source
func makeBoolArray(value bool, length int) arrow.Array

makeBoolArray creates a boolean array of constant value.

source
func nextEvalID() int64
source
func orDiscreteValues(or *OrFilter, name string) arrow.Array

orDiscreteValues returns the deduplicated union of discrete values for the named column across all OR branches, or nil if any branch leaves the column unbounded (a range/IS NULL branch, or a branch constraining a different column). Unlike the AND case, returning one branch’s values would be an unsafe subset — a pruning caller would skip the other branches’ rows. Descends one level only, consistent with collectColumnFilters.

source
func renderConstantRef(n *exprNode, values []scalarValueRef) (string, error)

renderConstantRef produces a SQL literal for a constant node by looking up its value_ref index in the value array.

source
func reprFilter(f Filter) string
source
func scalarGreater(a, b scalar.Scalar) bool
source
func scalarGreaterEqual(a, b scalar.Scalar) bool
source
func scalarLess(a, b scalar.Scalar) bool
source
func scalarLessEqual(a, b scalar.Scalar) bool
source
func scalarToAny(s scalar.Scalar) (any, bool)

scalarToAny extracts the underlying Go value from an Arrow scalar, returning values typed as the matching Go primitive for the FilterPrimitive set.

source
func scalarToArray(s scalar.Scalar) arrow.Array
source
func scalarToFloat64(s scalar.Scalar) (float64, bool)
source
func scalarToGo(s scalar.Scalar) interface{}