Filter pushdown

On this page

Receiving the predicates DuckDB pushed toward the scan.

interface AndFilter

source

export interface AndFilter

Fields

type“and”

columnNamestring

columnIndexnumber

childrenFilter[]

function buildJoinKeysLookup

source

export function buildJoinKeysLookup(
joinKeyBatches: VgiBatch[],
): (columnName: string) => any[] | null

Description

Build a getJoinKeysColumn lookup from an InitRequest’s joinKeys batches. Each batch typically has a single column whose name is the join-keys column referenced by spec.keys_column on a join_keys filter.

enum ComparisonOp

source

export enum ComparisonOp {
EQ = "eq",
NE = "ne",
GT = "gt",
GE = "ge",
LT = "lt",
LE = "le",
}

interface ConstantFilter

source

export interface ConstantFilter

Fields

type“constant”

columnNamestring

columnIndexnumber

opComparisonOp

valueany

function deserializeFilters

source

export function deserializeFilters(
batch: VgiBatch,
getJoinKeysColumn?: (columnName: string) => any[] | null,
): PushdownFilters

Description

Deserialize pushdown filters from an Arrow RecordBatch.

Wire format:

• Column 0: UTF8 — JSON array of filter specs. Field metadata: vgi_filter_version: “1”
• Columns 1+: Scalar values referenced by value_ref → column N+1

type Filter

source

export type Filter =
| ConstantFilter
| IsNullFilter
| IsNotNullFilter
| InFilter
| AndFilter
| OrFilter
| StructFilter
| ExpressionFilter;

class FilteringOutputCollector

source

export class FilteringOutputCollector

Description

Wraps an OutputCollector to automatically apply pushdown filters to every emitted batch.

Properties

property outputSchema

source

get outputSchema(): VgiSchema

property finished

source

get finished(): boolean

Methods

method emit

source

emit(batch: VgiBatch, metadata?: Map<string, string>): void;

method emitRow

source

emitRow(values: Record<string, any>): void

method finish

source

finish(): void

method clientLog

source

clientLog(
  level: string,
  message: string,
  extra?: Record<string, string>,
): void

function formatPushedFilters

source

export function formatPushedFilters(filters: PushdownFilters | undefined): string

Description

Format pushdown filters as a human-readable SQL-like string. Returns “(none)” when no filters exist.

interface InFilter

source

export interface InFilter

Fields

type“in”

columnNamestring

columnIndexnumber

valuesSet<any>

interface IsNotNullFilter

source

export interface IsNotNullFilter

Fields

type“is_not_null”

columnNamestring

columnIndexnumber

interface IsNullFilter

source

export interface IsNullFilter

Fields

type“is_null”

columnNamestring

columnIndexnumber

interface OrFilter

source

export interface OrFilter

Fields

type“or”

columnNamestring

columnIndexnumber

childrenFilter[]

class PushdownFilters

source

export class PushdownFilters

Methods

method toSql

source

toSql(): string

Convert filters to a human-readable SQL-like string.

method evaluate

source

evaluate(batch: VgiBatch): Uint8Array

Evaluate filters against a batch, returning a Uint8Array mask (0=fail, 1=pass).

method apply

source

apply(batch: VgiBatch): VgiBatch

Apply filters to a batch, returning only passing rows.

method filteredColumns

source

filteredColumns(): string[]

The set of column names referenced by the pushed-down filters, sorted ascending. Descends one level into AndFilter children (DuckDB pushes col IN (...) AND col>=min as a single AndFilter), consistent with getColumnValues / hasFilterForColumn. Mirrors the cross-language filtered_columns accessor (vgi-python / vgi-go / vgi-rust).

method hasFilterForColumn

source

hasFilterForColumn(columnName: string): boolean

Whether any pushed-down filter references columnName. Mirrors the cross-language has_filter_for_column accessor.

method getColumnValues

source

getColumnValues(columnName: string): any[] | null

Discrete values a column could take, from equality (=) or IN filters. Useful for partition pruning (resolve the key set up front, fetch only those keys). Returns null when the predicate is not enumerable — no filter, a bare range, or an OR with a non-discrete branch.

Mirrors vgi-python’s PushdownFilters.get_column_values: descends one level into an AndFilter (DuckDB pushes col IN (...) AND col>=min AND col<=max as a single AndFilter from a semi-join), and unions OR branches only when every branch pins the column to discrete values.

function reprPushedFilters

source

export function reprPushedFilters(filters: PushdownFilters | undefined): string

Description

Format pushdown filters using Python-style repr() — e.g. PushdownFilters([ConstantFilter(n < 4999)]). Matches vgi-python’s _format_pushed_filters_safe output so tests that pattern-match the repr (e.g. dynamic_filter checking for ConstantFilter(n <) work identically across workers.

interface StructFilter

source

export interface StructFilter

Fields

type“struct”

columnNamestring

columnIndexnumber

childIndexnumber

childNamestring

childFilterFilter