Pushdown & statistics

On this page

Receiving the predicates DuckDB pushed toward the scan, and reporting what you know.

struct CatColStat

source

pub struct CatColStat {
  pub column_name: String,
  pub min: StatValue,
  pub max: StatValue,
  pub has_null: bool,
  pub has_not_null: bool,
  pub distinct_count: Option<i64>,
  pub contains_unicode: Option<bool>,
  pub max_string_length: Option<u64>,
}

Description

Optimizer statistics for one column.

struct ColumnBounds

source

pub struct ColumnBounds {
  pub min: Option<i64>,
  pub max: Option<i64>,
}

Description

Numeric [min, max] bounds on a column implied by pushed-down comparison filters (integer-coerced). Returned by [PushdownFilters::get_column_bounds]; mirrors the Python/Go ColumnBounds.

enum Filter

source

pub enum Filter {
  Constant { column_name: String, op: String },
  In { column_name: String },
  JoinKeys { column_name: String },
  IsNull { column_name: String },
  IsNotNull { column_name: String },
  And(Vec<Filter>),
  Or(Vec<Filter>),
  Struct {
      column_name: String,
      child_name: String,
      child: Box<Filter>,
  },
  Other { kind: String, column_name: String },
}

Description

A read-only view of one pushed-down filter — a structural projection of the internal filter AST, mirroring the Python/Go filter objects. Returned by [PushdownFilters::get_column_filters] / [PushdownFilters::filters].

Methods

method column_name

source

pub fn column_name(&self) -> &str

The column this filter references (empty string for And/Or).

constant PARTITION_COLUMN_KEY

source

pub const PARTITION_COLUMN_KEY: &str = “vgi.partition_column”;

Description

Field-metadata key marking a column as a VGI partition column.

constant PARTITION_VALUES_META

source

pub const PARTITION_VALUES_META: &str = “vgi_partition_values#b64”;

Description

Per-batch wire-metadata key carrying base64 partition (min,max) values.

struct PushdownFilters

source

pub struct PushdownFilters {
  specs: Vec<FilterSpec>,
  values: Vec<ArrayRef>, // value_ref N → values[N]
  join_keys: std::collections::HashMap<String, ArrayRef>,
}

Description

A parsed, evaluable set of pushdown filters.

Methods

method apply

source

pub fn apply(&self, batch: &RecordBatch) -> Result<RecordBatch>

Filter batch to the rows that satisfy all top-level filters.

method column_summary

source

pub fn column_summary(&self, column: &str) -> (usize, Option<i64>, Option<i64>)

Summarize the filters on integer column: the total IN-list / join-key value count, and the min/max range bounds (from >/</= constants). Used by late-materialization witnesses to report the pushed rowid filter.

method evaluate

source

pub fn evaluate(&self, batch: &RecordBatch) -> Result<BooleanArray>

Evaluate to a boolean mask (AND of all top-level filters).

method filtered_columns

source

pub fn filtered_columns(&self) -> std::collections::HashSet<String>

The set of column names referenced by the top-level filters (and their AND/OR/struct children). Mirrors Python {f.column_name for f in pf} and the Go FilteredColumns(). Lets a worker discover which columns a query constrains (e.g. to enforce a required-column rule).

method filters

source

pub fn filters(&self) -> Vec<Filter>

All top-level filters as read-only [Filter] views (the conjunction DuckDB pushed down). Mirrors Python iterating a PushdownFilters.

method format_pushed

source

pub fn format_pushed(&self) -> String

Format the pushed-down filters as a human-readable SQL-like string, matching the Python fixtures’ _format_pushed_filters. Returns "(none)" when there are no filters.

method format_repr

source

pub fn format_repr(&self) -> String

Format the filters as the Python repr(PushdownFilters): PushdownFilters([ConstantFilter(n < X), …]). Returns "(none)" when empty (matching the dynamic-filter witness fixtures).

method get_column_bounds

source

pub fn get_column_bounds(&self, column: &str) -> Option<ColumnBounds>

The numeric [min, max] bounds implied by comparison filters on column (from =/</<=/>/>=/IN), or None if the column is not constrained. Mirrors Python get_column_bounds and the Go GetColumnBounds. Bounds are integer-coerced (see [ColumnBounds]).

method get_column_constant

source

pub fn get_column_constant(&self, column: &str) -> Option<ArrayRef>

The single = constant for column as a length-1 Arrow array, or None if there is no equality filter on it. Mirrors Python get_column_constant and the Go GetColumnConstant. Descends one level into a top-level AND.

method get_column_filters

source

pub fn get_column_filters(&self, column: &str) -> Vec<Filter>

The top-level filters that reference column, as read-only [Filter] views. Mirrors Python get_column_filters and the Go GetColumnFilters.

method get_column_in_values

source

pub fn get_column_in_values(&self, column: &str) -> Option<ArrayRef>

The IN (...) value set for column as a typed Arrow array, or None if there is no IN filter on it. Mirrors Python get_column_in_values and the Go GetColumnInValues. Descends one level into a top-level AND.

method get_column_values

source

pub fn get_column_values(&self, column: &str) -> Option<ArrayRef>

Resolve the discrete =/IN value set for column as a typed Arrow array (preserving the column’s native type, so string columns like path are usable). Descends one level into a top-level AND. Mirrors Python PushdownFilters.get_column_values and the Go GetColumnValues. Returns None when the predicate is not a simple enumerable equality/IN on column. For an integer-coerced Vec<i64>, use [PushdownFilters::get_column_values_i64].

method get_column_values_i64

source

pub fn get_column_values_i64(&self, column: &str) -> Option<Vec<i64>>

Resolve the discrete value set for a column as i64s (the partition-pruning idiom; values coerced to integer). Returns None when the predicate is not enumerable (no filter, bare range, OR with a non-discrete branch). For string columns or to preserve the native type, use [PushdownFilters::get_column_values].

method has_filter_for_column

source

pub fn has_filter_for_column(&self, column: &str) -> bool

Whether any pushed-down filter references column. Mirrors Python column in pf and the Go HasFilterForColumn.

method parse

source

pub fn parse(bytes: &[u8]) -> Result<PushdownFilters>

Parse the pushdown_filters IPC blob (no join keys).

method parse_b64

source

pub fn parse_b64(encoded: &str, join_keys: &[Vec<u8>]) -> Option<PushdownFilters>

Parse a per-tick dynamic filter from the base64-encoded IPC carried in the vgi_pushdown_filters request metadata. None for empty/invalid.

method parse_with_join_keys

source

pub fn parse_with_join_keys(bytes: &[u8], join_keys: &[Vec<u8>]) -> Result<PushdownFilters>

Parse the filter blob, resolving join_keys filters against the supplied side join-keys IPC batches (one column each).

enum StatValue

source

pub enum StatValue {
  Int64(i64),
  Float64(f64),
  Utf8(String),
  Binary(Vec<u8>),
}

Description

A typed min/max statistic value.

function partition_field

source

pub fn partition_field(name: &str, ty: DataType) -> Field

Description

Build a field marked as a VGI partition column.

function partition_metadata

source

pub fn partition_metadata(
  full_schema: &SchemaRef,
  batch: &RecordBatch,
) -> Result<Option<HashMap<String, String>>>

Description

Build the per-batch metadata map for a partition-aware emit.

function partition_values_b64

source

pub fn partition_values_b64(
  full_schema: &SchemaRef,
  batch: &RecordBatch,
) -> Result<Option<String>>

Description

Compute the vgi_partition_values#b64 metadata value for a batch, or None if the (full) schema declares no partition columns or the batch is empty. The value is base64(IPC of a 2-row batch holding [min, max] for each partition column).

function serialize_column_statistics

source

pub fn serialize_column_statistics(stats: &[CatColStat]) -> Result<Vec<u8>>

Description

Serialize per-column statistics to the IPC batch the extension expects.