vgi.table_function

Module overview

Base classes for table functions with cardinality hints and callback-based processing.

TableFunctionGenerator produces output batches via a per-tick callback. Each call to process() either emits a batch via out.emit() or signals completion via out.finish().

function bind_fixed_schema

source

bind_fixed_schema() -> type[T]

Class decorator to return FIXED_SCHEMA from on_bind for a TableFunctionGenerator subclass.

Sets cls._inline_bind_safe = True only when the decorator actually installs its own on_bind. The catalog framework reads this marker to decide whether Table(inline_bind=True) is allowed — the contract is “the decorator’s bind is in control, output is exactly cls.FIXED_SCHEMA, no kwargs inspected.” If the class already defined its own on_bind, the decorator silently leaves it alone and we must not set the marker; otherwise the framework would inline a bind it doesn’t actually control.

Subclasses inherit the marker via Python attribute lookup. A subclass that overrides on_bind adds it to its own __dict__; the catalog framework’s eligibility check is getattr(cls, "_inline_bind_safe", False) and "on_bind" not in cls.__dict__, which correctly excludes such subclasses.

Returns

The same class, with an on_bind returning cls.FIXED_SCHEMA injected if it did not already define one.

class BindParams

source

Description

Parameters passed to on_bind().

Attributes

attribute args

TArgs

The parsed function arguments.

attribute bind_call

BindRequest

The underlying bind request from the client.

attribute settings

dict[str, pa.Scalar[Any]]

DuckDB settings extracted from the bind_call, keyed by name.

attribute secrets

SecretsAccessor

Accessor for pre-resolved and dynamically-requested secrets.

attribute transaction_storage

TransactionBoundStorage | None

Transaction-scoped storage view that lets cardinality() / statistics() cache expensive lookups (e.g. Kafka watermarks) in the same store on_init reads/writes for snapshot isolation. None when bind_call.transaction_opaque_data is unset.

attribute storage

BoundStorage | None

Execution-scoped storage view, populated only on call paths that carry a global_execution_id (currently dynamic_to_string). None for bind/cardinality/statistics (they predate execution).

attribute auth_context

AuthContext

Authentication context for the caller.

attribute attach_opaque_data

bytes | None

The catalog’s attach bytes, unwrapped by the framework (shard-UUID prefix stripped). None without an ATTACH.

attribute at_unit

str | None

The AT (TIMESTAMP|VERSION) unit for this scan, or None without an AT clause.

NOTE: for inline-bound (function-backed) tables on_bind runs once at attach with no AT, so this is None here — read AT at init/process via ProcessParams.at_value. See BindRequest.at_unit.

attribute at_value

str | None

The AT (TIMESTAMP|VERSION) value for this scan, or None. See at_unit.

function init_single_worker

source

init_single_worker() -> type[T]

Class decorator to set max_workers=1 for a TableFunctionGenerator subclass.

Returns

The same class, with an on_init returning max_workers=1 injected if it did not already define one.

class InitParams

source

Description

Parameters passed to on_init().

Attributes

attribute args

TArgs

The parsed function arguments.

attribute init_call

InitRequest

The underlying init request from the client.

attribute execution_id

bytes

Unique identifier for this execution.

attribute output_schema

pa.Schema

The projected output schema (based on projection_ids) that the function should produce.

attribute settings

dict[str, pa.Scalar[Any]]

DuckDB settings extracted from the bind_call, keyed by name.

attribute secrets

ResolvedSecrets

Resolved secrets as dicts keyed by secret_type.

attribute storage

BoundStorage

Execution-scoped storage view for this init.

attribute auth_context

AuthContext

Authentication context for the caller.

attribute attach_opaque_data

bytes | None

The catalog’s attach bytes, unwrapped by the framework (uuid prefix stripped). None without an ATTACH.

attribute at_unit

str | None

AT (TIMESTAMP|VERSION) unit for this scan, or None.

Carried on the per-scan bind embedded in the init request. See BindRequest.at_unit.

attribute at_value

str | None

AT (TIMESTAMP|VERSION) value for this scan, or None. See at_unit.

class OrderByDirection

source

Bases: Enum

Description

ORDER BY direction pushed down from DuckDB’s RowGroupPruner optimizer.

Attributes

attribute ASC

Ascending order.

attribute DESC

Descending order.

class OrderByNullOrder

source

Bases: Enum

Description

NULL ordering pushed down from DuckDB’s RowGroupPruner optimizer.

Attributes

attribute NULLS_FIRST

Nulls sort before non-null values.

attribute NULLS_LAST

Nulls sort after non-null values.

class ProcessParams

source

Description

Parameters passed to process() and finalize().

Attributes

attribute args

TArgs

The parsed function arguments.

attribute init_call

InitRequest | None

The init request, or None for aggregate functions.

attribute init_response

BaseInitResponse | None

The init response, or None for aggregate functions.

attribute output_schema

pa.Schema

The projected output schema (based on projection_ids) that the function should produce.

attribute settings

dict[str, pa.Scalar[Any]]

DuckDB settings extracted from the bind_call, keyed by name.

attribute secrets

ResolvedSecrets

Resolved secrets as dicts keyed by secret_type.

attribute storage

BoundStorage

Execution-scoped storage view.

attribute auth_context

AuthContext

Authentication context for the caller.

attribute current_pushdown_filters

Any

Current pushdown filters (PushdownFilters | None), updated dynamically from tick metadata (e.g. for Top-N queries) before each process() call. None if no filters have been received.

attribute batch_index

int | None

Globally-unique monotonic batch index for this process() call. Populated only for TableBufferingFunction subclasses with Meta.requires_input_batch_index=True, letting workers reconstruct source order under parallel ingest. None for every other call path.

attribute attach_opaque_data

bytes | None

The catalog’s attach bytes, unwrapped by the framework (uuid prefix stripped). None without an ATTACH.

attribute if_none_match

str | None

Conditional-revalidation validator (client’s stored ETag). Set when the client holds a stale-but-revalidatable cached result and asks the worker to confirm freshness cheaply; a worker that advertised revalidatable compares it and, if unchanged, emits a 0-row CacheControl(not_modified=True) batch. None otherwise.

attribute if_modified_since

str | None

Conditional-revalidation validator (client’s stored Last-Modified). Companion to if_none_match. None otherwise.

attribute at_unit

str | None

AT (TIMESTAMP|VERSION) unit for this scan, or None.

Carried on the per-scan bind embedded in the init request; None for aggregate functions (no init_call). See BindRequest.at_unit.

attribute at_value

str | None

AT (TIMESTAMP|VERSION) value for this scan, or None. See at_unit.

attribute substream_id

bytes | None

Stable client-minted id for this streaming table-in-out substream.

Present (identical across init / every process() / finalize) when the client fanned this function out across per-substream workers; use it to key per-substream accumulated state in shared storage so a finalize() that lands on a different HTTP backend than the process() calls still finds it. None for the serial path, aggregate functions (no init_call), or an old client that did not supply one. See InitRequest.substream_id.

function project_schema

source

project_schema(
projection_ids: list[int] | None,
schema: pa.Schema,
) -> pa.Schema

Create the projected schema if projection_ids are supplied.

Parameters

projection_ids

Column indices to project, or None for all columns.

schema

The full output schema to project from.

Returns

The projected schema, or the original schema when projection_ids is None.

class ResolvedSecrets

source

Bases: dict[str, dict[str, Any]]

Description

Resolved secrets keyed by secret name, with type- and scope-aware lookup.

A plain dict (so secrets[name] and secrets.get(name) still work) plus selectors that read each secret’s connector-serialized type and scope fields. Mirrors vgi::Secrets in the Rust SDK.

Methods

method secret_type

source

secret_type(name: str) -> str | None

The DuckDB secret type of the named secret (its type field).

method of_type

source

of_type(secret_type: str) -> list[dict[str, Any]]

Every resolved secret whose type field matches secret_type.

method for_scope

source

for_scope(path: str) -> dict[str, Any] | None

The secret whose scope is the longest prefix of path.

The connector serializes each secret’s scope as a newline-joined list of prefixes; a secret with no (or empty) scope matches as a last-resort fallback. Returns None only when there are no candidate secrets.

method for_scope_of_type

source

for_scope_of_type(
path: str,
secret_type: str,
) -> dict[str, Any] | None

Like :meth:for_scope but only over secrets of secret_type.

method field_for

source

field_for(path: str, field: str) -> Any | None

A field of the best scope-matching secret for path.

class SecretsAccessor

source

Description

Unified access to secrets — pre-resolved and dynamically requested.

Pre-resolved secrets (from Secret() annotations with static scope/name, or unscoped lookups) are available immediately. Dynamic lookups (computed scope from function arguments) register pending requests — the framework automatically triggers a two-phase bind retry to resolve them.

Attributes

attribute all_resolved

bool

True if all requested secrets have been resolved (no pending lookups).

Use this to distinguish ‘not yet resolved’ from ‘genuinely not found’ when not using required=True on get().

attribute needs_resolution

bool

True if there are pending lookups that need resolution.

attribute pending_lookups

list[SecretLookupEntry]

Return the list of pending secret lookups.

Methods

method get

source

get(
secret_type: str,
*,
name: str | None = None,
scope: str | None = None,
required: bool = False,
) -> dict[str, pa.Scalar[Any]] | None

Get a secret by type, with optional name and/or scope.

Parameters

secret_type

The secret type (e.g., “vgi_example”, “s3”).

name

Optional secret name for name-based lookup.

scope

Optional scope for scoped lookup (longest-prefix match).

required

If True, raises ValueError when the secret is genuinely not found (after resolution).

Returns

dict of string keys to Arrow scalars, or None if not found.

method to_dict

source

to_dict() -> ResolvedSecrets

Return all resolved secrets keyed by secret name.

Resolved secrets are keyed by their unique DuckDB secret name, so several secrets of the same type (e.g. one per S3 bucket) coexist. Each carries a type field (the DuckDB secret type) and a scope field (newline-joined scope prefixes). Scoped secret_N columns (keyed by secret_type from Arrow field metadata) are merged in. Null/unresolved entries are omitted.

Returns

class:ResolvedSecrets (a dict keyed by secret name) with type- and scope-aware selection helpers.

class TableCardinality

source

Bases: ArrowSerializableDataclass

Description

Cardinality hints for query optimization.

Provides optional row count estimates that can help query planners make better decisions about join ordering, memory allocation, and parallelization.

Attributes

attribute estimate

int | None

Estimated number of output rows, or None if unknown.

attribute max

int | None

Maximum possible output rows, or None if unbounded.

class TableFunctionBase

source

Bases: vgi.function.Function

Description

Base class for table functions with cardinality and schema validation.

Extends Function with:

• Cardinality hints for query optimization
• Projection pushdown support

This class is not meant to be used directly. Subclass either:

• TableFunctionGenerator: For simple generators that produce output
• TableInOutGenerator: For functions that transform input batches

See also

TableFunctionGenerator: Simple generator base class TableInOutGenerator: Full streaming with input batches

Attributes

attribute FunctionArguments

type

The dataclass type describing this function’s arguments, auto-extracted from the generic parameter if not set.

Methods

method on_bind

source

on_bind(params: BindParams[TArgs]) -> BindResponse

Produce the output schema and perform other bind-time logic.

Subclasses must override. Common patterns:

• Pass through: return BindResponse(output_schema=params.bind_call.input_schema)
• Custom shape: build a pa.Schema from params.args and return it.
• Dynamic secrets: declare *, my_secret: Annotated[..., Secret()] = None or call params.secrets.get(...); the framework will issue a secret-scope retry automatically.

Parameters

params

Bind parameters including arguments and schema.

Returns

BindResponse with output_schema and optional opaque_data.

method bind

source

bind(
input: BindRequest,
*,
ctx: CallContext | None = None,
attach_plaintext: bytes | None = None,
) -> BindResponse

Bind protocol entry point. Do not override; use on_bind().

Validates type bounds when an input schema is present (table-input functions), constructs BindParameters, calls on_bind(), and wraps the result for transmission to global_init. If on_bind() triggered dynamic secret lookups via SecretsAccessor, returns a secret-scope request to trigger two-phase bind.

Note: we do NOT auto-request secrets before on_bind(). Table functions handle secrets via on_bind kwargs (Secret() annotations) and SecretsAccessor.get() calls, which may use dynamic scopes computed from function arguments.

Parameters

input

The bind request from the client.

ctx

Call context carrying the caller’s auth, if any.

attach_plaintext

Full framework attach plaintext, or None.

Returns

The BindResponse from on_bind(), or a secret-scope request when dynamic secret lookups need a two-phase bind.

method on_init

source

on_init(params: InitParams[TArgs]) -> GlobalInitResponse

One-time setup after bind, before processing batches.

Override to perform per-execution setup (open external resources, allocate caches, etc.). Default is a no-op.

Parameters

params

Init parameters including arguments, schema, and storage.

Returns

A GlobalInitResponse (default empty).

method global_init

source

global_init(
input: InitRequest,
*,
ctx: CallContext | None = None,
attach_plaintext: bytes | None = None,
) -> GlobalInitResponse

Global init protocol entry point. Do not override; use on_init().

Parameters

input

The init request from the client.

ctx

Call context carrying the caller’s auth, if any.

attach_plaintext

Full framework attach plaintext, or None.

Returns

The GlobalInitResponse with worker count and execution id.

method cardinality

source

cardinality(params: BindParams[TArgs]) -> TableCardinality

Return the cardinality for the output.

Override to provide row count estimates that help query planners make better decisions about join ordering and memory allocation.

Parameters

params

Bind parameters — function args, settings, and secrets.

Returns

TableCardinality with estimate and/or max, or None if unknown.

method dynamic_to_string

source

dynamic_to_string(
params: BindParams[TArgs],
execution_id: bytes,
) -> Mapping[str, str]

Return diagnostics rendered as Extra Info under EXPLAIN ANALYZE.

Fired once per parallel scan thread at end-of-stream. The function class is responsible for persisting whatever diagnostics it cares about during process() (shared storage, external service, in-memory class state for single-worker setups) and retrieving them by execution_id here.

DuckDB merges the per-thread maps with last-write-wins semantics, so the last thread to finish — by which time every thread has persisted — supplies the visible final view.

Best-effort: must not raise. The dispatcher catches exceptions and returns an empty map so EXPLAIN ANALYZE never breaks the query.

Parameters

params

Same BindParams cardinality and statistics receive — function args, settings, secrets.

execution_id

VgiTableFunctionGlobalState::global_execution_id, stable for the duration of the query.

Returns

Ordered key/value pairs. Insertion order is preserved on the wire and re-emitted into the C++ profiler’s InsertionOrderPreservingMap. The C++ wrapper appends intrinsic keys (Worker, Function, Rows Read, Threads) after this map; user keys override on conflict.

method statistics

source

statistics(params: BindParams[TArgs]) -> list[ColumnStatistics] | None

Return per-output-column statistics for this invocation.

Override to provide min/max/distinct/null stats so DuckDB’s optimizer can do filter elimination (e.g. prune a scan entirely when the filter is out of range), improve join ordering, and fold always-true/always-false predicates at plan time.

params is the same BindParams[TArgs] used by cardinality and initial_state, so stats can be derived directly from user-supplied arguments.

Parameters

params

Bind parameters — function args, settings, and secrets.

Returns

A list of ColumnStatistics (one entry per column for which stats are known — columns not listed get unknown stats), or None when no stats are available (same effect as today: optimizer receives no column stats).

method pushdown_filters

source

pushdown_filters(
pushdown_filters: pa.RecordBatch,
join_keys: list[pa.RecordBatch] | None = None,
) -> PushdownFilters | None

Get deserialized pushdown filters, or None if not present.

Use this property to access the filter AST for:

• Custom filter handling (push to SQL, APIs, etc.)
• Extracting column bounds for partition pruning
• Checking column constants for optimized lookups

For automatic filtering, set auto_apply_filters=True in Meta.

Parameters

pushdown_filters

Arrow RecordBatch containing serialized filters.

join_keys

Optional list of single-column Arrow RecordBatches, one per IN filter column. Available via get_join_keys_batch() / get_join_keys_batches() on the returned PushdownFilters.

Returns

PushdownFilters container with parsed filter AST, or None.

Inherited members (4)

• get_metadata method · from MetadataMixin — Get the resolved metadata for this function class.
• describe method · from MetadataMixin — Get metadata as a dictionary (for JSON serialization).
• logger attribute · from Function
• storage attribute · from Function

class TableFunctionGenerator

source

Bases: TableFunctionBase[TArgs]

Description

Callback-based table function that produces output batches.

Each call to process() should either:

• Emit a batch via out.emit(batch)
• Signal completion via out.finish()

Use TState to persist state between process() calls.

For functions that transform input batches, use TableInOutGenerator.

Methods

method initial_state

source

initial_state(params: ProcessParams[TArgs]) -> TState | None

Create initial processing state. Override when TState is used.

Called once during init to create the state object that will be passed to process() on each tick.

Parameters

params

Process parameters including arguments and schemas.

Returns

Initial state, or None if no state is needed.

method process

source

process(
params: ProcessParams[TArgs],
state: TState,
out: OutputCollector,
) -> None

Produce output for one tick.

Called repeatedly by the framework. Each call should either:

• Call out.emit(batch) to produce one output batch
• Call out.finish() to signal that generation is complete

Use out.client_log(level, message) for in-band logging.

Parameters

params

Process parameters including arguments and schemas.

state

Mutable state persisted between calls. None if TState not used.

out

OutputCollector for emitting batches, logging, and signaling finish.

method on_cancel

source

on_cancel(params: ProcessParams[TArgs], state: TState) -> None

Inherited members (13)

• get_metadata method · from MetadataMixin — Get the resolved metadata for this function class.
• describe method · from MetadataMixin — Get metadata as a dictionary (for JSON serialization).
• logger attribute · from Function
• storage attribute · from Function
• FunctionArguments attribute · from TableFunctionBase
• on_bind method · from TableFunctionBase — Produce the output schema and perform other bind-time logic.
• bind method · from TableFunctionBase — Bind protocol entry point. Do not override; use on_bind().
• on_init method · from TableFunctionBase — One-time setup after bind, before processing batches.
• global_init method · from TableFunctionBase — Global init protocol entry point. Do not override; use on_init().
• cardinality method · from TableFunctionBase — Return the cardinality for the output.
• dynamic_to_string method · from TableFunctionBase — Return diagnostics rendered as Extra Info under EXPLAIN ANALYZE.
• statistics method · from TableFunctionBase — Return per-output-column statistics for this invocation.
• pushdown_filters method · from TableFunctionBase — Get deserialized pushdown filters, or None if not present.

class TableInOutFunctionInitPhase

source

Bases: Enum

Description

Init-call phase for table functions.

INPUT / FINALIZE drive the streaming TableInOutGenerator path. TABLE_BUFFERING is the Sink+Source init phase for TableBufferingFunction — after init, traffic moves to table_buffering_process / _combine (unary) and TABLE_BUFFERING_FINALIZE opens a producer-mode finalize stream per finalize_state_id.

Attributes

attribute INPUT

Streaming input phase for the table-in-out generator path.

attribute FINALIZE

End-of-input finalize phase for the streaming path.

attribute TABLE_BUFFERING

Sink+Source init phase for TableBufferingFunction.

attribute TABLE_BUFFERING_FINALIZE

Producer-mode finalize stream phase, opened per finalize_state_id.