vgi.metadata

Module overview

Function metadata for introspection, documentation, and DuckDB registration.

This module provides declarative metadata classes that enable functions to describe themselves. Metadata is used for:

1. Documentation generation
2. Worker registration (serialized to Arrow for IPC)
3. DuckDB function catalog integration
4. Tooling and discovery

DESIGN

Users define a nested Meta class with attributes. No inheritance required:

The system automatically:

• Resolves metadata from the class hierarchy (inheritance works)
• Extracts parameter info from Arg descriptors
• Infers function name from class name if not specified
• Uses docstring as description fallback

ARROW SERIALIZATION

For worker registration, metadata can be serialized to Arrow:

from vgi.metadata import functions_to_arrow, arrow_to_functions

# Worker sends available functions to client
batch = functions_to_arrow([MyFunction, OtherFunction])

# Client receives and deserializes
function_infos = arrow_to_functions(batch)

function arrow_to_functions

source

arrow_to_functions(batch: pa.RecordBatch) -> list[ResolvedMetadata]

Deserialize Arrow RecordBatch to list of ResolvedMetadata.

Parameters

batch

RecordBatch with one row per function.

Returns

List of deserialized ResolvedMetadata objects.

function arrow_to_metadata

source

arrow_to_metadata(batch: pa.RecordBatch) -> ResolvedMetadata

Deserialize Arrow RecordBatch to ResolvedMetadata.

Parameters

batch

RecordBatch with one row containing metadata.

Returns

Deserialized ResolvedMetadata.

class CatalogFunctionType

source

Bases: Enum

Description

Type of function for DuckDB registration.

Attributes

attribute SCALAR

Scalar function: one output per input row.

attribute AGGREGATE

Aggregate function: many inputs → one output.

attribute TABLE

Table function: returns a table (streaming producer or streaming exchange).

attribute TABLE_BUFFERING

Buffered table function: Sink+Source PhysicalOperator that sees all input before producing output. Dispatched to the custom PhysicalVgiTableBufferingFunction operator instead of the streaming in_out_function registration. The class hierarchy is the dispatch key — set automatically for TableBufferingFunction subclasses.

class DistinctDependence

source

Bases: Enum

Description

Aggregate DISTINCT modifier sensitivity.

Maps to DuckDB’s AggregateDistinctDependent enum.

Attributes

attribute DISTINCT_DEPENDENT

DISTINCT changes the result (e.g., COUNT DISTINCT).

attribute NOT_DISTINCT_DEPENDENT

DISTINCT has no effect (e.g., MAX, MIN).

function extract_parameters

source

extract_parameters(
*,
validate_table_input: bool = True,
) -> list[ParameterInfo]

Extract parameter information from Arg descriptors on a class.

Walks the class and its bases to find all Arg descriptors and converts them to ParameterInfo objects. Also handles the new Param/ConstParam API for ScalarFunction subclasses.

Parameters

validate_table_input

If True, validates TableInput requirements for TableInOutFunction subclasses.

Returns

List of ParameterInfo objects, sorted by position.

Raises

TableInputValidationError

If TableInput validation fails.

class FunctionExample

source

Description

An example usage of a function.

Attributes

attribute sql

str

SQL query demonstrating the function.

attribute description

str

What this example demonstrates.

attribute expected_output

str | None

Optional expected result description.

Methods

method to_dict

source

to_dict() -> dict[str, str | None]

Convert to dictionary for serialization.

method from_dict

source

from_dict(d: dict[str, Any]) -> FunctionExample

Create from dictionary.

function functions_to_arrow

source

functions_to_arrow(function_classes: Sequence[type]) -> pa.RecordBatch

Serialize multiple function classes to Arrow RecordBatch.

Convenience function that resolves metadata for each class, then serializes. For pre-resolved metadata, use metadatas_to_arrow() directly.

Parameters

function_classes

Sequence of function classes to serialize.

Returns

RecordBatch with one row per function.

class FunctionStability

source

Bases: Enum

Description

Function output stability classification.

Maps to DuckDB’s FunctionStability enum.

Attributes

attribute CONSISTENT

Same input always produces same output (deterministic).

attribute VOLATILE

Output may change per row even with same input (e.g., random()).

attribute CONSISTENT_WITHIN_QUERY

Same within a query, but may vary across queries (e.g., now()).

class FunctionTypeError

source

Bases: TypeError

Description

Raised when a function’s type cannot be determined from its class hierarchy.

function metadata_to_arrow

source

metadata_to_arrow(metadata: ResolvedMetadata) -> pa.RecordBatch

Serialize a single ResolvedMetadata to Arrow RecordBatch.

Parameters

metadata

The metadata to serialize.

Returns

RecordBatch with one row containing the metadata.

class MetadataMixin

source

Description

Mixin that provides metadata access for function classes.

Add this to the base Function class to enable metadata resolution.

Methods

method get_metadata

source

get_metadata() -> ResolvedMetadata

Get the resolved metadata for this function class.

method describe

source

describe() -> dict[str, Any]

Get metadata as a dictionary (for JSON serialization).

function metadatas_to_arrow

source

metadatas_to_arrow(
metadatas: Sequence[ResolvedMetadata],
) -> pa.RecordBatch

Serialize multiple ResolvedMetadata objects to Arrow RecordBatch.

Parameters

metadatas

Sequence of ResolvedMetadata objects to serialize.

Returns

RecordBatch with one row per metadata object.

class NullHandling

source

Bases: Enum

Description

NULL input handling behavior.

Maps to DuckDB’s FunctionNullHandling enum.

Attributes

attribute DEFAULT

NULL in → NULL out (standard SQL behavior).

attribute SPECIAL

Function handles NULLs specially (e.g., COALESCE, IFNULL).

class OrderDependence

source

Bases: Enum

Description

Aggregate order sensitivity.

Maps to DuckDB’s AggregateOrderDependent enum.

Attributes

attribute ORDER_DEPENDENT

Result changes based on row order (e.g., FIRST, LAST, LISTAGG).

attribute NOT_ORDER_DEPENDENT

Result is the same regardless of order (e.g., SUM, COUNT).

class OrderPreservation

source

Bases: Enum

Description

Row order preservation behavior.

Maps to DuckDB’s OrderPreservationType enum:

• PRESERVES_ORDER → OrderPreservationType::INSERTION_ORDER (DuckDB default — operator maintains child operator order).
• NO_ORDER_GUARANTEE → OrderPreservationType::NO_ORDER (operator may freely reorder its input/output).
• FIXED_ORDER → OrderPreservationType::FIXED_ORDER (operator outputs rows in a fixed, mandatory order — DuckDB serializes the pipeline so a single worker produces all rows).

Attributes

attribute PRESERVES_ORDER

Output rows are in same order as input rows (DuckDB INSERTION_ORDER).

attribute NO_ORDER_GUARANTEE

Output order is undefined; may be reordered (DuckDB NO_ORDER).

attribute FIXED_ORDER

Output is in a fixed mandatory order; DuckDB serializes the pipeline (single worker) to preserve it (DuckDB FIXED_ORDER).

class ParameterInfo

source

Description

Metadata about a function parameter.

Automatically extracted from Arg descriptors.

Attributes

attribute name

str

Parameter name (attribute name from class).

attribute position

int | str

Positional index (int) or named key (str).

attribute type_name

str | None

Type name as string (e.g., “int”, “str”, TableInput).

attribute description

str

Documentation from Arg.doc.

attribute required

bool

True if no default value.

attribute default

Any

Default value, or None if required.

attribute constraints

dict[str, Any]

Validation constraints as dict.

attribute is_table_input

bool

True if this is the table input parameter.

attribute is_varargs

bool

True if this accepts multiple trailing values.

attribute is_const

bool

True if this is a constant parameter (ConstParam).

Methods

method to_dict

source

to_dict() -> dict[str, str | int | bool | None]

Convert to dictionary for serialization.

method from_dict

source

from_dict(d: dict[str, Any]) -> ParameterInfo

Create from dictionary.

class PartitionKind

source

Bases: Enum

Description

Partition shape declared by a table function.

Declared over its vgi.partition_column-annotated bind-schema fields.

Mirrors DuckDB’s TablePartitionInfo at duckdb/src/include/duckdb/function/partition_stats.hpp:20.

The C++ extension returns this from TableFunction::get_partition_info; DuckDB’s planner currently consumes only SINGLE_VALUE_PARTITIONS (to plan PhysicalPartitionedAggregate over PhysicalHashAggregate; see plan_aggregate.cpp:109). The other values are declarable so the protocol is future-proof; today they fall back to HASH_GROUP_BY.

Only set this to a non-default value when at least one field in the bind schema is annotated with {b"vgi.partition_column": b"true"} (use :func:vgi.schema_utils.partition_field to construct such fields). The reverse is also required — annotated fields without a matching partition_kind raise at worker startup.

Attributes

attribute NOT_PARTITIONED

Function does not declare partitioning over the annotated columns (default; same effect as leaving fields un-annotated).

attribute SINGLE_VALUE_PARTITIONS

Each emitted chunk has exactly one distinct value per partition column. Unlocks PhysicalPartitionedAggregate for GROUP BY over those columns.

attribute OVERLAPPING_PARTITIONS

Partitions overlap only at boundaries (bounds = [1,2] [2,3] [3,4]). Wire-level declarable; DuckDB has no consumer today.

attribute DISJOINT_PARTITIONS

Partitions are pairwise disjoint (bounds = [1,2] [3,4] [5,6]). Wire-level declarable; DuckDB has no consumer today.

function resolve_metadata

source

resolve_metadata() -> ResolvedMetadata

Resolve metadata for a function class.

Results are cached since class metadata doesn’t change at runtime.

This function:

1. Walks the class hierarchy to find and merge Meta classes
2. Extracts parameter info from Arg descriptors
3. Infers function name from class name if not specified
4. Uses docstring as description fallback

Returns

ResolvedMetadata with all resolved values.

class ResolvedMetadata

source

Description

Fully resolved metadata for a function.

This is the result of resolving a Meta class hierarchy and extracting parameter information from Arg descriptors.

Attributes

attribute name

str

Function name used for registration.

attribute class_name

str

The Python class name of the function.

attribute function_type

CatalogFunctionType

The CatalogFunctionType (scalar, table, aggregate, or table-buffering).

attribute description

str

Human-readable description of the function.

attribute examples

list[FunctionExample]

SQL usage examples for the function.

attribute categories

list[str]

Classification/category labels.

attribute tags

dict[str, str]

Free-form key/value metadata tags.

attribute parameters

list[ParameterInfo]

Resolved per-argument information from the Arg descriptors.

attribute stability

FunctionStability

Scalar evaluation stability (CONSISTENT, VOLATILE, …).

attribute null_handling

NullHandling

Whether NULL inputs are passed through or handled.

attribute required_settings

list[str]

DuckDB settings the function needs at runtime.

attribute required_secrets

list[SecretLookupEntry]

Secrets the function needs (each entry carries secret_type, optional secret_name, optional scope).

attribute projection_pushdown

bool

Whether the table function accepts projection pushdown.

attribute filter_pushdown

bool

Whether the table function accepts filter pushdown.

attribute sampling_pushdown

bool

Whether the table function accepts TABLESAMPLE pushdown.

attribute late_materialization

bool

When True, the table function participates in DuckDB’s late-materialization optimizer (TOP_N/LIMIT/SAMPLE over the scan is rewritten into a SEMI join on the rowid virtual column). Requires a unique, deterministic, snapshot-stable rowid column plus projection and filter pushdown.

attribute supported_expression_filters

list[str]

Names of expression-filter classes the function can accept pushed down.

attribute preserves_order

OrderPreservation

Whether the function preserves input row order.

attribute max_workers

int | None

Maximum parallel workers, or None for unbounded.

attribute supports_batch_index

bool

Whether the function opts into per-batch vgi_batch_index tagging for ordered parallel output.

attribute partition_kind

PartitionKind

Partition shape the function declares over its partition-column bind fields.

attribute order_dependent

OrderDependence

Whether the aggregate result depends on input order.

attribute distinct_dependent

DistinctDependence

Whether the aggregate result depends on DISTINCT.

attribute supports_window

bool

Whether the aggregate implements the window() callback.

attribute streaming_partitioned

bool

Whether the aggregate opts into the streaming-partitioned protocol.

attribute has_finalize

bool

True if the function has a meaningful finalize phase (override of finalize()/finish()); the C++ extension uses this to decide whether to register in_out_function_final.

attribute source_order_dependent

bool

Only meaningful for TABLE_BUFFERING — when True the source phase is single-threaded and finalize_state_ids drain in combine-returned order.

attribute sink_order_dependent

bool

Only meaningful for TABLE_BUFFERING — when True the sink phase runs single-threaded, so every process() call arrives in source order. Mutually exclusive with requires_input_batch_index.

attribute requires_input_batch_index

bool

Only meaningful for TABLE_BUFFERING — when True the C++ Sink declares RequiredPartitionInfo()= BatchIndex() so each process() call carries a globally-unique monotonic batch_index for reconstructing source order under parallel ingest. Mutually exclusive with sink_order_dependent.

attribute input_from_args

bool

True for a blended RowTransformFunction — its positional args ARE its per-row input columns, so one registration serves the literal / column / LATERAL call shapes. Detected from RowTransformFunction subclassing.

Methods

method to_dict

source

to_dict() -> dict[str, Any]

Convert to dictionary for JSON serialization.

method from_dict

source

from_dict(d: dict[str, Any]) -> ResolvedMetadata

Create from dictionary.

class TableInputValidationError

source

Bases: ValueError

Description

Raised when TableInput parameter validation fails.

class VarargsValidationError

source

Bases: ValueError

Description

Raised when varargs parameter validation fails.