vgi.catalog

Module overview

VGI Catalog Interface for exposing catalogs, schemas, tables, and views.

This module provides the abstract base class and data types for implementing catalog interfaces in VGI workers, enabling DuckDB ATTACH support.

class AttachCatalogInfo

source

Bases: ArrowSerializableDataclass

Description

A companion catalog the client should ATTACH when this VGI catalog attaches.

Advertised via :attr:CatalogAttachResult.attach_catalogs. The VGI DuckDB extension attaches each entry at VGI-attach time (into the client’s DatabaseManager) so that multi-branch catalog-table branches — and direct user queries — can resolve tables in a companion lakehouse (DuckLake / iceberg / postgres / …) without the client hand-attaching anything.

Trust: companion attach is remote-influenced, so the extension applies a scheme allowlist and a never-clobber conflict policy (it will never replace a catalog it did not itself create). See docs/companion_catalogs.md.

Attributes

attribute alias

str

Catalog name to attach as. Also the source_catalog a catalog-table :class:ScanBranch references. Namespace this by your catalog identity (e.g. "acme_lake") so two workers don’t both claim "lake" on the same client — collisions are rejected, never silently merged.

attribute target

str

The ATTACH target — a path or DSN, e.g. "ducklake:sqlite:/data/meta.sqlite" or "postgres:dbname=… host=…".

attribute db_type

str

DuckDB storage/db type (e.g. "ducklake", "postgres"). Empty ⇒ the extension infers it from the target scheme prefix.

attribute options

dict[str, str]

Extra ATTACH options forwarded verbatim (e.g. DuckLake DATA_PATH). Keys are matched case-insensitively by DuckDB.

attribute hidden

bool

When true, attach hidden (excluded from duckdb_databases()); still resolvable by qualified name and by branches. Use for branch-only companions the user shouldn’t see.

attribute required

bool

When true, a failure to attach (unreachable / conflict) fails the whole VGI ATTACH loudly. When false, the failure is logged and skipped; branches referencing it then error at bind.

attribute secret_ref

str

Optional name of a credential to pre-resolve (via the catalog’s Orchard secret provider) and inject into the companion’s ATTACH options — for metadata connections (e.g. a postgres DSN) where the catch-all path-keyed secret lookup isn’t enough. Empty ⇒ rely on the automatic catch-all provider for data-file creds.

class Catalog

source

Description

Declarative catalog definition containing schemas.

The single entry point for defining all catalog metadata on a Worker.

Attributes

attribute name

str

The catalog name (used in SQL as the database name).

attribute default_schema

str

Schema to use for unqualified table/view/function names.

attribute schemas

Sequence[Schema]

Sequence of Schema objects defining the catalog contents.

attribute comment

str | None

Optional comment describing the catalog.

attribute tags

dict[str, str]

Optional key-value tags associated with the catalog.

attribute source_url

str | None

Where this worker’s code lives — repo, build, or docs homepage. None (the default) when the worker doesn’t advertise a source location. Surfaced via the catalog_catalogs() discovery record (CatalogInfo.source_url).

attribute global_functions

Sequence[type[Function]]

Functions to additionally publish into the client’s global function namespace (DuckDB’s system.main), so they can be called unqualified without naming this catalog — the way ducklake_table_info is reachable after LOAD ducklake. Use this for utility functions that aren’t about a particular attached catalog (diagnostics, converters, helpers).

Every entry must also appear in exactly one Schema.functions: bind dispatch is keyed on (schema_name, name), so a function that exists only here would be registered but never dispatchable. The schema-qualified name keeps working and is the unambiguous fallback when a global name is claimed by another worker.

Registration is first-attach-wins and best-effort — a name already owned by a different worker is skipped, and the client may disable the whole mechanism per-ATTACH. Never rely on a global name resolving; treat it as an ergonomic alias.

attribute global_function_prefix

str | None

Prefix applied to each global_functions entry to form its globally visible name (<prefix>_<name>). None publishes bare names, which is more collision-prone — prefer a prefix that identifies this worker.

class CatalogAttachResult

source

Bases: ArrowSerializableDataclass

Description

Result from attaching to a catalog.

Attributes

attribute attach_opaque_data

AttachOpaqueData

The unique id for the attached catalog.

attribute supports_transactions

bool

Indicate if the worker supports transactions or not. If false, all transaction related methods will not be called and all transaction_opaque_data parameters will be None.

attribute supports_time_travel

bool

Indicate if tables support time travel.

attribute catalog_version_frozen

bool

Indicate that the catalog version id is frozen and the schema and object information will not change.

attribute catalog_version

int

The initial catalog version, it increments when schemas, tables or other objects change.

attribute attach_opaque_data_required

bool

Indicate if the attach_opaque_data must be persisted across commands. True: Catalog is stateful; attach_opaque_data represents a session. False: Catalog is stateless; CLI can auto-attach on each command.

attribute default_schema

str

The name of the default schema for this catalog.

attribute settings

list[bytes]

Extension options (settings) exposed by this catalog/worker. Each ExtensionOption is serialized as bytes for Arrow compatibility.

attribute secret_types

list[bytes]

Secret types registered with DuckDB’s SecretManager. Each SecretTypeSpec is serialized as bytes for Arrow compatibility.

attribute attach_catalogs

list[bytes]

Companion catalogs the client should ATTACH alongside this VGI catalog (lakehouse federation). Each :class:AttachCatalogInfo is serialized as bytes for Arrow compatibility. Attached at VGI-attach time; detached (refcounted) on DETACH. See AttachCatalogInfo and docs/companion_catalogs.md.

attribute comment

str | None

Optional comment describing this catalog/database.

attribute tags

dict[str, str]

Optional key-value tags associated with this catalog/database.

attribute supports_column_statistics

bool

Whether any tables in this catalog can provide column statistics. Global gate — if False, GetStatistics() returns nullptr for all tables.

attribute global_functions

list[bytes]

Functions this catalog asks the client to publish into the global function namespace (DuckDB’s system.main), in addition to their normal schema-qualified registration. Each :class:FunctionInfo is serialized as bytes for Arrow compatibility. name/schema_name stay the real dispatch coordinates — the client derives the globally visible name by applying global_function_prefix. Registration is first-attach-wins: a name already owned by a different worker is skipped (logged), and the ATTACH still succeeds. Empty list = this catalog publishes nothing globally. See docs/global-functions.md.

attribute global_function_prefix

str

Prefix applied to every global_functions entry to form its globally visible name (<prefix>_<name>). Empty string = publish bare names, which is more collision-prone; prefer a prefix that identifies the worker.

attribute resolved_data_version

str | None

Concrete data version the worker resolved for this attach. None = worker has no opinion or the request omitted data_version_spec.

attribute resolved_implementation_version

str | None

Concrete implementation version the worker resolved for this attach. None = worker has no opinion or the request omitted implementation_version.

class CatalogDataVersionRelease

source

Bases: ArrowSerializableDataclass

Description

One published data version of a catalog.

data_version_spec advertises a compatibility range; this record advertises what’s actually been published. Together they let a client (the describe page, Cupola, programmatic consumers) render a discoverable release timeline without scraping the worker’s repo.

Contracts on the CatalogInfo.releases list this belongs to:

• Ordering — entries MUST appear newest-first. Unspecified order would force consumers to sort by version string, which requires a comparator the protocol does not define (semver vs. calver vs. date-stamped vs. RC tags are all valid).
• Uniqueness — each version MUST appear at most once. Mirrors the same invariant on attach_option_specs’s name. Consumers defend against duplicates (log-and-skip later entries) since Arrow cannot enforce key uniqueness at the wire level.

Long-form release notes do not live here — link to a CHANGELOG anchor, GitHub release page, PR, or migration guide via notes_url.

Attributes

attribute version

str

Concrete version, not a spec. e.g. “1.0.0”, “2.4.1”. Semver carries the breaking-change signal directly — major bumps are breaking, minor/patch are not.

attribute released_at

Annotated[datetime | None, ArrowType(pa.timestamp(us, tz=UTC))]

Release date (UTC). None when the worker doesn’t track dates.

attribute summary

str

One-line human summary. Empty string when unknown.

attribute notes_url

str | None

Optional per-release link to detailed notes. Distinct from CatalogInfo.source_url (which points at the repo as a whole): this points at what changed in this release.

class CatalogExample

source

Bases: ArrowSerializableDataclass

Description

An example usage of a function for catalog serialization.

Attributes

attribute sql

str

SQL query demonstrating the function.

attribute description

str

What this example demonstrates.

attribute expected_output

str | None

Optional expected result description.

class CatalogInfo

source

Bases: ArrowSerializableDataclass

Description

Discovery record for a catalog exposed by a worker.

Returned by catalog_catalogs() so clients can inspect per-catalog version metadata before attaching.

Attributes

attribute name

str

Catalog name — pass to catalog_attach() to open it.

attribute implementation_version

str | None

Worker software version (singular per worker). None = worker declares no implementation version.

attribute data_version_spec

str | None

Semver range the catalog serves (e.g. “>=1.0.0,<2.0.0”). None = worker declares no data-version opinion.

attribute attach_option_specs

list[bytes]

Attach-time options the catalog accepts (distinct from session settings). Each AttachOptionSpec is serialized as bytes for Arrow compatibility. Enables pre-attach discovery via the catalogs() RPC.

attribute releases

list[CatalogDataVersionRelease]

Concrete published data versions, newest-first. Empty when the worker doesn’t track release history. See CatalogDataVersionRelease for the per-entry ordering and uniqueness contracts.

attribute source_url

str | None

Where this worker’s code lives — repo, build, docs. None when the worker doesn’t advertise a source location.

class CatalogInterface

source

Bases: ABC

Description

Provides an interface to manage catalogs, schemas, tables, and views for VGI.

This interface defines methods for creating, dropping, and managing catalogs, schemas, tables, and views. It also supports transactions and provides methods for discovering catalog contents.

Implementors of this interface should provide concrete implementations for all abstract methods and properties.

Api limitations

• Functions are not able to be created or dropped.
• Tags are not able to be updated on catalog objects.
• Comments and tags are not updatable on schemas (SchemaInfo).
• Constraints cannot be added/dropped (except NOT NULL).

A VGI worker will offer a single implementation of this interface to clients to manage their catalogs.

Attributes

attribute interface_feature_flags

set[str]

Get the feature flags supported by this CatalogInterface.

Feature flags indicate optional capabilities of the implementation. The default implementation returns an empty set.

Methods

method loggable_attach_options

source

loggable_attach_options(
options: Mapping[str, Any],
) -> Mapping[str, Any]

Return a redacted view of attach/create options safe for logs and Sentry breadcrumbs.

Called by the worker when emitting catalog lifecycle events (catalog.attach, catalog.create). Override to opt in to logging the option fields you know are safe — host names, regions, bucket names, etc. Never return credentials such as passwords, tokens, or connection strings containing secrets.

Default returns an empty mapping, so by default nothing from the options dict is logged. This fail-closed behaviour avoids leaking credentials when an implementer has not explicitly chosen which fields are safe to emit.

Parameters

options

The raw options dict the client passed to ATTACH / CREATE (the same dict handed to :meth:catalog_attach or :meth:catalog_create).

Returns

A mapping of safe-to-log key/value pairs. Returning an empty mapping (the default) suppresses the options field from lifecycle events entirely.

method catalogs

source

catalogs() -> list[CatalogInfo]

Get a list of catalog discovery records provided by the VGI worker.

Each record carries the catalog name and — if the worker has opinions — its implementation_version and data_version_spec, so clients can prevalidate ATTACH requests.

This is a discovery only method.

method catalog_create

source

catalog_create(
*,
name: str,
on_conflict: OnConflict,
options: dict[str, Any],
) -> None

Create a new catalog with the given name.

If on_conflict is IGNORE and the catalog already exists, do nothing. If on_conflict is REPLACE and the catalog already exists, replace it. If on_conflict is ERROR and the catalog already exists, raise an error.

method catalog_drop

source

catalog_drop(*, name: str) -> None

Drop the catalog with the given name.

method catalog_transaction_begin

source

catalog_transaction_begin(
*,
attach_opaque_data: AttachOpaqueData,
) -> TransactionOpaqueData | None

Begin a new transaction for the given attach_opaque_data.

If the implementation does not support transactions, it can return None.

method catalog_transaction_commit

source

catalog_transaction_commit(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData,
) -> None

Commit the transaction for the given attachment.

If the transaction cannot be committed, an exception should be raised.

method catalog_transaction_rollback

source

catalog_transaction_rollback(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData,
) -> None

Rollback the transaction for the given attachment.

If the transaction cannot be rolled back, an exception should be raised.

method catalog_attach

source

catalog_attach(
*,
name: str,
options: dict[str, Any],
data_version_spec: str | None,
implementation_version: str | None,
ctx: CallContext | None = None,
) -> CatalogAttachResult

Attach to a catalog with the given name and options.

data_version_spec and implementation_version carry the semver constraints the client requested at ATTACH time. Pass-through strings — subclasses interpret and validate them. None means the client did not constrain that dimension. Implementations that cannot satisfy a requested version MUST raise an exception with a human-readable message; the error surfaces on the client as the ATTACH failure.

ctx is injected by the RPC dispatcher when available. Over HTTP it enables setting a per-session routing cookie via ctx.set_cookie(); over subprocess it may be None or have empty cookie support.

Returns a CatalogAttachResult containing the attach ID, other catalog metadata, and the resolved concrete versions chosen by the worker.

method catalog_detach

source

catalog_detach(*, attach_opaque_data: AttachOpaqueData) -> None

Detach from the catalog with the given attach_opaque_data.

Any open transactions should be rolled back. The default implementation does nothing.

method catalog_version

source

catalog_version(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
ctx: CallContext | None = None,
) -> int

Get the current catalog version for the given attach_opaque_data and transaction_opaque_data.

Returns an integer representing the current catalog version.

Changes to schemas, tables, and objects increment this version. It is used to expire cached catalog/schema/object information inside a VGI client or process.

ctx is injected by the RPC dispatcher when available. Subclasses that use HTTP-session cookies can consult ctx.cookies to verify routing stickiness.

The default implementation returns 0.

method schemas

source

schemas(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
) -> list[SchemaInfo]

Get a list of schemas for the given attach_opaque_data and transaction_opaque_data.

The default returns a schema called “main” with no comment or tags.

method schema_create

source

schema_create(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
name: str,
on_conflict: OnConflict = OnConflict.ERROR,
comment: str | None,
tags: dict[str, str],
) -> None

Create a new schema with the given name, comment, and tags.

method schema_drop

source

schema_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
name: str,
ignore_not_found: bool,
cascade: bool,
) -> None

Drop the schema with the given name.

method schema_contents

source

schema_contents(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
name: str,
type: SchemaObjectType,
) -> Sequence[TableInfo | ViewInfo | FunctionInfo | MacroInfo | IndexInfo]

Get the contents of the schema with the given name.

Schemas can contain tables, views, functions, macros, and indexes.

Parameters

attach_opaque_data

The attachment identifier.

transaction_opaque_data

The transaction identifier, if any.

name

The name of the schema.

type

The type of objects to return. Must be a SchemaObjectType enum:
• SchemaObjectType.TABLE: Return only tables
• SchemaObjectType.VIEW: Return only views
• SchemaObjectType.SCALAR_FUNCTION: Scalar functions
• SchemaObjectType.TABLE_FUNCTION: Table functions
• SchemaObjectType.SCALAR_MACRO: Scalar macros
• SchemaObjectType.TABLE_MACRO: Table macros
• SchemaObjectType.INDEX: Indexes

Returns

A list of TableInfo, ViewInfo, FunctionInfo, or MacroInfo objects depending on the type parameter.

method schema_get

source

schema_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
name: str,
) -> SchemaInfo | None

Get information about the schema with the given name.

Returns a SchemaInfo object if the schema exists, or None if it does not.

method table_get

source

table_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
at_unit: str | None = None,
at_value: str | None = None,
) -> TableInfo | None

Get information about the table with the given name in the specified schema.

When at_unit / at_value are provided the implementation should return the table schema for the requested point in time (time travel).

Returns a TableInfo object if the table exists, or None if it does not.

method table_create

source

table_create(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
columns: SerializedSchema,
on_conflict: OnConflict,
not_null_constraints: list[int],
unique_constraints: list[list[int]],
check_constraints: list[str],
primary_key_constraints: list[list[int]] | None = None,
foreign_key_constraints: list[bytes] | None = None,
) -> None

Create a new table with the given name and schema.

Comments and tags are not supported on table creation.

method table_drop

source

table_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
ignore_not_found: bool,
cascade: bool = False,
) -> None

Drop the table with the given name.

method table_comment_set

source

table_comment_set(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
comment: str | None,
ignore_not_found: bool,
) -> None

Set the comment for the table with the given name.

method table_column_comment_set

source

table_column_comment_set(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
column_name: str,
comment: str | None,
ignore_not_found: bool,
) -> None

Set the comment for a column in the table.

method table_rename

source

table_rename(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
new_name: str,
ignore_not_found: bool,
) -> None

Rename the table with the given name to the new name.

method table_column_add

source

table_column_add(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
column_definition: SerializedSchema,
ignore_not_found: bool,
if_column_not_exists: bool,
) -> None

Add a column to the table with the given name.

method table_column_drop

source

table_column_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
column_name: str,
ignore_not_found: bool,
if_column_exists: bool,
cascade: bool,
) -> None

Drop the column from the table with the given name.

method table_column_rename

source

table_column_rename(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
column_name: str,
new_column_name: str,
ignore_not_found: bool,
) -> None

Rename the column in the table with the given name.

method table_column_default_set

source

table_column_default_set(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
column_name: str,
expression: SqlExpression,
ignore_not_found: bool,
) -> None

Set the default expression for the column.

method table_column_default_drop

source

table_column_default_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
column_name: str,
ignore_not_found: bool,
) -> None

Drop the default expression for the column.

method table_column_type_change

source

table_column_type_change(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
column_definition: SerializedSchema,
expression: SqlExpression | None,
ignore_not_found: bool,
) -> None

Change the type of the column in the table with the given name.

The name of the column to change is taken from the field in the provided schema.

method table_not_null_drop

source

table_not_null_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
column_name: str,
ignore_not_found: bool,
) -> None

Drop the NOT NULL constraint from the column.

method table_not_null_set

source

table_not_null_set(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
column_name: str,
ignore_not_found: bool,
) -> None

Set the NOT NULL constraint on the column.

method table_scan_function_get

source

table_scan_function_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
at_unit: str | None,
at_value: str | None,
) -> ScanFunctionResult

Get the ScanFunctionResult for scanning the table.

Returns information about the VGI table function to call when scanning this table. The at_unit and at_value support time travel queries.

method table_scan_branches_get

source

table_scan_branches_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
at_unit: str | None,
at_value: str | None,
) -> ScanBranchesResult

Get the list of scan branches for a multi-source table.

Multi-branch tables compose a logical scan from N physical sources (canonical case: Kafka hot tier + Iceberg cold tier). The VGI DuckDB extension’s optimizer-extension rewrites the placeholder LogicalGet into LogicalSetOperation(UNION_ALL, ...), one arm per branch.

Default implementation: delegate to :meth:table_scan_function_get and wrap the single ScanFunctionResult as a one-branch list. This makes every existing single-source worker automatically compatible with the new branches-aware C++ side, while letting workers that genuinely need multi-source override this method.

Workers that override should NOT also raise from :meth:table_scan_function_get — the legacy method must keep working for old C++ extensions that don’t yet probe for the new branches RPC. Common pattern: a worker implements both, where :meth:table_scan_function_get returns branches[0] (the primary branch) and :meth:table_scan_branches_get returns the full list.

Parameters

attach_opaque_data

Per-attach session token.

transaction_opaque_data

Optional transaction token.

schema_name

Schema containing the table.

name

Table name.

at_unit

Optional time-travel unit (e.g., “VERSION” / “TIMESTAMP”). The VGI C++ side refuses AT(…) on multi-branch tables (>1 branch) at bind time, so workers returning multiple branches should expect at_unit / at_value to always be None; single-branch returns still honour them.

at_value

Optional time-travel value matching at_unit.

Returns

class:ScanBranchesResult carrying one or more class:ScanBranch entries plus the union of required extensions across all branches.

method table_column_statistics_get

source

table_column_statistics_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> TableColumnStatisticsResult | None

Get column statistics for all columns in a table.

Returns a :class:TableColumnStatisticsResult containing per-column statistics and an optional cache TTL, or None if statistics are not available for this table.

The default implementation returns None (no statistics). Workers that provide statistics should override this method.

method table_insert_function_get

source

table_insert_function_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
writable_branch_function_name: str | None = None,
) -> ScanFunctionResult

Get the write function for INSERT operations on the table.

Returns a ScanFunctionResult identifying the TableInOutGenerator function to call for inserting rows into this table.

writable_branch_function_name is set by the C++ extension when the table is multi-branch and a branch declared writable=True: the value is the writable arm’s ScanBranch.function_name. Workers serving multi-branch tables can use this to dispatch the INSERT to the correct underlying storage without re-resolving the writable arm internally. For single-branch tables this is None (or unset for legacy overrides).

method table_update_function_get

source

table_update_function_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> ScanFunctionResult

Get the write function for UPDATE operations on the table.

Returns a ScanFunctionResult identifying the TableInOutGenerator function to call for updating rows in this table. Input batches will include a rowid column plus the columns being updated.

method table_delete_function_get

source

table_delete_function_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> ScanFunctionResult

Get the write function for DELETE operations on the table.

Returns a ScanFunctionResult identifying the TableInOutGenerator function to call for deleting rows from this table. Input batches will contain a rowid column identifying the rows to delete.

method view_create

source

view_create(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
definition: str,
on_conflict: OnConflict,
) -> None

Create a new view with the given definition.

method view_drop

source

view_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
ignore_not_found: bool,
cascade: bool = False,
) -> None

Drop the view with the given name.

method view_rename

source

view_rename(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
new_name: str,
ignore_not_found: bool,
) -> None

Rename the view to the new name.

method view_get

source

view_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> ViewInfo | None

Get information about the view with the given name.

Returns a ViewInfo object if the view exists, or None if it does not.

method view_comment_set

source

view_comment_set(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
comment: str | None,
ignore_not_found: bool,
) -> None

Set the comment for the view with the given name.

method macro_get

source

macro_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> MacroInfo | None

Get information about the macro with the given name.

Returns a MacroInfo object if the macro exists, or None if it does not.

method macro_create

source

macro_create(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
macro_type: MacroType,
parameters: list[str],
definition: str,
on_conflict: OnConflict,
parameter_default_values: pa.RecordBatch | None = None,
arguments_schema: pa.Schema | None = None,
) -> None

Create a new macro with the given definition.

Parameters

attach_opaque_data

Per-attach catalog session token.

transaction_opaque_data

Optional transaction handle.

schema_name

Schema to create the macro in.

name

Name for the new macro.

macro_type

Whether this is a scalar or table macro.

parameters

Ordered list of parameter names.

definition

SQL expression (scalar) or query (table).

on_conflict

Behavior if the macro already exists.

parameter_default_values

One-row RecordBatch with typed defaults.

arguments_schema

Optional Arrow schema (one nullable field per parameter, in parameters order) carrying per-parameter descriptions via the vgi_doc field metadata key. None when no per-parameter docs are supplied.

method macro_drop

source

macro_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
ignore_not_found: bool,
) -> None

Drop the macro with the given name.

method index_get

source

index_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> IndexInfo | None

Get information about the index with the given name.

Returns an IndexInfo object if the index exists, or None if it does not. The default implementation returns None (no indexes).

method index_create

source

index_create(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
table_name: str,
index_type: str,
constraint_type: IndexConstraintType,
expressions: list[str],
on_conflict: OnConflict,
options: dict[str, str] | None = None,
) -> None

Create a new index on the specified table.

method index_drop

source

index_drop(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
ignore_not_found: bool,
cascade: bool = False,
) -> None

Drop the index with the given name.

method copy_from_formats

source

copy_from_formats(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
) -> list[CopyFromFormatInfo]

List custom COPY ... FROM formats advertised by this catalog.

Catalog-level (not schema-scoped). The default returns an empty list, so catalogs that don’t define COPY formats are unaffected. The VGI extension registers one DuckDB CopyFunction per returned entry at ATTACH time.

class CatalogObject

source

Description

All objects have the following common properties.

Attributes

attribute comment

str | None

This is a generic comment about the object.

attribute tags

dict[str, str]

These are key-value tags associated with the object.

class CatalogSchemaObject

source

Bases: CatalogObject

Description

Objects that exist within a schema have the following common properties.

Attributes

attribute name

str

The name of the object.

attribute schema_name

str

The name of the schema containing the object.

Inherited members (2)

• comment attribute · from CatalogObject
• tags attribute · from CatalogObject

class CatalogStorage

source

Bases: Protocol

Description

Storage protocol for VGI catalog state persistence.

Provides two access patterns for catalog state:

Attachments - Track catalog attachments with their options. Stores the mapping from attach_opaque_data to catalog name and options.

Transactions - Track active transactions. Stores transaction state for catalogs that support transactions.

Methods

method attach_put

source

attach_put(
attach_opaque_data: AttachOpaqueData,
catalog_name: str,
options: dict[str, Any],
) -> None

Store attachment state.

Parameters

attach_opaque_data

Unique identifier for the attachment.

catalog_name

Name of the attached catalog.

options

Options passed during attachment.

method attach_get

source

attach_get(
attach_opaque_data: AttachOpaqueData,
) -> tuple[str, dict[str, Any]] | None

Retrieve attachment state by attach_opaque_data.

Parameters

attach_opaque_data

Unique identifier for the attachment.

Returns

Tuple of (catalog_name, options), or None if not found.

method attach_delete

source

attach_delete(attach_opaque_data: AttachOpaqueData) -> None

Delete attachment state.

Parameters

attach_opaque_data

Unique identifier for the attachment.

method attach_list

source

attach_list() -> list[AttachOpaqueData]

List all active attachments.

Returns

List of all attach opaque data values in storage.

method transaction_put

source

transaction_put(
transaction_opaque_data: TransactionOpaqueData,
attach_opaque_data: AttachOpaqueData,
state: bytes,
) -> None

Store transaction state.

Parameters

transaction_opaque_data

Unique identifier for the transaction.

attach_opaque_data

Attachment the transaction belongs to.

state

Serialized transaction state.

method transaction_get

source

transaction_get(
transaction_opaque_data: TransactionOpaqueData,
) -> tuple[AttachOpaqueData, bytes] | None

Retrieve transaction state.

Parameters

transaction_opaque_data

Unique identifier for the transaction.

Returns

Tuple of (attach_opaque_data, state bytes), or None if not found.

method transaction_delete

source

transaction_delete(
transaction_opaque_data: TransactionOpaqueData,
) -> None

Delete transaction state.

Parameters

transaction_opaque_data

Unique identifier for the transaction.

class CatalogStorageSqlite

source

Description

SQLite-backed storage for VGI catalog state.

This implementation uses SQLite with WAL mode to allow multiple worker processes to share catalog state. It manages two tables:

• catalog_attachments: Maps attach_opaque_data to catalog name and options
• catalog_transactions: Tracks active transactions

Attributes

attribute db_path

str

Filesystem path to the backing SQLite database file.

Methods

method attach_put

source

attach_put(
attach_opaque_data: AttachOpaqueData,
catalog_name: str,
options: dict[str, Any],
) -> None

Store attachment state.

method attach_get

source

attach_get(
attach_opaque_data: AttachOpaqueData,
) -> tuple[str, dict[str, Any]] | None

Retrieve attachment state by attach_opaque_data.

method attach_delete

source

attach_delete(attach_opaque_data: AttachOpaqueData) -> None

Delete attachment state.

method attach_list

source

attach_list() -> list[AttachOpaqueData]

List all active attachment IDs.

method transaction_put

source

transaction_put(
transaction_opaque_data: TransactionOpaqueData,
attach_opaque_data: AttachOpaqueData,
state: bytes,
) -> None

Store transaction state.

method transaction_get

source

transaction_get(
transaction_opaque_data: TransactionOpaqueData,
) -> tuple[AttachOpaqueData, bytes] | None

Retrieve transaction state.

method transaction_delete

source

transaction_delete(
transaction_opaque_data: TransactionOpaqueData,
) -> None

Delete transaction state.

method generate_attach_opaque_data

source

generate_attach_opaque_data() -> AttachOpaqueData

Generate a new unique attach_opaque_data.

Returns

A new AttachOpaqueData based on UUID4.

method generate_transaction_opaque_data

source

generate_transaction_opaque_data() -> TransactionOpaqueData

Generate a new unique transaction_opaque_data.

Returns

A new TransactionOpaqueData based on UUID4.

method cleanup_old_entries

source

cleanup_old_entries(max_age_days: float = 7.0) -> int

Remove entries older than the specified age from all tables.

Parameters

max_age_days

Maximum age in days for entries to keep.

Returns

Total number of entries deleted.

class ColumnStatistics

source

Description

Statistics for a single column in a table.

Workers provide these to help DuckDB’s optimizer make cost-based decisions (filter elimination, join reordering, etc.).

Attributes

attribute column_name

str

Name of the column these statistics describe.

attribute min

pa.Scalar | None

Minimum value as a typed PyArrow scalar (e.g., pa.scalar(0, pa.int64())), or None if unknown.

attribute max

pa.Scalar | None

Maximum value as a typed PyArrow scalar, or None if unknown. Must have the same Arrow type as min.

attribute has_null

bool

Whether the column contains any null values.

attribute has_not_null

bool

Whether the column contains any non-null values.

attribute distinct_count

int | None

Approximate count of distinct values, or None if unknown.

attribute contains_unicode

bool | None

String/binary columns only — whether values contain non-ASCII characters. None for non-string columns.

attribute max_string_length

int | None

String/binary columns only — maximum byte length of values. None for non-string columns.

class CopyFromFormatInfo

source

Bases: CatalogObject, ArrowSerializableDataclass

Description

A custom COPY ... FROM format advertised by a VGI catalog.

The VGI DuckDB extension registers one DuckDB CopyFunction per advertised format (into the system catalog, keyed by format_name) so users can run COPY target FROM 'path' (FORMAT <format_name>, opt val, ...) and have a worker function parse the source and stream rows into a local table. Discovery is catalog-level via :meth:VgiProtocol.catalog_copy_from_formats; see the C++ vgi_copy_from_impl.cpp and docs/copy_from.md.

Inherits comment and tags from :class:CatalogObject.

Attributes

attribute format_name

str

The FORMAT identifier users type. Lives in a single global namespace shared with built-ins (csv/parquet/json) and every other attached catalog’s formats — collisions are rejected at ATTACH by the extension.

attribute handler

str

Registered name of the worker function that performs the read.

attribute options

SerializedSchema

Serialized Arrow schema of the format’s options, built from the handler’s Arg-annotated arguments (same encoding as :attr:FunctionInfo.arguments); each field’s metadata carries the option type / default / vgi_doc description. The reserved file_path positional is excluded.

attribute direction

str

"from" | "to" | "both" — which COPY direction(s) this format serves; surfaced so the C++ vgi_copy_formats() diagnostic can split FROM vs TO and so registration wires the right callbacks.

attribute description

str

Intrinsic documentation from the handler’s Meta.description.

attribute ordered

bool

COPY … TO only — when true the worker requires rows in source order, so the extension uses a single-threaded sink (REGULAR_COPY_TO_FILE) instead of the default parallel sharded write. Set via Meta.ordered = True on a CopyToFunction. Ignored for readers.

Inherited members (2)

• comment attribute · from CatalogObject
• tags attribute · from CatalogObject

function deserialize_column_statistics

source

deserialize_column_statistics(data: bytes) -> list[ColumnStatistics]

Deserialize column statistics — the inverse of serialize_column_statistics.

Reads the sparse-union wire batch back into typed ColumnStatistics, so a client can consume the bytes returned by catalog_table_column_statistics_get without re-implementing the union layout. See Client.table_column_statistics for the wrapper that fetches and decodes in one call.

The optional cache_max_age_seconds travels in the IPC batch’s custom metadata rather than in the statistics themselves; read it with deserialize_record_batch if you need it.

Parameters

data

IPC-serialized bytes of a statistics RecordBatch.

Returns

Per-column statistics, in the order the worker serialized them. An empty list when the batch carries no rows.

class ForeignKeyDef

source

Description

A foreign key constraint definition.

Attributes

attribute columns

tuple[str, …]

Column names in THIS table that form the FK.

attribute referenced_table

str

Name of the referenced table.

attribute referenced_columns

tuple[str, …]

Column names in the referenced table.

attribute referenced_schema

str | None

Schema of the referenced table. Defaults to None meaning same schema as this table.

class FunctionInfo

source

Bases: CatalogSchemaObject, ArrowSerializableDataclass

Description

Information about a function in a schema.

Attributes

attribute function_type

FunctionType

The type of function from VGI.

attribute arguments

SerializedSchema

The arguments as a serialized Apache arrow schema using schema.serialize().to_pybytes().

attribute output_schema

SerializedSchema

The output schema as a serialized Apache arrow schema using schema.serialize().to_pybytes().

attribute stability

FunctionStability | None

Scalar function behavior field (None for non-scalar functions).

attribute null_handling

NullHandling | None

Scalar function behavior field (None for non-scalar functions).

attribute description

str

Intrinsic documentation from function metadata (Meta.description). The user-settable comment (via COMMENT ON FUNCTION) is inherited from the base object.

attribute examples

list[CatalogExample]

Usage examples for the function.

attribute categories

list[str]

Category labels for the function.

attribute projection_pushdown

bool | None

Table-function capability (None for scalar functions).

attribute filter_pushdown

bool | None

Table-function capability (None for scalar functions).

attribute sampling_pushdown

bool | None

Table-function capability (None for scalar functions).

attribute late_materialization

bool | None

True if the table participates in DuckDB’s late-materialization optimizer (Meta.late_materialization). The DuckDB extension only honours this when the table also exposes a rowid virtual column plus filter/projection pushdown — see GetScanFunctionImpl in the C++ vgi_table_entry.cpp.

attribute supported_expression_filters

list[str]

Expression-filter classes the function can accept pushed down.

attribute order_preservation

OrderPreservation | None

Whether the function preserves input ordering.

attribute max_workers

Annotated[int | None, ArrowType(pa.int32())]

Maximum parallel workers. Uses ArrowType to specify int32 instead of the default int64.

attribute supports_batch_index

bool

True if the function opts in to per-batch vgi_batch_index tagging: the worker emits an integer partition id in each Arrow batch’s KeyValueMetadata; the DuckDB extension threads it through TableFunction::get_partition_data so ordered sinks (BatchCollector, BatchInsert, BatchCopyToFile, Limit) can reassemble parallel output in partition-id order. Opting in also skips the FIXED_ORDER MaxThreads=1 clamp; the source stays parallel and the sink does the ordering.

attribute partition_kind

PartitionKind

Partition shape declared by the function over its vgi.partition_column-annotated bind-schema fields. When non-NOT_PARTITIONED, the DuckDB extension installs TableFunction::get_partition_info returning the corresponding TablePartitionInfo value so the planner can pick PhysicalPartitionedAggregate for GROUP BY queries (today, only SINGLE_VALUE_PARTITIONS materially changes planner behavior). Per-column annotation lives in the bind schema’s field-level metadata — see vgi.schema_utils.partition_field.

attribute order_dependent

OrderDependence

Aggregate function field (future).

attribute distinct_dependent

DistinctDependence

Aggregate function field (future).

attribute supports_window

bool

True if the aggregate implements the window() callback.

attribute streaming_partitioned

bool

True if the aggregate opts into the streaming-partitioned protocol — aggregate_streaming_open / _chunk / _close. The DuckDB extension’s optimizer rule may rewrite eligible LogicalWindow nodes to use this path.

attribute has_finalize

bool

True if a table-in-out function declares a finalize/finish stage. The C++ extension uses this to conditionally register in_out_function_final; DuckDB rejects LATERAL with correlated input on functions that register a finalize callback.

attribute source_order_dependent

bool

Only meaningful when function_type == FunctionType.TABLE_BUFFERING (i.e. the function is registered through the Sink+Source path). When true, the source phase is single-threaded and finalize_state_ids drain in combine-returned order. Default false enables parallel finalize.

attribute sink_order_dependent

bool

Only meaningful when function_type == FunctionType.TABLE_BUFFERING. When true, the SINK phase runs single-threaded — every process() call arrives in source order on one worker. Mutually exclusive with requires_input_batch_index.

attribute requires_input_batch_index

bool

Only meaningful when function_type == FunctionType.TABLE_BUFFERING. When true, the C++ Sink operator declares RequiredPartitionInfo()=BatchIndex(); each process() RPC carries a globally-unique monotonic batch_index from DuckDB’s source. Workers can sort by it in combine() to reconstruct source order under parallel ingest. Mutually exclusive with sink_order_dependent.

attribute input_from_args

bool

True for a blended RowTransformFunction — its positional args ARE its per-row input columns (real typed args, no TABLE placeholder), so one registration serves the literal / column / LATERAL call shapes. Set from RowTransformFunction subclassing; the C++ extension reads it to enter the in-out registration branch with real-typed args and drive the literal single-row scan-mode.

attribute required_settings

list[str]

Settings required by the function.

attribute required_secrets

list[SecretLookupEntry]

Secrets required by the function (each entry has secret_type, optional secret_name, optional scope).

Inherited members (4)

• comment attribute · from CatalogObject
• tags attribute · from CatalogObject
• name attribute · from CatalogSchemaObject
• schema_name attribute · from CatalogSchemaObject

class FunctionType

source

Bases: Enum

Description

The type of function in a schema.

Attributes

attribute SCALAR

A scalar function.

attribute TABLE

A table function.

attribute TABLE_BUFFERING

A table-buffering (table-in-out) function.

attribute AGGREGATE

An aggregate function.

class Index

source

Description

Declarative index definition.

Immutable.

Attributes

attribute name

str

Index name.

attribute table_name

str

Name of the table this index is on.

attribute expressions

tuple[str, …]

SQL expression strings or column names defining the index. For column-based indexes: (“col_a”, “col_b”) For expression indexes: (“lower(col_a)”, “col_b + 1”)

attribute index_type

str

The index type (e.g., “” for default).

attribute constraint_type

IndexConstraintType

NONE for regular, UNIQUE for unique indexes.

attribute options

dict[str, str]

Key-value index options.

attribute comment

str | None

Optional index comment.

attribute tags

dict[str, str]

Optional metadata tags.

Methods

method to_index_info

source

to_index_info(schema_name: str) -> IndexInfo

Convert to IndexInfo for catalog response.

class IndexConstraintType

source

Bases: Enum

Description

The constraint type of an index.

Attributes

attribute NONE

Regular index (no constraint enforcement).

attribute UNIQUE

Index enforces a UNIQUE constraint.

attribute PRIMARY

Index enforces a PRIMARY KEY constraint.

class IndexInfo

source

Bases: CatalogSchemaObject, ArrowSerializableDataclass

Description

Information about an index in a schema.

Attributes

attribute table_name

str

The name of the table this index is on.

attribute index_type

str

The index type string (e.g., “ART”, or empty for default).

attribute constraint_type

IndexConstraintType

The constraint enforcement type (NONE, UNIQUE, PRIMARY).

attribute expressions

list[str]

SQL expression strings defining the indexed expressions. For column-based indexes, these are column references (e.g., “col_a”). For expression indexes, these are arbitrary SQL (e.g., “lower(col_a)”).

attribute options

dict[str, str]

Key-value index options (WITH clause).

Inherited members (4)

• comment attribute · from CatalogObject
• tags attribute · from CatalogObject
• name attribute · from CatalogSchemaObject
• schema_name attribute · from CatalogSchemaObject

class Macro

source

Description

Declarative macro definition.

Attributes

attribute name

str

Macro name.

attribute macro_type

MacroType

Whether this is a scalar or table macro.

attribute parameters

list[str]

Ordered list of parameter names.

attribute parameter_default_values

pa.RecordBatch | None

One-row RecordBatch where columns are parameter names and values are typed defaults. None if no defaults. Example: pa.RecordBatch.from_pydict({“b”: [5]}) for b := 5.

attribute parameter_docs

dict[str, str]

Optional mapping of parameter name to a human/agent-facing description. Keys must appear in parameters. Descriptions flow over the wire via the macro arguments_schema’s vgi_doc field metadata (the same channel functions use), so the DuckDB extension’s vgi_function_arguments() can surface them. Empty/default = no docs.

attribute definition

str

SQL expression (scalar) or query (table).

attribute comment

str | None

Optional macro comment.

attribute tags

dict[str, str]

Optional metadata tags.

Methods

method to_macro_info

source

to_macro_info(schema_name: str) -> MacroInfo

Convert to MacroInfo for catalog response.

class MacroInfo

source

Bases: CatalogSchemaObject, ArrowSerializableDataclass

Description

Information about a macro in a schema.

Attributes

attribute macro_type

MacroType

Whether this is a scalar or table macro.

attribute parameters

list[str]

Ordered list of parameter names.

attribute parameter_default_values

Annotated[pa.RecordBatch | None, ArrowType(pa.binary())]

One-row RecordBatch where column names are parameter names and values are typed defaults. None if no defaults. Serialized as IPC bytes over the wire.

attribute definition

str

The SQL expression (scalar) or query (table).

attribute arguments_schema

Annotated[pa.Schema | None, ArrowType(pa.binary())]

Optional Arrow schema (serialized as IPC bytes) with one nullable field per parameter, in parameters order. Each field’s type is the parameter’s default value type when known (else null), and the vgi_doc field metadata key carries the parameter’s description (UTF-8, presence-only — omitted when undocumented). Mirrors the per-argument doc channel functions expose via FunctionInfo.arguments. None means the worker did not supply per-parameter docs (older workers); the extension falls back to parameters for names. Built with vgi.argument_spec.macro_arguments_schema.

Inherited members (4)

• comment attribute · from CatalogObject
• tags attribute · from CatalogObject
• name attribute · from CatalogSchemaObject
• schema_name attribute · from CatalogSchemaObject

class MacroType

source

Bases: Enum

Description

The type of macro in a schema.

Attributes

attribute SCALAR

A scalar macro.

attribute TABLE

A table macro.

class OnConflict

source

Bases: Enum

Description

Behavior when a conflict occurs during creation of an object.

Attributes

attribute ERROR

Raise an error if the object already exists.

attribute IGNORE

Do nothing if the object already exists.

attribute REPLACE

Replace the existing object if it already exists.

class ReadOnlyCatalogInterface

source

Bases: CatalogInterface

Description

A read-only catalog interface that does not support DDL operations.

This is a convenience base class for catalogs that only support reading metadata and data, not creating or modifying objects.

There are two ways to use this class:

1. Subclass and implement abstract methods:

   • catalogs() - List available catalogs
   • catalog_attach() - Attach to a catalog
   • schema_get() - Get schema information
   • table_get() - Get table information (return None for function-only catalogs)
   • view_get() - Get view information (return None for function-only catalogs)

2. Use with functions list (simpler for function-only catalogs): Set the functions class attribute to expose VGI functions:

   • catalog_name - Name of the catalog (default: “functions”)
   • functions - List of function classes to expose in the “main” schema

   This provides automatic implementations of catalogs(), catalog_attach(), schema_get(), table_get(), view_get(), and schema_contents().

Optional methods that can be overridden:

• catalog_detach() - Custom detach logic
• schemas() - Custom schema listing (default returns ‘main’)
• schema_contents() - List schema contents
• table_scan_function_get() - Get scan function for tables

All DDL operations (create, drop, rename, modify) will raise CatalogReadOnlyError.

Attributes

attribute supports_transactions

Always False – read-only catalogs do not support transactions.

attribute catalog_version_frozen

Always True – the catalog version never changes.

attribute catalog_name

str

Name of the catalog exposed when using the functions-list mode (default "functions").

attribute functions

list[type]

Function classes to expose in the "main" schema.

attribute settings

list[SettingSpec]

DuckDB setting specs the catalog declares.

attribute secret_types

list[SecretTypeSpec]

Secret type specs the catalog declares.

attribute attach_option_specs

list[AttachOptionSpec]

Attach option specs accepted at ATTACH time.

attribute attach_catalogs

list[AttachCatalogInfo]

Companion catalogs (lakehouse federation) the client should ATTACH when this catalog attaches.

attribute catalog

Catalog | None

Optional declarative Catalog object describing the catalog’s schemas, tables, and views.

attribute catalog_create

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute catalog_drop

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute catalog_transaction_begin

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute catalog_transaction_commit

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute catalog_transaction_rollback

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute schema_create

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute schema_drop

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_create

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_drop

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_comment_set

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_column_comment_set

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_rename

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_column_add

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_column_drop

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_column_rename

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_column_default_set

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_column_default_drop

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_column_type_change

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_not_null_drop

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute table_not_null_set

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute view_create

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute view_drop

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute view_rename

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute view_comment_set

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute macro_create

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute macro_drop

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute index_create

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

attribute index_drop

DDL stub that raises [CatalogReadOnlyError`](/vgi/docs/python/api/vgi-exceptions/#vgi.exceptions.CatalogReadOnlyError).

Methods

method catalogs

source

catalogs() -> list[CatalogInfo]

Return the list of available catalogs.

Default discovery record carries just the catalog name — subclasses that want to advertise version metadata should override.

method catalog_attach

source

catalog_attach(
*,
name: str,
options: dict[str, Any],
data_version_spec: str | None,
implementation_version: str | None,
ctx: CallContext | None = None,
) -> CatalogAttachResult

Attach to the catalog. Version constraints are ignored by default.

method schemas

source

schemas(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
) -> list[SchemaInfo]

Get a list of schemas for the given attach_opaque_data.

method schema_get

source

schema_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
name: str,
) -> SchemaInfo | None

Get information about a schema (case-insensitive lookup).

method table_get

source

table_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
at_unit: str | None = None,
at_value: str | None = None,
) -> TableInfo | None

Get information about a table (case-insensitive lookup).

When at_unit / at_value are provided, the default implementation returns the same table info (no schema evolution). Override this method to return version-specific schemas for time-travel queries.

method view_get

source

view_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> ViewInfo | None

Get information about a view (case-insensitive lookup).

method macro_get

source

macro_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> MacroInfo | None

Get information about a macro (case-insensitive lookup).

method index_get

source

index_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> IndexInfo | None

Get information about an index (case-insensitive lookup).

method table_column_statistics_get

source

table_column_statistics_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> TableColumnStatisticsResult | None

Get column statistics from the Table descriptor’s statistics dict.

Automatically resolves plain Python values to typed PyArrow scalars using the column’s Arrow type from the table schema. Override this method for dynamic or computed statistics.

method table_scan_function_get

source

table_scan_function_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
at_unit: str | None,
at_value: str | None,
) -> ScanFunctionResult

Get scan function for a table.

For function-backed tables (Table.function is set), automatically returns a ScanFunctionResult that invokes the linked function.

For tables with explicit columns, override this method in your Worker to provide scan functions.

method table_insert_function_get

source

table_insert_function_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
writable_branch_function_name: str | None = None,
) -> ScanFunctionResult

Get insert function for a table.

method table_update_function_get

source

table_update_function_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> ScanFunctionResult

Get update function for a table.

method table_delete_function_get

source

table_delete_function_get(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
schema_name: str,
name: str,
) -> ScanFunctionResult

Get delete function for a table.

method schema_contents

source

schema_contents(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
name: str,
type: SchemaObjectType,
) -> Sequence[TableInfo | ViewInfo | FunctionInfo | MacroInfo | IndexInfo]

List contents of a schema.

Returns tables, views, functions, macros, or indexes based on the type parameter. Uses case-insensitive schema name lookup.

Parameters

attach_opaque_data

The attachment identifier.

transaction_opaque_data

The transaction identifier, if any.

name

The name of the schema.

type

The type of objects to return. Must be a SchemaObjectType enum.

Returns

A list of TableInfo, ViewInfo, FunctionInfo, MacroInfo, or IndexInfo objects.

method copy_from_formats

source

copy_from_formats(
*,
attach_opaque_data: AttachOpaqueData,
transaction_opaque_data: TransactionOpaqueData | None,
) -> list[CopyFromFormatInfo]

Advertise every custom COPY format registered in this catalog.

Introspects the catalog’s function list for :class:vgi.copy_from_function.CopyFromFunction (direction='from') and :class:vgi.copy_to_function.CopyToFunction (direction='to') subclasses and converts each into a :class:CopyFromFormatInfo. (The RPC name is historical; it returns all directions, distinguished by direction.) The option schema reuses the same argument serialization as :meth:_function_to_info, so option types / defaults / doc descriptions surface identically to vgi_function_arguments().

Inherited members (5)

• interface_feature_flags attribute · from CatalogInterface — Get the feature flags supported by this CatalogInterface.
• loggable_attach_options method · from CatalogInterface — Return a redacted view of attach/create options safe for logs and Sentry breadcrumbs.
• catalog_detach method · from CatalogInterface — Detach from the catalog with the given attach_opaque_data.
• catalog_version method · from CatalogInterface — Get the current catalog version for the given attach_opaque_data and transaction_opaque_data.
• table_scan_branches_get method · from CatalogInterface — Get the list of scan branches for a multi-source table.

class ScanBranch

source

Description

One physical source backing a multi-branch scan.

A branch is one of two kinds:

• Function branch (the default) — function_name names a DuckDB table function bound with positional_arguments/named_arguments.
• Catalog-table branch — function_name is empty ("") and source_table is set; the branch scans the base table source_catalog.source_schema.source_table in an attached catalog (typically an :class:AttachCatalogInfo companion, e.g. a DuckLake table). The extension binds it via the catalog’s own scan function, so a companion’s snapshot/pruning semantics are honored.

The discriminator is simply “source_table present ⇒ catalog-table kind”.

Attributes

attribute function_name

str

The DuckDB function to call for a function branch (e.g., "read_parquet", "iceberg_scan", or a VGI table function). The C++ rewriter resolves this name against DuckDB’s function catalog and binds it at optimize time. Empty string for a catalog-table branch.

attribute positional_arguments

list[pa.Scalar]

Positional arguments as PyArrow scalars, passed through to the function’s bind.

attribute named_arguments

dict[str, pa.Scalar]

Named arguments as PyArrow scalars.

attribute branch_filter

str | None

Optional SQL expression text (parsed by DuckDB’s parser, bound against the branch’s bound column list). The rewriter AND’s this into every scan of this branch BEFORE filter pushdown, so the branch only ever sees rows in its declared scope. Used to make overlapping physical sources (Kafka 7d retention + Iceberg nightly batches with ~24h overlap) non-overlapping at scan time, without changing the worker code. None means unconstrained.

attribute writable

bool

Declares this branch as the INSERT target for the multi-branch table. At most one branch per table may set this true (enforced at catalog-load by the C++ extension — multiple writable arms would violate DuckDB’s single- writable-catalog-per-transaction rule). When no branch is writable, the table is read-only. UPDATE/DELETE/MERGE remain refused on multi-branch tables regardless of this flag; the contract is INSERT-only until cross-arm semantics have customer-driven evidence.

attribute source_catalog

str | None

Catalog-table branch only — the attached catalog name (matches an :attr:AttachCatalogInfo.alias). None for function branches.

attribute source_schema

str | None

Catalog-table branch only — the schema of the source table. None for function branches.

attribute source_table

str | None

Catalog-table branch only — the base table name; its presence selects the catalog-table kind. None for function branches.

attribute ARROW_SCHEMA

pa.Schema

Arrow IPC schema used to (de)serialize this branch over the wire.

Methods

method to_row_dict

source

to_row_dict() -> dict[str, Any]

Convert to a dictionary for batch construction.

Arguments are serialized as nested Arrow IPC bytes (same trick as :class:ScanFunctionResult).

method serialize

source

serialize() -> bytes

Serialize to Arrow IPC bytes (1-row batch using ARROW_SCHEMA).

method deserialize

source

deserialize(batch: pa.RecordBatch) -> Self

Deserialize from a 1-row Arrow RecordBatch.

class ScanBranchesResult

source

Description

Result from getting the list of scan branches for a multi-branch table.

The result tells the VGI DuckDB extension which DuckDB function(s) to call to obtain the data for the table. Each branch is bound independently and the rewriter unions their output.

Attributes

attribute branches

list[ScanBranch]

One ScanBranch per physical source. Order is meaningful for stable diagnostic output (vgi_table_branches()) but not for query semantics (UNION ALL is unordered).

attribute required_extensions

list[str]

Union of all DuckDB extensions needed across all branches (e.g., ["iceberg", "httpfs"]). The C++ side auto-loads unloaded entries before running the rewrite; missing extensions surface the existing extension-load diagnostic. Hoisted to the top level so workers don’t repeat "iceberg" on every branch that uses it.

attribute ARROW_SCHEMA

pa.Schema

Arrow IPC schema used to (de)serialize this result over the wire.

Methods

method to_row_dict

source

to_row_dict() -> dict[str, Any]

Convert to a dictionary for batch construction.

method serialize

source

serialize() -> bytes

Serialize to Arrow IPC bytes (1-row batch using ARROW_SCHEMA).

method deserialize

source

deserialize(batch: pa.RecordBatch) -> Self

Deserialize from a 1-row Arrow RecordBatch.

Empty branches list is rejected — workers must return at least one branch. (See the design memo’s “loud at attach” rule.)

class ScanFunctionResult

source

Description

Result from getting a table scan function.

This result tells the VGI DuckDB extension which DuckDB function to call to obtain the data for a table. This enables catalogs to delegate scanning to any DuckDB function (e.g., read_parquet, iceberg_scan, or a custom VGI table function) with appropriate arguments.

Attributes

attribute function_name

str

The DuckDB function to call (e.g., “read_parquet”).

attribute positional_arguments

list[pa.Scalar]

Positional arguments as PyArrow scalars.

attribute named_arguments

dict[str, pa.Scalar]

Named arguments as PyArrow scalars.

attribute required_extensions

list[str]

DuckDB extensions to load before calling.

attribute ARROW_SCHEMA

pa.Schema

Arrow IPC schema used to (de)serialize this result over the wire.

Methods

method to_row_dict

source

to_row_dict() -> dict[str, Any]

Convert to a dictionary for batch construction.

The arguments field is serialized as nested Arrow IPC bytes.

method serialize

source

serialize() -> bytes

Serialize to Arrow IPC bytes.

method deserialize

source

deserialize(batch: pa.RecordBatch) -> Self

Deserialize from Arrow RecordBatch.

class Schema

source

Description

Declarative schema definition grouping tables, views, functions, macros, and indexes.

Attributes

attribute name

str

Schema name.

attribute comment

str | None

Optional schema comment.

attribute tags

dict[str, str]

Optional metadata tags.

attribute tables

Sequence[Table]

Sequence of Table definitions.

attribute views

Sequence[View]

Sequence of View definitions.

attribute functions

Sequence[type[Function]]

Sequence of Function classes (scalar, table, or aggregate).

attribute macros

Sequence[Macro]

Sequence of Macro definitions.

attribute indexes

Sequence[Index]

Sequence of Index definitions.

Methods

method to_schema_info

source

to_schema_info(attach_opaque_data: AttachOpaqueData) -> SchemaInfo

Convert to SchemaInfo for catalog response.

Populates estimated_object_count from the declared population so the C++ extension’s eager-load gate can choose between bulk LoadEntries and per-name single-entry RPCs without an extra round trip. Functions are partitioned by get_metadata().function_type into the three keys (scalar_function, aggregate_function, table_function) so DuckDB’s per-type catalog probes (a name lookup walks scalar → aggregate → table) skip the bulk RPC for any category the schema doesn’t populate.

Zero counts are load-bearing. Empty declarative collections (e.g. views=()) emit 0 here, which the C++ client treats as a hard guarantee and uses to skip the corresponding bulk + per-name RPCs entirely. Do not “optimize” this into omitting empty keys — absence reads as count=1 (unknown), suppressing the RPC bypass.

class SchemaInfo

source

Bases: CatalogObject, ArrowSerializableDataclass

Description

Information about a schema in a catalog.

Attributes

attribute attach_opaque_data

AttachOpaqueData

The unique id for the attached catalog.

attribute name

str

The name of the schema.

attribute estimated_object_count

dict[str, int] | None

Approximate population per object kind, keyed by the same names the C++ extension uses for its set-cache instrumentation: "table", "view", "scalar_function", "aggregate_function", "table_function", "macro", "index". Used by the client to pick between bulk LoadEntries and per-name single-entry RPCs. Workers may omit the field entirely or any individual key — the client treats absent counts as 1, so unspecified populations bias toward eager bulk-load.

The value 0 is a hard guarantee, not an estimate. When a count is exactly 0 the client skips the corresponding catalog_schema_contents_* bulk RPC entirely and short-circuits per-name lookups (catalog_table_get / catalog_view_get / catalog_index_get). If a worker reports 0 for a kind that actually has entries, SELECT … FROM s.x silently returns “not found” — only declare 0 for kinds the worker knows are empty in its current view of the schema. Cross-session DDL on the same catalog (another connection creating a view in a schema this connection has cached as zero-views) is handled the same way as any other stale catalog cache: vgi_clear_cache() or re-attach. Time-travel AT-clause queries do not honor the bypass — they always issue the per-name RPC because a historical version may have had entries the current view does not.

Inherited members (2)

• comment attribute · from CatalogObject
• tags attribute · from CatalogObject

class SchemaObjectType

source

Bases: Enum

Description

The type of object that can exist within a schema.

Used to filter results from schema_contents().

Attributes

attribute TABLE

A table.

attribute VIEW

A view.

attribute SCALAR_FUNCTION

A scalar function.

attribute TABLE_FUNCTION

A table function.

attribute AGGREGATE_FUNCTION

An aggregate function.

attribute SCALAR_MACRO

A scalar macro.

attribute TABLE_MACRO

A table macro.

attribute INDEX

An index.

class SecretTypeSpec

source

Description

Specification for a custom secret type registered at ATTACH.

Defines the secret type name, description, and parameter schema. The schema is a standard Arrow schema where each field represents a secret parameter (key name -> value type). Fields that should be redacted in SHOW SECRETS are marked with {“redact”: “true”} in their Arrow field metadata.

Example

SecretTypeSpec(

name=“vgi_example”, description=“Example VGI secret for testing”, schema=pa.schema([ pa.field(“secret_string”, pa.string(), metadata={“redact”: “true”}), pa.field(“api_key”, pa.string(), metadata={“redact”: “true”}), pa.field(“port”, pa.int32()), pa.field(“use_ssl”, pa.bool_()), pa.field(“timeout”, pa.float64()), ]),

)

Attributes

attribute name

str

The secret type name (e.g., “vgi_example”).

attribute description

str

Human-readable description.

attribute schema

pa.Schema

Arrow schema defining the secret’s key-value parameters.

attribute ARROW_SCHEMA

pa.Schema

Arrow IPC schema used to (de)serialize this spec over the wire.

Methods

method serialize

source

serialize() -> bytes

Serialize to Arrow IPC bytes.

method deserialize

source

deserialize(batch: pa.RecordBatch) -> Self

Deserialize from Arrow RecordBatch.

function serialize_column_statistics

source

serialize_column_statistics(
stats: list[ColumnStatistics],
cache_max_age_seconds: int | None = None,
) -> bytes

Serialize column statistics into a single RecordBatch with sparse union min/max.

The min and max columns use an Arrow sparse union whose child types are the distinct column types present in stats. This keeps everything in a single IPC stream regardless of how many column types the table has.

Parameters

stats

Per-column statistics to serialize.

cache_max_age_seconds

Optional cache TTL embedded in schema metadata.

Returns

IPC-serialized bytes of the statistics RecordBatch.

class Setting

source

Bases: _DescriptorBase

Description

Descriptor for declarative setting definitions using Annotated.

Use with Annotated type hints to declare settings in a Worker’s Settings class. The Arrow type is resolved from the base type in the Annotated hint. See _DescriptorBase (in vgi.catalog._descriptor_spec) for the desc and arrow_type attributes.

Inherited members (3)

• desc attribute · from _DescriptorBase
• arrow_type attribute · from _DescriptorBase
• extra_spec_kwargs method · from _DescriptorBase — Extra keyword arguments this descriptor passes to its spec factory.

class SettingSpec

source

Bases: _SpecBase

Description

Extracted setting metadata for catalog serialization.

This is the resolved form of a Setting, with all types inferred and ready for serialization. See _SpecBase (in vgi.catalog._descriptor_spec) for the field and wire-format definition.

Inherited members (7)

• name attribute · from _SpecBase
• desc attribute · from _SpecBase
• type attribute · from _SpecBase
• default attribute · from _SpecBase
• ARROW_SCHEMA attribute · from _SpecBase
• serialize method · from _SpecBase — Serialize to Arrow IPC bytes.
• deserialize method · from _SpecBase — Deserialize from Arrow RecordBatch.

class Sql

source

Bases: str

Description

A raw SQL expression, passed through verbatim as a default value.

Use this when the default is a SQL expression rather than a Python literal:

defaults={"created_at": Sql("current_timestamp")}

class Table

source

Description

Declarative table definition.

Immutable. Can be defined in two ways:

1. Explicit columns: Provide columns schema directly.
2. Function-backed: Provide function reference — the schema is derived by calling bind() on the function class. If the function requires arguments, supply them via arguments.

Attributes

attribute name

str

Table name.

attribute columns

pa.Schema | None

Explicit PyArrow schema (mutually exclusive with function).

attribute function

type[TableFunctionGenerator[Any, Any]] | None

TableFunctionGenerator class to derive schema from (mutually exclusive with columns).

attribute arguments

Arguments | None

Arguments to pass when calling bind() on a function-backed table. Required when the function has mandatory parameters.

attribute supports_time_travel

bool

Whether this table supports time-travel (AT-clause) queries.

attribute insert_function

type[TableInOutGenerator[Any, Any]] | None

TableInOutGenerator class backing INSERT. None means INSERT is unsupported.

attribute update_function

type[TableInOutGenerator[Any, Any]] | None

TableInOutGenerator class backing UPDATE. Requires a scan function to provide row IDs. None means UPDATE is unsupported.

attribute delete_function

type[TableInOutGenerator[Any, Any]] | None

TableInOutGenerator class backing DELETE. Requires a scan function to provide row IDs. None means DELETE is unsupported.

attribute not_null

tuple[str, …]

Tuple of column names with NOT NULL constraints.

attribute unique

tuple[tuple[str, …], …]

Tuple of column name tuples for UNIQUE constraints.

attribute check

tuple[str, …]

Tuple of SQL expressions for CHECK constraints.

attribute primary_key

tuple[tuple[str, …], …]

Tuple of column-name tuples forming the primary key. At most one primary-key constraint is allowed.

attribute foreign_key

tuple[ForeignKeyDef, …]

Tuple of ForeignKeyDef foreign-key constraints.

attribute defaults

dict[str, DefaultValue]

Dict mapping column names to default values. Accepts Python literals (str, int, float, bool, None) which are auto-converted, or SqlExpression for raw SQL.

attribute generated_columns

dict[str, str]

Dict mapping column names to SQL expressions for generated (virtual) columns. Generated columns are computed on read by DuckDB and are mutually exclusive with defaults.

attribute column_comments

dict[str, str]

Dict mapping column names to comment strings. Comments are transported as Arrow field metadata and visible via duckdb_columns() in DuckDB.

attribute required_filters

tuple[tuple[str, …], …]

Required WHERE-filter groups in conjunctive normal form — an AND (outer tuple) of OR-groups (inner tuples) of dotted-path column references that MUST appear in a WHERE expression for any scan of this table. A group is satisfied when any one of its paths has a filter; every group must be satisfied. So (("accession_number",), ("ticker", "cik")) means “accession_number AND one of (ticker, cik)”; a single-path group (("country",),) is a plain mandatory filter. Paths are top-level names ("country") or struct subfields ("bbox.xmin", "nested.outer.inner"). Empty (default) means no enforcement. Satisfaction is prefix-based: a present filter on a shorter path satisfies any required path it is a prefix of (a whole-struct filter on bbox satisfies all of bbox.xmin / .xmax / .ymin / .ymax). The VGI DuckDB extension’s optimizer pass consults this at bind time and throws BinderException listing any unsatisfied groups.

attribute statistics

dict[str, ColumnStatisticsInput]

Mapping of column name to ColumnStatisticsInput providing inlined column statistics for the optimizer.

attribute statistics_cache_max_age_seconds

int | None

How long clients may cache the inlined statistics, in seconds. None means cache indefinitely.

attribute cardinality_estimate

int | None

Optional inlined cardinality. When set, the C++ extension uses these values directly and skips the per-bind table_function_cardinality RPC. Use for read-only or slow-changing tables. Leave both this and cardinality_max as None to keep the existing per-bind RPC behavior.

attribute cardinality_max

int | None

Optional inlined maximum cardinality. Same caching contract as cardinality_estimate.

attribute inline_bind

bool

Opt into pre-binding the function during schema_contents and inlining the result on TableInfo.bind_result; the C++ extension then skips the per-scan bind RPC. Only valid when function is a @bind_fixed_schema-decorated TableFunctionGenerator subclass — the decorator’s contract (output is exactly cls.FIXED_SCHEMA, no per-call inputs) matches what’s safe to freeze for the catalog cache lifetime. Setting this on a descriptor whose function is not decorated raises at descriptor build.

attribute comment

str | None

Optional table comment.

attribute tags

dict[str, str]

Optional metadata tags.

attribute resolved_columns

pa.Schema

The resolved column schema (explicit or derived from function).

Methods

method to_table_info

source

to_table_info(schema_name: str) -> TableInfo

Convert to TableInfo for catalog response.

method resolve_column_statistics

source

resolve_column_statistics() -> TableColumnStatisticsResult | None

Resolve the statistics dict into a :class:TableColumnStatisticsResult.

Returns None if no statistics are defined. Otherwise, converts each entry to a :class:ColumnStatistics with properly typed PyArrow scalars inferred from the table’s column schema.

class TableInfo

source

Bases: CatalogSchemaObject, ArrowSerializableDataclass

Description

Information about a table in a schema.

Attributes

attribute columns

SerializedSchema

The columns of the table as a PyArrow schema that is serialized as bytes.

attribute not_null_constraints

Annotated[list[int], ArrowType(pa.list_(pa.int32()))]

Column indices with a NOT NULL constraint. Uses ArrowType to specify int32 instead of the default int64.

attribute unique_constraints

Annotated[list[list[int]], ArrowType(pa.list_(pa.list_(pa.int32())))]

Column-index groups with a UNIQUE constraint.

attribute check_constraints

list[str]

SQL CHECK constraint expressions.

attribute primary_key_constraints

Annotated[list[list[int]], ArrowType(pa.list_(pa.list_(pa.int32())))]

Column-index groups forming the primary key.

attribute foreign_key_constraints

Annotated[list[bytes], ArrowType(pa.list_(pa.binary()))]

Serialized foreign-key constraint specs.

attribute supports_insert

bool

Write-support flag — whether the table supports INSERT.

attribute supports_update

bool

Write-support flag — whether the table supports UPDATE.

attribute supports_delete

bool

Write-support flag — whether the table supports DELETE.

attribute supports_returning

bool

When False (the default), the C++ extension rejects INSERT/UPDATE/DELETE … RETURNING at plan time with a BinderException. Workers that can emit the affected rows from their write functions must opt in by setting this to True.

attribute supports_column_statistics

bool

Statistics capability flag — indicates this table can provide column statistics.

attribute scan_function

Annotated[bytes | None, ArrowType(pa.binary())]

Optional inlined function-discovery result. When populated, the C++ extension uses the cached value and skips the corresponding catalog_table_scan_function_get RPC. Bytes are the IPC payload from ScanFunctionResult.serialize(). Populating this freezes the function args for the lifetime of the catalog cache (until catalog_version bumps); workers whose function args change more frequently than catalog_version (rotating credentials, presigned URLs, per-transaction snapshots) MUST leave it null so the per-bind RPC continues to fire.

attribute insert_function

Annotated[bytes | None, ArrowType(pa.binary())]

Optional inlined INSERT function-discovery result. Same caching contract as scan_function.

attribute update_function

Annotated[bytes | None, ArrowType(pa.binary())]

Optional inlined UPDATE function-discovery result. Same caching contract as scan_function.

attribute delete_function

Annotated[bytes | None, ArrowType(pa.binary())]

Optional inlined DELETE function-discovery result. Same caching contract as scan_function.

attribute cardinality_estimate

Annotated[int | None, ArrowType(pa.int64())]

Optional inlined cardinality estimate. When populated, the C++ extension uses it directly and skips the table_function_cardinality RPC — saving one round-trip per bind. Use for read-only or slow-changing tables where cardinality is statically known. Freezes the cardinality for the catalog cache lifetime (until catalog_version bumps); workers whose cardinality changes faster (e.g. live counters) MUST leave it null.

attribute cardinality_max

Annotated[int | None, ArrowType(pa.int64())]

Optional inlined maximum cardinality. Same caching contract as cardinality_estimate.

attribute column_statistics

Annotated[bytes | None, ArrowType(pa.binary())]

Optional inlined column statistics. When populated, the C++ extension uses the cached value and skips the per-bind / per-table catalog_table_column_statistics_get RPC and the per-scan table_function_statistics RPC. Bytes are the IPC payload from serialize_column_statistics(stats, cache_max_age_seconds). Freezes the resolved stats for the catalog cache lifetime (until catalog_version bumps); workers whose statistics change faster than catalog_version (e.g. live counters, rapidly-mutating dimensions) MUST leave this null so the on-demand RPC continues to fire.

attribute bind_result

Annotated[bytes | None, ArrowType(pa.binary())]

Optional inlined bind result. Bytes are the IPC payload of BindResponse.serialize_to_bytes(). When populated, the C++ extension uses these bytes verbatim and skips the per-scan bind RPC, threading the deserialized BindResult straight into bind_data. The catalog framework only populates this for tables marked Table(inline_bind=True) whose function class is @bind_fixed_schema-decorated — the decorator’s contract (output is exactly cls.FIXED_SCHEMA, no per-call inputs, no opaque_data) matches what’s safe to freeze for the catalog cache lifetime. Functions with custom on_bind are not eligible via the framework path; workers can still inline manually inside their own schema_contents override when the bind output is independently known to be stable.

attribute required_filters

Annotated[list[list[str]], ArrowType(pa.list_(pa.list_(pa.string())))]

Required WHERE-filter groups the VGI extension’s optimizer pass verifies against any scan (conjunctive normal form): the outer list is an AND of groups, each inner group is an OR of dotted-path column references (top-level names like "country" or struct subfields like "bbox.xmin"). A group is satisfied when any one of its paths has a filter; every group must be satisfied. So [["accession_number"], ["ticker", "cik"]] means “accession_number AND one of (ticker, cik)”. Empty (default) means no enforcement — the zero-cost fast path. Satisfaction is prefix-based: a present filter on a shorter dotted path satisfies any required path it’s a prefix of (a whole-struct filter on bbox satisfies every "bbox.*" path). The C++ extension throws BinderException listing any unsatisfied groups.

Inherited members (4)

• comment attribute · from CatalogObject
• tags attribute · from CatalogObject
• name attribute · from CatalogSchemaObject
• schema_name attribute · from CatalogSchemaObject

class View

source

Description

Declarative view definition.

Immutable.

Attributes

attribute name

str

View name.

attribute definition

str

SQL definition of the view.

attribute comment

str | None

Optional view comment.

attribute column_comments

dict[str, str]

Optional mapping of view output column name to comment. The extension aligns these by name against the columns DuckDB binds from the view’s query, so only the names that actually appear in the result need entries; unmatched names are ignored.

attribute tags

dict[str, str]

Optional metadata tags.

Methods

method to_view_info

source

to_view_info(schema_name: str) -> ViewInfo

Convert to ViewInfo for catalog response.

class ViewInfo

source

Bases: CatalogSchemaObject, ArrowSerializableDataclass

Description

Information about a view in a schema.

Attributes

attribute definition

str

The definition of the view which is a SQL query string.

attribute column_comments

dict[str, str]

Per-column comments, keyed by the view’s output column name. Unlike tables (whose column comments ride along as Arrow field metadata on the serialized columns schema), a view ships only its SQL definition — DuckDB binds that query to derive the columns — so view column comments need their own channel. The C++ extension aligns these by name against the bound output columns and feeds them into CreateViewInfo.column_comments_map; names that don’t match a bound column are ignored.

Inherited members (4)

• comment attribute · from CatalogObject
• tags attribute · from CatalogObject
• name attribute · from CatalogSchemaObject
• schema_name attribute · from CatalogSchemaObject