Protocol & metadata

On this page

Enums, protocol shapes, secrets, and the function metadata DuckDB reads.

const arrowStateSerializer

source

const arrowStateSerializer: StateSerializer

Description

Arrow IPC state serializer — stores all data as native binary columns.

function arrowToMetadatas

source

export function arrowToMetadatas(batch: VgiBatch): ResolvedMetadata[]

Description

Deserialize Arrow RecordBatch to ResolvedMetadata array.

interface BindRequest

source

export interface BindRequest

Fields

function_namestring

argumentsArguments

function_typeFunctionType

input_schemaVgiSchema | null

settingsVgiBatch | null

secretsVgiBatch | null

attach_opaque_dataUint8Array | null

transaction_opaque_dataUint8Array | null

resolved_secrets_providedboolean

copy_fromCopyFromContext | nulloptional

COPY … FROM context — null/absent unless this bind/init opens a COPY-FROM scan. Additive + name-keyed, so ordinary scans (which omit it on the wire) deserialize to null. Mirrors vgi-python’s BindRequest.copy_from.

copy_toCopyToContext | nulloptional

COPY … TO context — null/absent unless this bind/init opens a COPY-TO sink. Additive + name-keyed, same wire-safe rationale as copy_from. Mirrors vgi-python’s BindRequest.copy_to.

at_unitstring | nulloptional

Time travel: the AT (TIMESTAMP|VERSION …) clause for this scan, threaded from DuckDB’s per-reference bind. Both null when the scan has no AT clause. For inline-bound (function-backed) tables the actual on_bind RPC runs once at attach with no AT, so the per-scan AT is carried on the bind request embedded in each InitRequest — read it at init via init_call.bind_call.at_unit (or TableProcessParams.atUnit). Mirrors vgi-python’s BindRequest.at_unit.

at_valuestring | nulloptional

schema_namestring | nulloptional

Catalog schema that declares the function being bound. A worker may register the same name in more than one schema, so the bare name does not identify an implementation — resolution is by (schema_name, function_name). Null for callers with no catalog context (COPY handler binds). Mirrors vgi-python’s BindRequest.schema_name.

interface BindResponse

source

export interface BindResponse

Fields

output_schemaVgiSchema

opaque_dataUint8Array | null

lookup_secret_typesstring[]optional

lookup_scopesstring[]optional

lookup_namesstring[]optional

const BindResultSchema

source

const BindResultSchema

enum CatalogFunctionType

source

export enum CatalogFunctionType {
SCALAR = "SCALAR",
TABLE = "TABLE",
TABLE_BUFFERING = "TABLE_BUFFERING",
AGGREGATE = "AGGREGATE",
}

interface CopyFromContext

source

export interface CopyFromContext

Description

Context for a COPY ... FROM read, threaded onto {@link BindRequest}.

Present only when the bind/init opens a COPY-FROM scan (null/absent otherwise — set by the VGI extension’s copy_from_bind). InitRequest embeds the same BindRequest as bind_call, so process()/init also reach it via initCall.bind_call.copy_from. The handler’s options arrive through the normal BindRequest.arguments (built from the COPY options), so they are not duplicated here. Mirrors vgi-python’s protocol.CopyFromContext.

Fields

formatstring

The FORMAT name resolved at COPY bind time.

file_pathstring

The source path from the COPY ... FROM 'path' statement.

expected_schemaVgiSchema

The COPY target’s column schema (name + type, in target order). The worker must bind its output to, and emit columns whose types match, this schema exactly — DuckDB inserts no cast between the scan and the INSERT. Holds the parsed Arrow Schema (the wire field is binary; deserialized here).

interface CopyToContext

source

export interface CopyToContext

Description

Context for a COPY ... TO write, threaded onto {@link BindRequest}.

Present only when the bind/init opens a COPY-TO sink (null/absent otherwise — set by the VGI extension’s copy_to_bind). InitRequest embeds the same BindRequest as bind_call, so process()/combine() also reach it via initCall.bind_call.copy_to. The handler’s options arrive through the normal BindRequest.arguments; the source columns ride the existing BindRequest.input_schema (so they are not duplicated here). Mirrors vgi-python’s protocol.CopyToContext.

Fields

formatstring

The FORMAT name resolved at COPY bind time.

file_pathstring

The destination path from the COPY ... TO 'path' statement.

const DEFAULT_MAX_WORKERS

source

const DEFAULT_MAX_WORKERS = 99999

function deserializeUserState

source

export function deserializeUserState(bytes: Uint8Array | null): any

Description

Deserialize userState from Arrow IPC bytes. Reconstructs a plain JS object, preserving BigInt for Int64 columns.

enum DistinctDependence

source

export enum DistinctDependence {
DISTINCT_DEPENDENT = "DISTINCT_DEPENDENT",
NOT_DISTINCT_DEPENDENT = "NOT_DISTINCT_DEPENDENT",
}

const EXCHANGE_STATE_SCHEMA

source

const EXCHANGE_STATE_SCHEMA

Description

Schema for the exchange state carried in HTTP state tokens.

interface FunctionExample

source

export interface FunctionExample

Fields

sqlstring

descriptionstring

expectedOutputstringoptional

interface FunctionMeta

source

export interface FunctionMeta

Fields

namestring

descriptionstringoptional

stabilityFunctionStabilityoptional

nullHandlingNullHandlingoptional

examplesFunctionExample[]optional

categoriesstring[]optional

tagsRecord<string, string>optional

projectionPushdownbooleanoptional

filterPushdownbooleanoptional

samplingPushdownbooleanoptional

lateMaterializationbooleanoptional

table (generator): opt in to DuckDB’s late-materialization rewrite. A TOP_N/LIMIT/SAMPLE over a rowid-bearing table is rewritten into a SEMI join — a narrow ordering scan selects survivors, then the wide scan re-fetches their columns with the surviving rowids pushed down. Surfaces as FunctionInfo late_materialization. Only honoured by the C++ extension for tables whose worker also guarantees a UNIQUE, snapshot-stable rowid.

supportedExpressionFiltersstring[]optional

autoApplyFiltersbooleanoptional

preservesOrderOrderPreservationoptional

maxWorkersnumberoptional

requiredSettingsstring[]optional

requiredSecretsstring[]optional

orderDependentOrderDependenceoptional

distinctDependentDistinctDependenceoptional

hasFinalizebooleanoptional

For table_in_out functions: whether the user defined a finalize callback. DuckDB issues a separate FINALIZE init() phase only when this is true; otherwise calling FinalExecute is unsupported and crashes the C++ side.

inputFromArgsbooleanoptional

Blended (“UNNEST-style”) table-in-out: the function’s positional args ARE its per-row input columns (real typed args, no synthetic TABLE placeholder), so ONE registration serves f(52,13) (literal -> 1 input row), FROM t, f(t.x, t.y) (columns -> streaming), and LATERAL f(t.x,t.y). Set by {@link defineRowTransformFunction }; surfaces as FunctionInfo.input_from_args. The C++ extension reads it to enter the in-out registration branch with real-typed args and drive the literal single-row scan-mode. Mirrors vgi-python’s RowTransformFunction / ResolvedMetadata.input_from_args.

sinkOrderDependentbooleanoptional

table_buffering: force ParallelSink=false in the C++ operator (single-thread, source-ordered ingest).

sourceOrderDependentbooleanoptional

table_buffering: force serial Source drain in finalize_queue order (ParallelSource=false, SourceOrder=FIXED_ORDER).

requiresInputBatchIndexbooleanoptional

table_buffering: thread DuckDB’s per-chunk batch_index into every process() call (RequiredPartitionInfo=BatchIndex).

supportsBatchIndexbooleanoptional

table (generator): the function tags every emitted Arrow batch with a per-partition vgi_batch_index so DuckDB’s ordered sinks reassemble parallel output in partition order. Surfaces as FunctionInfo supports_batch_index.

partitionKind“NOT_PARTITIONED” | “SINGLE_VALUE_PARTITIONS” | “OVERLAPPING_PARTITIONS” | “DISJOINT_PARTITIONS”optional

table (generator): Hive-style partition-columns mode. Functions declare a PartitionKind and annotate bind-schema fields; emitted batches carry vgi_partition_values#b64 metadata. Surfaces as FunctionInfo partition_kind.

copyFromFormatstringoptional

COPY … FROM custom format reader: the SQL FORMAT identifier this function backs (e.g. example_lines). Set by {@link defineCopyFromFunction }. When present, the catalog advertises this function via copyFromFormats() / the catalog_copy_from_formats RPC so the VGI extension registers a DuckDB CopyFunction for it. Mirrors vgi-python’s CopyFromFunction.COPY_FROM_FORMAT.

copyFromDirectionstringoptional

COPY direction; only "from" is supported today. Default "from".

copyFromCommentstring | nulloptional

Optional free-text comment surfaced by vgi_copy_formats().

copyToFormatstringoptional

COPY … TO custom format writer: the SQL FORMAT identifier this function backs (e.g. example_lines_out). Set by {@link defineCopyToFunction }. When present, the catalog advertises this function via copyFromFormats() (the catalog_copy_from_formats RPC returns all directions) so the VGI extension registers a DuckDB CopyFunction for it. The writer is a table_buffering function under the hood, reusing the table_buffering_process / table_buffering_combine RPCs. Mirrors vgi-python’s CopyToFunction.COPY_TO_FORMAT.

copyToDirectionstringoptional

COPY direction for a TO writer; always "to". Default "to".

copyToCommentstring | nulloptional

Optional free-text comment surfaced by vgi_copy_formats().

class FunctionRegistry

source

export class FunctionRegistry

Methods

method register

source

register(func: VgiFunction): void

method registerInSchema

source

registerInSchema(func: VgiFunction, schemaName: string, catalogName?: string): void

Record that func is declared in schemaName, in addition to the flat by-name index. Idempotent, and independent of register() so a function reachable from several schemas (e.g. a scan function referenced by tables in more than one schema) resolves from any of them.

method schemasFor

source

schemasFor(functionName: string): string[]

Schemas that declare functionName, sorted — for error messages.

method get

source

get(name: string, context?: OverloadContext): VgiFunction

method has

source

has(name: string): boolean

method names

source

names(): string[]

method all

source

all(): VgiFunction[]

enum FunctionStability

source

export enum FunctionStability {
CONSISTENT = "CONSISTENT",
VOLATILE = "VOLATILE",
CONSISTENT_WITHIN_QUERY = "CONSISTENT_WITHIN_QUERY",
}

enum FunctionType

source

export enum FunctionType {
SCALAR = "scalar",
TABLE = "table",
// Sink+source function bound through defineTableBufferingFunction. Mirrors
// vgi-python's FunctionType.TABLE_BUFFERING; without it a buffering bind fell
// through deserializeBindRequest's normalization to a bare `as` cast.
TABLE_BUFFERING = "table_buffering",
AGGREGATE = "aggregate",
}

interface GlobalInitResponse

source

export interface GlobalInitResponse

Fields

max_workersnumber

execution_idUint8Array

opaque_dataUint8Array | null

interface HandlerState

source

export interface HandlerState<T = any>

Description

Convention for handler state: mutable user state lives in .state.

Fields

stateT

Serializable user state for HTTP exchange round-trips.

interface InitRequest

source

export interface InitRequest

Fields

bind_callBindRequest

output_schemaVgiSchema

bind_opaque_dataUint8Array | null

projection_idsnumber[] | null

pushdown_filtersVgiBatch | null

join_keysVgiBatch[]

Join-key value batches, one per join-keys column. Keyed by the column name inside each batch’s schema. Populated when DuckDB promotes IN/OR lists or join predicates to batched join-keys pushdowns.

phaseTableInOutPhase | null

finalize_state_idUint8Array | null

Buffered-table finalize stream: which finalize_state_id this stream serves. Set when phase=TABLE_BUFFERING_FINALIZE; null otherwise. Opaque bytes the worker’s combine() chose.

execution_idUint8Array | null

init_opaque_dataUint8Array | null

substream_idUint8Array | null

Per-substream identity for parallel streaming table-in-out functions. A stable, CLIENT-minted id for one substream (one DuckDB PipelineExecutor), identical across this substream’s init / every process tick / finalize. Unlike a worker-minted execution_id, it survives an HTTP load balancer dispatching each request to an arbitrary backend: a finalize that lands on a different backend than the process() calls can still key the substream’s accumulated state (in shared storage) by substream_id. null when the client did not supply one (serial path, non-table-in-out functions, old clients). Mirrors vgi-python’s InitRequest.substream_id.

order_by_column_namestring | null

order_by_directionOrderByDirection | null

order_by_null_orderOrderByNullOrder | null

order_by_limitbigint | null

tablesample_percentagenumber | null

tablesample_seedbigint | null

function metadatasToArrow

source

export function metadatasToArrow(metadatas: ResolvedMetadata[]): VgiBatch

Description

Serialize multiple ResolvedMetadata to a single Arrow RecordBatch.

enum NullHandling

source

export enum NullHandling {
DEFAULT = "DEFAULT",
SPECIAL = "SPECIAL",
}

enum OrderDependence

source

export enum OrderDependence {
ORDER_DEPENDENT = "ORDER_DEPENDENT",
NOT_ORDER_DEPENDENT = "NOT_ORDER_DEPENDENT",
}

enum OrderPreservation

source

export enum OrderPreservation {
/** Output rows are in same order as input rows (DuckDB INSERTION_ORDER). */
PRESERVES_ORDER = "PRESERVES_ORDER",
/** Output order is undefined; may be reordered (DuckDB NO_ORDER). */
NO_ORDER_GUARANTEE = "NO_ORDER_GUARANTEE",
/** Output is in a fixed mandatory order; DuckDB serialises the pipeline
 *  (single worker) to preserve it (DuckDB FIXED_ORDER). */
FIXED_ORDER = "FIXED_ORDER",
}

interface ParameterInfo

source

export interface ParameterInfo

Fields

namestring

positionnumber | null

positionNamestring | null

typeNamestring | null

descriptionstring

requiredboolean

defaultstring | null

constraintsstring | null

isTableInputboolean

isVarargsboolean

isConstboolean

interface ResolvedMetadata

source

export interface ResolvedMetadata

Fields

namestring

classNamestring

functionTypeCatalogFunctionType

descriptionstring

examplesFunctionExample[]

categoriesstring[]

tagsRecord<string, string>

parametersParameterInfo[]

stabilityFunctionStability

nullHandlingNullHandling

requiredSettingsstring[]

requiredSecretsstring[]

projectionPushdownboolean

filterPushdownboolean

samplingPushdownboolean

supportedExpressionFiltersstring[]

preservesOrderOrderPreservation

maxWorkersnumber | null

orderDependentOrderDependence

distinctDependentDistinctDependence

function resolveMetadata

source

export function resolveMetadata(func: VgiFunction): ResolvedMetadata

const ScanFunctionResultSchema

source

const ScanFunctionResultSchema = schema([
field("function_name", utf8(), false),
field("arguments", binary(), false),
field("required_extensions", list(field("item", utf8(), true)), false),
])

type SecretFields

source

export type SecretFields = Record<string, any>;

function secretForScope

source

export function secretForScope(secrets: SecretsDict, path: string): SecretFields | undefined

Description

The secret whose scope is the longest prefix of path. The connector serializes each secret’s scope as a newline-joined list of prefixes; a secret with no (or empty) scope matches as a last-resort fallback. Returns undefined only when there are no candidate secrets.

function secretForScopeOfType

source

export function secretForScopeOfType(
secrets: SecretsDict,
path: string,
type: string,
): SecretFields | undefined

Description

Like {@link secretForScope} but only over secrets of type.

type SecretsDict

source

export type SecretsDict = Record<string, SecretFields>;

function secretsOfType

source

export function secretsOfType(secrets: SecretsDict, type: string): SecretFields[]

Description

Every resolved secret whose serialized type field matches type.

function secretType

source

export function secretType(secrets: SecretsDict, name: string): string | undefined

Description

The DuckDB secret type of the named secret (its serialized type field).

function serializeUserState

source

export function serializeUserState(userState: any): Uint8Array | null

Description

Serialize userState to Arrow IPC bytes. Infers schema from the JS object at runtime. Arrow IPC is self-describing, so deserialization doesn’t need the schema ahead of time.

For an empty object {}, we emit a 0-row batch with empty schema — deserializeUserState recognizes that shape and returns {} rather than null. (null is reserved for “no userState declared at all”.)

interface StreamHandlers

source

export interface StreamHandlers

Fields

outputSchemaVgiSchema

producerInit() => anyoptional

producerFn(state: any, out: OutputCollector) => void | Promise<void>optional

exchangeInit() => anyoptional

exchangeFn( state: any, input: VgiBatch, out: OutputCollector ) => void | Promise<void>optional

inputSchemaVgiSchemaoptional

onTick(state: any, tickMetadata: Map<string, string> | undefined) => void | Promise<void>optional

Fires once per tick batch on the producer path, before producerFn. Receives the tick’s Arrow custom metadata so the handler can pick up per-tick signals like vgi_pushdown_filters (dynamic filter updates from DuckDB’s Top-N optimizer).

interface TableCardinality

source

export interface TableCardinality

Fields

estimatenumber | null

maxnumber | null

enum TableInOutPhase

source

export enum TableInOutPhase {
INPUT = "INPUT",
FINALIZE = "FINALIZE",
// Sink+Source (TableBufferingFunction) init phases. TABLE_BUFFERING is the
// sink-side init (persist init metadata so any pool worker can serve
// process/combine); TABLE_BUFFERING_FINALIZE is the per-finalize_state_id
// Source-stream init.
TABLE_BUFFERING = "TABLE_BUFFERING",
TABLE_BUFFERING_FINALIZE = "TABLE_BUFFERING_FINALIZE",
}

interface VgiFunction

source

export interface VgiFunction

Description

A resolved, registered VGI function definition.

Fields

kind“scalar” | “table” | “table_in_out” | “table_buffering”

metaFunctionMeta

argumentSpecsArgumentSpec[]

bind(request: BindRequest): BindResponse | Promise<BindResponse>

globalInit(request: InitRequest): GlobalInitResponse | Promise<GlobalInitResponse>

createStreamHandlers( request: InitRequest, response: GlobalInitResponse, accumulatedState?: any, ): StreamHandlers

defaultOutputSchemaVgiSchemaoptional

Default output schema (for catalog registration). May be overridden at bind time.

cardinality(request: TableFunctionCardinalityRequest): TableCardinality | Promise<TableCardinality>optional

statistics(request: TableFunctionCardinalityRequest): import(“../util/statistics.js”).ColumnStatistics[] | nulloptional

Per-column statistics for this table function’s output given the user’s bind-time arguments. Returns null/[] when stats are unknown. Wired to the table_function_statistics RPC; DuckDB uses the bounds for plan-time filter elimination.

dynamicToString(request: DynamicToStringRequest): Record<string, string> | Promise<Record<string, string>>optional

Per-execution diagnostics for EXPLAIN ANALYZE. DuckDB calls this at pipeline FinishSource via the table_function_dynamic_to_string RPC. Returns ordered key→value strings; the C++ extension merges these with the intrinsic keys (Function, Rows Read, Threads).