Protocol & metadata

On this page

Function metadata and the on-the-wire request/response types.

struct ArgSpec

source

type ArgSpec struct {
// Name is the argument name (empty for positional-only).
Name  string
// Position is the positional index (0-based). -1 for named-only.
Position  int
// ArrowType is the Arrow type string (e.g., "int64", "varchar", "any").
ArrowType  string
// Doc is a documentation string.
Doc  string
// IsConst is true for constant (scalar) parameters.
IsConst  bool
// IsVarargs is true for variadic parameters.
IsVarargs  bool
// HasDefault is true if the parameter has a default value.
HasDefault  bool
// DefaultValue is the string representation of the default.
DefaultValue  string
// ArrowDataType is an optional concrete Arrow DataType for the argument.
// When set, it takes precedence over ArrowType for schema building.
// Use this for complex types like structs where the string representation
// is insufficient (e.g., arrow.StructOf(...) for typed struct params).
ArrowDataType  arrow.DataType
// TypeBound is an optional slice of type predicates for "any"-typed arguments.
// At bind time, the input schema field type must satisfy at least one predicate
// (OR logic). Nil means no type constraint (any type is accepted).
TypeBound  []TypeBoundPredicate

// Choices is an optional closed set of allowed values for this argument.
// Surfaced as the vgi_choices field-metadata key (JSON array). Element
// values should match the argument's value type. Nil/empty = unrestricted.
Choices  []any
// Ge is an optional inclusive lower bound (value >= Ge). nil = absent.
Ge  *float64
// Le is an optional inclusive upper bound (value <= Le). nil = absent.
Le  *float64
// Gt is an optional exclusive lower bound (value > Gt). nil = absent.
Gt  *float64
// Lt is an optional exclusive upper bound (value < Lt). nil = absent.
Lt  *float64
// Pattern is an optional regex the value must match. Surfaced as the
// vgi_pattern field-metadata key (raw). Empty = no pattern constraint.
Pattern  string
}

Description

ArgSpec describes a single argument in a function’s signature.

Methods

function DeriveArgSpecs

source

func DeriveArgSpecs(args any) []ArgSpec

DeriveArgSpecs returns []ArgSpec derived from the struct’s vgi:"..." tags. args may be a struct value or pointer to one. Panics on malformed tags; these are developer errors that should fail at startup.

function getArgSpecsFromFn

source

func getArgSpecsFromFn(fn interface{}) []ArgSpec

getArgSpecsFromFn extracts ArgumentSpecs from any function type.

function parseArgTag

source

func parseArgTag(f reflect.StructField, tag string) (ArgSpec, error)

parseArgTag parses one vgi:"..." tag, applying Go-type-based inference for unspecified fields (name, ArrowType, ArrowDataType).

function writableArgumentSpecs

source

func writableArgumentSpecs() []ArgSpec

type DistinctDependence

source

type DistinctDependence string

Description

DistinctDependence declares whether a DISTINCT modifier changes an aggregate’s result. Wire-protocol dictionary constants — must not be changed.

struct FinalizeProducerState

source

type FinalizeProducerState struct {
Recipe    InitRecipe    // exported, serialized
BatchIPC  [][]byte    // exported, serialized finalize batch IPC bytes
BatchIdx  int      // exported, current emission position
batches    []arrow.RecordBatch  // transient (deserialized from BatchIPC)
// CacheMeta, when non-nil, is attached to every emitted finalize batch.
// Set from FunctionMetadata.CacheControl so a table-buffering function can
// advertise vgi.cache.* on its finalize output for the exchange-mode
// result cache (the C++ side latches the first batch's keys). Exported so
// it survives the HTTP state-token round-trip.
CacheMeta  map[string]string
}

Description

FinalizeProducerState implements ProducerState for table-in-out FINALIZE phase.

Methods

method Produce

source

func (s *FinalizeProducerState) Produce(ctx context.Context, out *vgirpc.OutputCollector, callCtx *vgirpc.CallContext) error

Produce emits the next buffered finalize batch to the output collector, finishing the stream once all batches have been emitted.

struct FunctionMetadata

source

type FunctionMetadata struct {
// Description is a human-readable description.
Description  string
// Stability controls caching/optimization hints.
Stability  FunctionStability
// NullHandling controls whether NULLs are passed to the function.
NullHandling  NullHandling
// ProjectionPushdown indicates support for projection pushdown.
ProjectionPushdown  bool
// FilterPushdown indicates support for filter pushdown.
FilterPushdown  bool
// SamplingPushdown indicates support for TABLESAMPLE SYSTEM pushdown.
SamplingPushdown  bool
// LateMaterialization advertises that the function participates in DuckDB's
// late-materialization rewrite. DuckDB only honours this when the function
// also exposes a rowid virtual column and supports projection + filter
// pushdown; the rowid must be unique, deterministic, and snapshot-stable.
LateMaterialization  bool
// SupportedExpressionFilters lists DuckDB expression names that the
// function can absorb into its scan (e.g. "&&", "list_contains",
// "starts_with"). Without this, DuckDB inserts a separate FILTER node.
SupportedExpressionFilters  []string
// SupportsWindow indicates an aggregate also implements the windowed
// callbacks (window_init, window, window_destructor). Ignored for
// non-aggregate functions.
SupportsWindow  bool
// StreamingPartitioned indicates an aggregate also implements the
// streaming-partitioned protocol (aggregate_streaming_open/_chunk/_close).
// DuckDB's optimizer can replace a LogicalWindow with the streaming
// operator when the frame is cumulative. The function MUST also keep its
// standard update/combine/finalize path for the fallback case.
StreamingPartitioned  bool
// HasFinalize signals a TableInOut function whose Finalize method emits
// meaningful batches. Defaults to false; set to true only for functions
// that accumulate during Process and flush at end-of-stream. DuckDB
// skips the FINALIZE phase RPC entirely when this is false — required
// for LATERAL compatibility (avoids "FinalExecute not supported for
// project_input").
HasFinalize  bool
// OrderPreservation declares how a table function's output rows relate
// to its inputs. Empty leaves the field null and uses the C++ extension
// default. Maps to DuckDB's OrderPreservationType.
OrderPreservation  OrderPreservation
// OrderDependent declares whether the aggregate result depends on the
// row order. Empty defaults to NOT_ORDER_DEPENDENT.
OrderDependent  OrderDependence
// DistinctDependent declares whether DISTINCT changes the result.
// Empty defaults to NOT_DISTINCT_DEPENDENT.
DistinctDependent  DistinctDependence
// AutoApplyFilters indicates the framework should auto-apply pushdown filters.
AutoApplyFilters  bool
// Categories is a list of classification tags for the function.
Categories  []string
// Tags are arbitrary key/value annotations surfaced through the catalog as
// duckdb_functions().tags. Distinct from Categories (a flat list); Tags is
// a map for structured metadata (e.g. {"category": "debug"}).
Tags  map[string]string
// Examples lists usage examples surfaced in the catalog's FunctionInfo.
// Each example carries SQL, a description, and an optional expected output.
Examples  []CatalogExample
// ReturnType is the static return type for scalar functions.
// When set, the catalog registers this concrete type instead of ANY.
// Leave nil for functions with dynamic return types (resolved at bind time).
ReturnType  arrow.DataType
// RequiredSecrets lists secret types the function needs at bind time.
RequiredSecrets  []SecretRequirement
// SupportsBatchIndex opts a table function into per-batch vgi_batch_index
// tagging (see EmitBatchIndex). The C++ extension enforces monotonicity.
SupportsBatchIndex  bool
// PartitionKind declares the partition shape of a table function's output
// (see PartitionField / EmitPartitioned). Empty = NOT_PARTITIONED.
PartitionKind  PartitionKind
// SourceOrderDependent / SinkOrderDependent / RequiresInputBatchIndex are
// table-buffering ordering hints (mirror the FunctionInfo fields).
SourceOrderDependent  bool
SinkOrderDependent  bool
RequiresInputBatchIndex  bool
// InputFromArgs marks a blended ("UNNEST-style") table-in-out function: the
// function's positional args ARE its per-row input columns (real typed
// args, no synthetic TABLE placeholder), so ONE registration serves
// f(52, 13) (literal -> 1 input row), FROM t, f(t.x, t.y) (columns ->
// streaming), and LATERAL f(t.x, t.y). A blended function is map-shaped and
// per-row: it must NOT set HasFinalize (DuckDB forbids FinalExecute under
// correlated LATERAL, one of the call shapes blended must serve), must not
// declare a "table"-typed arg, and must not take a positional const arg.
// The worker reads the positional args from the input batch (by declared
// name for fixed args, positionally for varargs); named args stay bind-time
// scalars on ProcessParams.Args. Mirrors vgi-python's RowTransformFunction.
InputFromArgs  bool
// CacheControl opts the function into the extension's result cache: when
// set, its vgi.cache.* metadata is attached by the framework to every
// output batch of a SCALAR function and to every finalize batch of a
// TABLE-BUFFERING function (the exchange-mode buffered result cache).
// Per-VALUE memoization of a scalar is a SEPARATE opt-in on top of this —
// set CacheControl.PerValue, and only when one call is more expensive than
// a cache probe plus a decode (see that field). A pure, deterministic function
// only — advertising this on a non-pure function serves stale rows.
// Streaming table(-in-out) functions attach cache control per-emit instead
// (vgi.WithCacheControl). Mirrors vgi-python's ScalarFunction.CACHE_CONTROL
// and the buffered finalize cache-control support.
CacheControl  *CacheControl
}

Description

FunctionMetadata holds descriptive metadata about a function.

Methods

function DefaultMetadata

source

func DefaultMetadata() FunctionMetadata

DefaultMetadata returns metadata with default values.

function writableMetadata

source

func writableMetadata(desc string) FunctionMetadata

type FunctionStability

source

type FunctionStability string

Description

FunctionStability describes when function results may change.

type FunctionType

source

type FunctionType string

Description

FunctionType identifies the kind of VGI function.

Methods

function normalizeFunctionType

source

func normalizeFunctionType(ft FunctionType) FunctionType

normalizeFunctionType converts DuckDB function type strings to our canonical values.

type NullHandling

source

type NullHandling string

Description

NullHandling describes how the function handles NULL inputs. These values are DuckDB wire-protocol constants and must not be changed.

type OrderByDirection

source

type OrderByDirection string

Description

OrderByDirection is the sort direction carried by an ORDER BY pushdown hint. Wire-protocol dictionary constants — must not be changed.

type OrderByNullOrder

source

type OrderByNullOrder string

Description

OrderByNullOrder is the NULL placement carried by an ORDER BY pushdown hint. Wire-protocol dictionary constants — must not be changed.

type OrderDependence

source

type OrderDependence string

Description

OrderDependence declares whether an aggregate’s result depends on row order. Wire-protocol dictionary constants — must not be changed.

type OrderPreservation

source

type OrderPreservation string

Description

OrderPreservation declares how a table function’s output rows relate to its inputs. These values are DuckDB wire-protocol dictionary constants and must not be changed. The empty value leaves the field null (C++ extension default).

type PartitionKind

source

type PartitionKind string

Description

PartitionKind describes the partition shape a table function declares over its vgi.partition_column-annotated bind-schema fields. These values are DuckDB wire-protocol dictionary constants and must not be changed.

type Phase

source

type Phase string

Description

Phase identifies the table-in-out init phase. Wire-protocol dictionary constants — must not be changed.

struct ScalarExchangeState

source

type ScalarExchangeState struct {
Recipe  InitRecipe  // exported, serialized
fn  ScalarFunction  // transient
params  *ProcessParams  // transient

// cacheMetaResolved memoizes the result-cache annotation for this stream.
// Metadata()/CacheControl.Metadata() are invariant across batches, so we
// resolve them once on the first Exchange and reuse the map (nil for a
// non-cacheable scalar). Transient: recomputed after rehydration.
cacheMetaResolved  bool      // transient
cacheMeta    map[string]string  // transient
}

Description

ScalarExchangeState implements ExchangeState for scalar functions.

Methods

method Exchange

source

func (s *ScalarExchangeState) Exchange(ctx context.Context, input arrow.RecordBatch, out *vgirpc.OutputCollector, callCtx *vgirpc.CallContext) error

Exchange processes one input batch through the scalar function and emits the resulting batch to the output collector.

struct SecretLookup

source

type SecretLookup struct {
SecretType  string
SecretName  string
Scope    string
}

Description

SecretLookup describes a scoped secret lookup request for two-phase bind.

struct SecretRequirement

source

type SecretRequirement struct {
SecretType  string
SecretName  string  // empty = not specified
Scope    string  // empty = not specified
}

Description

SecretRequirement describes a secret type that a function needs.

type Secrets

source

type Secrets map[string]map[string]interface{}

Description

Secrets holds the resolved secrets passed to a worker, keyed by each secret’s unique DuckDB secret name (not by type), so several secrets of the same type (e.g. one per S3 bucket) coexist. Each secret is a map of its fields, including the connector-serialized “type” (the DuckDB secret type) and “scope” (newline-joined scope prefixes), plus type-specific fields like “key_id”.

Secrets is a plain map, so direct access (s[name][field]) still works; the methods below add type- and scope-aware selection.

Methods

method Field

source

func (s Secrets) Field(field string) (string, bool)

Field returns the first secret (of any name) carrying field, rendered to a string; ok is false when no secret carries it.

method FieldForScope

source

func (s Secrets) FieldForScope(path, field string) (string, bool)

FieldForScope returns a field of the best scope-matching secret for path.

method ForScope

source

func (s Secrets) ForScope(path string) (map[string]interface{}, bool)

ForScope returns the fields of the resolved secret whose “scope” is the longest prefix of path. Use this when the worker requested secrets for several scopes (e.g. one per cloud path / bucket) and must pick the right one per path. The connector serializes each secret’s scope as a newline-joined list of prefixes; a secret with no (or empty) scope matches as a last-resort fallback. ok is false only when there are no candidate secrets.

method ForScopeOfType

source

func (s Secrets) ForScopeOfType(path, secretType string) (map[string]interface{}, bool)

ForScopeOfType is like ForScope but only over secrets of secretType — the precise selector for cloud paths (e.g. the s3 secret matching a given s3://… URL when several buckets are in play).

method NamedField

source

func (s Secrets) NamedField(name, field string) (string, bool)

NamedField returns a field of the named secret, rendered to a string.

method OfType

source

func (s Secrets) OfType(secretType string) []map[string]interface{}

OfType returns every resolved secret whose serialized “type” field matches secretType (since secrets are keyed by name, not type).

method SecretType

source

func (s Secrets) SecretType(name string) (string, bool)

SecretType returns the DuckDB secret type of the named secret (its serialized “type” field).

method selectForScope

source

func (s Secrets) selectForScope(path, secretType string) (map[string]interface{}, bool)

struct TableInOutExchangeState

source

type TableInOutExchangeState struct {
Recipe    InitRecipe    // exported, serialized
UserStateBytes  []byte      // exported, gob-serialized user state
fn    TableInOutFunction  // transient
params    *ProcessParams    // transient
state    interface{}    // transient
autoApply  *PushdownFilters  // transient
}

Description

TableInOutExchangeState implements ExchangeState for table-in-out INPUT phase.

Methods

method Exchange

source

func (s *TableInOutExchangeState) Exchange(ctx context.Context, input arrow.RecordBatch, out *vgirpc.OutputCollector, callCtx *vgirpc.CallContext) error

Exchange transforms one input batch through the table-in-out function and emits the resulting batches to the output collector.

method GobDecode

source

func (s *TableInOutExchangeState) GobDecode(data []byte) error

GobDecode restores the exported wire fields; transient fields are rebuilt by rehydrateTableInOut.

method GobEncode

source

func (s *TableInOutExchangeState) GobEncode() ([]byte, error)

GobEncode snapshots the live user state into UserStateBytes, then encodes the wire form. Each table-in-out exchange tick re-serializes the token, so the INPUT-phase state must reflect the latest Process mutation.

struct TableProducerState

source

type TableProducerState struct {
Recipe    InitRecipe    // exported, serialized
UserStateBytes  []byte      // exported, gob-serialized user state
AutoProjectIDs  []int32      // exported
fn    TableFunction    // transient
params    *ProcessParams    // transient
state    interface{}    // transient (reconstructed from UserStateBytes)
autoApply  *PushdownFilters  // transient
}

Description

TableProducerState implements ProducerState for table functions.

Methods

method GobDecode

source

func (s *TableProducerState) GobDecode(data []byte) error

GobDecode restores the exported wire fields; transient fields are rebuilt by rehydrateTableProducer.

method GobEncode

source

func (s *TableProducerState) GobEncode() ([]byte, error)

GobEncode snapshots the live user state into UserStateBytes, then encodes the wire form. See the file-level comment for why this is required.

method Produce

source

func (s *TableProducerState) Produce(ctx context.Context, out *vgirpc.OutputCollector, callCtx *vgirpc.CallContext) error

Produce advances the table function by one tick, applying any dynamic filter update carried on the tick’s metadata and emitting produced batches to the output collector.

func type TypeBoundPredicate

source

type TypeBoundPredicate func(arrow.DataType) bool

Description

TypeBoundPredicate is a function that validates whether an Arrow DataType is acceptable for a given argument. Used with ArgSpec.TypeBound to constrain “any”-typed arguments to specific type families (e.g., numeric types only).

Methods

function LookupTypeBound

source

func LookupTypeBound(name string) TypeBoundPredicate

LookupTypeBound returns the predicate registered under name, or nil.

function resolveBounds

source

func resolveBounds(spec string) ([]TypeBoundPredicate, error)

type WriteOp

source

type WriteOp string

Description

WriteOp identifies which DML operation a writable-table function lookup is for. Wire-protocol dictionary constants — must not be changed.

function RenderSecretValue

source

func RenderSecretValue(v interface{}) string

RenderSecretValue renders a secret field value to a string.

function applyTickFilters

source

func applyTickFilters(params *ProcessParams, meta arrow.Metadata)

applyTickFilters checks the tick-level custom metadata for a dynamic filter update and, if present, replaces params.CurrentPushdownFilters with the freshly decoded filter state. Silent on decode errors — the previous filter state is retained (DuckDB falls back to client-side filtering).

function applyTickValidators

source

func applyTickValidators(params *ProcessParams, meta arrow.Metadata)

applyTickValidators lifts the conditional-revalidation validators off a tick’s custom metadata onto ProcessParams. The client sends them when it holds a stale-but-revalidatable cached result and wants the worker to confirm freshness without recomputing. They arrive once — on the first tick (subprocess) or on the init request (HTTP) — and stay set for the stream.

function attachAAD

source

func attachAAD(auth *vgirpc.AuthContext) []byte

attachAAD is the AAD for an attach_opaque_data envelope.

function identityTail

source

func identityTail(auth *vgirpc.AuthContext) []byte

identityTail builds the identity portion of an opaque-data AAD. Mirrors the (domain, principal) convention: unauthenticated requests get a fixed anonymous tail, so an anonymous caller cannot open an envelope sealed for a real principal.

function normalizeCryptoKey

source

func normalizeCryptoKey(key []byte) []byte

normalizeCryptoKey stretches/compresses an arbitrary-length key to the 32 bytes XChaCha20-Poly1305 requires. Matches vgi-python’s normalize_key.

function openBytes

source

func openBytes(token, key, aad []byte, version byte) ([]byte, error)

openBytes opens and verifies an envelope produced by sealBytes. Every failure mode — malformed, wrong version, tampered, wrong key, wrong AAD (cross-principal/cross-attach replay) — returns errOpaqueDataRejected.

function projectBatch

source

func projectBatch(batch arrow.RecordBatch, ids []int32) arrow.RecordBatch

projectBatch selects only the columns at the given indices from a RecordBatch.

function sealBytes

source

func sealBytes(payload, key, aad []byte, version byte) ([]byte, error)

sealBytes seals payload into an AEAD envelope: version || nonce || ct+tag.

function selectColumnsByName

source

func selectColumnsByName(src arrow.RecordBatch, schema *arrow.Schema) (arrow.RecordBatch, error)

selectColumnsByName builds a batch containing only the columns named in schema, in schema order, taken from src by field name.

function transactionAAD

source

func transactionAAD(auth *vgirpc.AuthContext, attachEnvelope []byte) []byte

transactionAAD is the AAD for a transaction_opaque_data envelope. It binds both the caller identity and the parent attach envelope, so a transaction value minted under one attach cannot be replayed against a different attach even by the same principal.

function unaryCatalog

source

func unaryCatalog[P any, R any](w *Worker, s *vgirpc.Server, name string,
handler func(context.Context, *vgirpc.CallContext, P) (R, error))

unaryCatalog registers a catalog unary handler whose request opaque-data fields are unwrapped before the handler body runs.

function unaryVoidCatalog

source

func unaryVoidCatalog[P any](w *Worker, s *vgirpc.Server, name string,
handler func(context.Context, *vgirpc.CallContext, P) error)

unaryVoidCatalog is unaryCatalog for void-returning catalog handlers.