Table-in-out functions

On this page

Streaming a relation through, batch by batch.

function defineRowTransformFunction

source

export function defineRowTransformFunction<
TArgs = Record<string, any>,
>(config: RowTransformConfig<TArgs>): VgiFunction

Description

Define a blended (“UNNEST-style”) table-in-out function: its positional args ARE its per-row input columns, so ONE registration serves every call shape — f(52, 13) (literal -> one input row), FROM t, f(t.x, t.y) (columns -> streaming), and LATERAL f(t.x, t.y). Mirrors vgi-python’s RowTransformFunction (Phase B).

Registers as a TABLE function with FunctionInfo.input_from_args = true; the worker’s overload resolution matches blended overloads by INPUT-COLUMN count (the positional args are not on the wire). Map-shaped, no finalize.

function defineTableInOutFunction

source

export function defineTableInOutFunction<
TArgs = Record<string, any>,
TState = null,
>(config: TableInOutConfig<TArgs, TState>): VgiFunction

const PARENT_ROW_METADATA_KEY

source

const PARENT_ROW_METADATA_KEY = “vgi_rpc.parent_row#b64”

Description

Metadata key carrying per-output-row provenance: base64 of a raw little-endian int32[] mapping each output row to the input row that produced it. Shared by string with the C++ extension and vgi-python.

function parentRowsMetadata

source

export function parentRowsMetadata(
parentRows: number[],
outputRows: number,
extra?: Map<string, string>,
): Map<string, string>

Description

Fold per-output-row provenance into an emit metadata map.

Used by the batched correlated LATERAL operator (blended RowTransformFunction under FROM t, f(t.x) / LATERAL): the C++ extension ships a whole input chunk to the worker in ONE exchange and reads ONE output batch, then maps each output row back to the input row that produced it via this array — so a 1->N fan-out or 1->0 filter can be batched instead of driven row-by-row.

parentRows[i] is the 0-based index (into the input batch) of the row that produced output row i. Encoded as a raw little-endian int32 array (NOT Arrow IPC), base64-encoded, under vgi_rpc.parent_row#b64. Absent metadata means an identity 1->1 map (the common case: the extension assumes it, and requires output rows == input rows).

Contract: parentRows.length MUST equal the emitted batch’s row count (a mismatch is a worker bug that would corrupt the stamping). Values are range-checked against the input width on the C++ side. Mirrors vgi-python’s _merge_parent_rows / out.emit(..., parent_rows=[...]).

interface RowTransformConfig

source

export interface RowTransformConfig<TArgs = Record<string, any>>

Fields

namestring

onBind(params: TableInOutBindParams<TArgs>) => | { outputSchema: VgiSchema; opaqueData?: Uint8Array } | Promise<{ outputSchema: VgiSchema; opaqueData?: Uint8Array }>

Bind: return the output schema. The input schema (the declared per-row columns, typed by the C++ bind) is on params.bindCall.input_schema.

process( params: RowTransformProcessParams<TArgs>, batch: VgiBatch, out: OutputCollector, ) => void | Promise<void>

Per-row map: transform one input batch, emit exactly one output batch via out. 1->1, 1->N (with {@link parentRowsMetadata} provenance), and 1->0 (a 0-row emit) all work. There is NO finalize — a blended function is a per-row map (DuckDB forbids FinalExecute under correlated LATERAL, one of the call shapes blended must serve). Accumulating functions use a classic TableInput table-in-out or a TableBufferingFunction.

descriptionstringoptional

argsRecord<string, VgiDataType>optional

Positional args = the per-row INPUT COLUMNS (real typed args on the wire, no synthetic TABLE placeholder). Read from batch by declared name in process(); NOT surfaced on params.args.

varargs{ name: string; type: VgiDataType; doc?: string }optional

Trailing VARARGS input columns: the per-row input is N columns of the declared type. A varargs blended function has no per-column declared names (the C++ bind names them col0..colN-1), so process() reads the columns POSITIONALLY off batch.

namedArgsRecord<string, VgiDataType>optional

Named (string-position) args stay bind-time scalars on params.args.

argDefaultsRecord<string, any>optional

argDocsRecord<string, string>optional

Per-argument descriptions keyed by arg name (surfaced as vgi_doc).

projectionPushdownbooleanoptional

filterPushdownbooleanoptional

autoApplyFiltersbooleanoptional

stabilityFunctionStabilityoptional

examplesFunctionExample[]optional

categoriesstring[]optional

tagsRecord<string, string>optional

maxWorkersnumberoptional

requiredSettingsstring[]optional

requiredSecretsstring[]optional

type RowTransformProcessParams

source

export type RowTransformProcessParams<TArgs = Record<string, any>> =
TableInOutProcessParams<TArgs>;

Description

Process params for a blended row-transform function. args carries only the NAMED (bind-time scalar) options — the positional args are the per-row input columns, read from batch in process() (by declared name for fixed args, positionally for varargs).

interface TableInOutBindParams

source

export interface TableInOutBindParams<TArgs = Record<string, any>>

Fields

argsTArgs

bindCallBindRequest

settingsRecord<string, any>

secretsRecord<string, Record<string, any>>

interface TableInOutConfig

source

export interface TableInOutConfig<
TArgs = Record<string, any>,
TState = null,
>

Fields

namestring

descriptionstringoptional

argsRecord<string, VgiDataType>optional

namedArgsRecord<string, VgiDataType>optional

Named arguments (optional, DuckDB passes by name)

argDefaultsRecord<string, any>optional

Argument defaults

onBind(params: TableInOutBindParams<TArgs>) => | { outputSchema: VgiSchema; opaqueData?: Uint8Array } | Promise<{ outputSchema: VgiSchema; opaqueData?: Uint8Array }>optional

Bind: default passes through input schema. May be async.

onInit(params: { args: TArgs; initCall: InitRequest; outputSchema: VgiSchema; executionId: Uint8Array; }) => GlobalInitResponse | Promise<GlobalInitResponse>optional

initialState(params: TableInOutProcessParams<TArgs>) => TStateoptional

process( params: TableInOutProcessParams<TArgs>, state: TState, batch: VgiBatch, out: OutputCollector ) => void | Promise<void>optional

Process: transform input batch, emit output via out

finalize( params: TableInOutProcessParams<TArgs>, states: TState[] ) => VgiBatch[] | Promise<VgiBatch[]>optional

Finalize: emit final batches after all input processed. Receives all worker states collected from storage (matches Python’s finish(params, states)).

projectionPushdownbooleanoptional

filterPushdownbooleanoptional

autoApplyFiltersbooleanoptional

stabilityFunctionStabilityoptional

examplesFunctionExample[]optional

categoriesstring[]optional

tagsRecord<string, string>optional

maxWorkersnumberoptional

requiredSettingsstring[]optional

requiredSecretsstring[]optional

interface TableInOutProcessParams

source

export interface TableInOutProcessParams<TArgs = Record<string, any>>

Fields

argsTArgs

initCallInitRequest

initResponseGlobalInitResponse

outputSchemaVgiSchema

settingsRecord<string, any>

secretsRecord<string, Record<string, any>>

storageBoundStorage

Shared storage for cross-phase and cross-worker data (SQLite-backed).

pushdownFiltersPushdownFiltersoptional

substreamIdUint8Array | nulloptional

Stable client-minted id for this streaming table-in-out substream. Present (identical across init / every process() / finalize) when the client fanned this function out across per-substream workers; use it to key per-substream accumulated state in shared storage so a finalize() that lands on a different HTTP backend than the process() calls still finds it. null/undefined for the serial path or an old client. Mirrors vgi-python’s ProcessParams.substream_id.

ifNoneMatchstringoptional

Conditional-revalidation validator (exchange-mode result cache): the client holds a stale cached result for THIS input unit and asks the worker to confirm freshness cheaply. When set, process() may answer with a 0-row cacheControlMetadata({ notModified: true, ... }) batch instead of recomputing. Rides the input batch’s custom metadata (attached by the C++ WriteInputBatch). Undefined on a normal call.

ifModifiedSincestringoptional

RFC 3339 Last-Modified validator for conditional revalidation. Companion to {@link ifNoneMatch}. Undefined on a normal call.