Table-in-out functions

On this page

Streaming relation transforms that process Arrow record batches.

interface ITableInOutFunction

source

public interface ITableInOutFunction

Description

The raw contract a streaming table-in-out function implements — the VGI analog of Table.ITableFunction for a function that takes a TABLE argument and transforms it batch-by-batch (ExchangeState-shaped: client sends an input batch, this replies with exactly one output turn per input turn), optionally followed by a per-substream FINALIZE phase (ProducerState-shaped) once the input stream ends. Ported from vgi-python's TableInOutGenerator /vgi-java's raw TableInOutExchangeState pattern. IMPORTANT — this is the per-SUBSTREAM shape: DuckDB fans a streaming table-in-out call out to one worker PROCESS per substream by default (see vgi_table_in_out_impl.cpp 's "Phase A" comments), so ITableInOutProcessor instances never see input from more than one substream, and a FINALIZE phase (if any) runs on that SAME substream's own accumulated state — it is NOT a global cross-substream aggregation. A function that needs a single correct answer over the WHOLE input regardless of how DuckDB partitions it (e.g. a plain "sum every row") belongs to Buffering.ITableBufferingFunction instead, which is purpose-built for that (see its own doc comment for why a per-substream finalize is unsound for that case).

interface ITableInOutProcessor

source

public interface ITableInOutProcessor

Description

The per-substream cursor an ITableInOutFunction.CreateProcessor returns — driven once per input batch during the INPUT phase (Process, exchange-shaped: one output batch per input batch turn) and, if the function has a finalize step (ITableInOutFunction.HasFinalize), once per client "tick" during the FINALIZE phase that follows on the SAME connection (Finalize, producer-shaped — mirrors Table.ITableFunctionProducer.Produce).

class TableInOutBindParams

source

public sealed class TableInOutBindParams

Description

Parameters an ITableInOutFunction/Buffering.ITableBufferingFunction sees at bind time — the table-in-out analog of TableBindParams, with one addition: InputSchema is always populated (the TABLE argument's per-call column schema), since a table-in-out/table-buffering function without a TABLE input is meaningless.

Public members

public Protocol.CopyToContext? CopyTo { get; init; }

Non-null only when this bind opened a COPY … TO (FORMAT '<this function's name>', …) — the destination path (no expected-schema field; the source columns ride InputSchema instead). See Buffering.CopyToFunction.

public byte[] ArgumentsBytes { get; init; }

Opaque, not-yet-decoded serialized argument bytes for the NON-table positional/named arguments — decode with TableArgCodec.Decode (or use Arguments). The TABLE argument itself never appears here (the C++ side omits it entirely when building this struct — see Table.TableArgFields.Table's doc comment).

public byte[] AttachOpaqueData { get; init; }

Raw BindRequest.AttachOpaqueData — see Table.TableBindParams.AttachOpaqueData's doc comment.

public byte[]? Settings { get; init; }

public required Internal.SecretsAccessor Secrets { get; init; }

Secrets access for this bind attempt — see Table.TableBindParams.Secrets's doc comment (same static-vs-dynamic split, same two-phase retry mechanism).

public required Schema InputSchema { get; init; }

The TABLE argument's per-call column schema (its actual runtime column names/types — dynamic, since callers can pass any query as the TABLE argument).

public required TableArguments Arguments { get; init; }

public required string FunctionName { get; init; }

class TableInOutInitParams

source

public sealed class TableInOutInitParams

Description

Parameters an ITableInOutFunction sees when its per-substream processor is created (mirrors the init(phase=INPUT) RPC) — fires once per substream, before any input batches arrive.

Public members

public IReadOnlyList<long>? ProjectionIds { get; init; }

Zero-based indices (into OutputSchema) of the columns DuckDB actually needs for THIS call, or null for "all columns" — the table-in-out analog of Table.TableInitParams.ProjectionIds. Only meaningful when this function advertised ITableInOutFunction.ProjectionPushdown; unlike a plain table function's producer (where an unadvertised function can still emit full-width batches and let DuckDB trim them client-side), the batched correlated-LATERAL operator VALIDATES the wire schema strictly against whatever it negotiated — so a ITableInOutFunction.ProjectionPushdown-advertising processor MUST emit exactly ProjectedSchema, not OutputSchema, whenever this is non-null.

public Schema ProjectedSchema;

Convenience: OutputSchema narrowed to ProjectionIds (or the full schema when ProjectionIds is null) — the schema a projection-pushdown-aware processor should actually emit.

public byte[] AttachOpaqueData { get; init; }

Raw BindRequest.AttachOpaqueData — see Table.TableBindParams.AttachOpaqueData's doc comment.

public byte[]? ExecutionId { get; init; }

Stable across the INPUT and (if this function has one) FINALIZE phase of ONE substream — mint/reuse a per-substream accumulator keyed by this if needed, though for the common case (finalize runs on the very same connection right after INPUT's EOS) ordinary mutable state captured by the ITableInOutProcessor instance itself is simpler and sufficient; this is provided for parity with the wire protocol, not because most functions need it.

public byte[]? Secrets { get; init; }

Fully-RESOLVED secrets from the bind call that opened this processor — see Table.TableInitParams.Secrets's doc comment.

public byte[]? Settings { get; init; }

public byte[]? SubstreamId { get; init; }

public required Schema InputSchema { get; init; }

public required Schema OutputSchema { get; init; }

The resolved per-call output schema (from ITableInOutFunction.ResolveOutputSchema) — the function's FULL declared output shape, regardless of what this call actually requested. A processor that advertises ITableInOutFunction.ProjectionPushdown must emit ProjectedSchema instead (the wire itself is declared with the narrowed schema — see ProjectionIds's doc comment); every other processor emits this FULL schema unconditionally.

public required TableArguments Arguments { get; init; }

public required string FunctionName { get; init; }