Buffering functions

On this page

Sink, combine, and source phases with durable cross-process state.

class `CopyToFunction`

public abstract class CopyToFunction : ITableBufferingFunction

Description

Base class for a COPY … TO (FORMAT '<name>', …) writer — mechanically a table-buffering (Sink+Combine) function with NO Source phase: Process writes one input batch, Combine runs exactly once (on the coordinator, after every batch has been written) to close the destination, and the (never actually drained — COPY TO produces no rows) finalize producer just finishes immediately. Register via Worker.RegisterCopyToFormat, which also advertises it through catalog_copy_from_formats (the RPC that covers both directions) — including this class's SinkOrderDependent as the format's ordered flag. Mirrors vgi-python/ vgi-java's CopyToFunction . Cross-process invariant: Write (Sink) and Close (Combine, exactly once) may run on DIFFERENT worker processes (pool rotation, or pool false ) — any state one needs to hand the other MUST go through TableBufferingProcessParams.Storage/ TableBufferingCombineParams.Storage, never an in-memory field on this instance.

Public members

public IReadOnlyList<byte[]> Combine(IReadOnlyList<byte[]> stateIds, TableBufferingCombineParams combineParams) ;

public ITableFunctionProducer CreateFinalizeProducer(byte[] finalizeStateId, TableBufferingFinalizeParams finalizeParams) ;

public Schema OutputSchema { get; }

COPY TO produces no output rows.

public Schema ResolveOutputSchema(TableInOutBindParams bindParams) ;

public abstract Schema ArgumentsSchema { get; }

The format's OPTIONS — every field a named argument; never a positional or TABLE-typed argument (the input TABLE argument this Sink+Source shape implies is the COPY source itself, wired automatically — Worker.RegisterCopyToFormat never adds an explicit TableArgFields.Table field for it).

public abstract string Description { get; }

public abstract string Name { get; }

public byte[] Process(RecordBatch batch, TableBufferingProcessParams processParams) ;

public virtual bool SinkOrderDependent;

true forces a single-threaded, source-ordered sink — override when row order in the destination must match source order (the C++ operator otherwise shards the sink across threads/processes, arbitrary interleaving).

public virtual string SchemaName;

public void Bind(TableInOutBindParams bindParams) ;

interface `IFunctionStorage`

source

public interface IFunctionStorage

Description

Durable, cross-PROCESS storage scoped to one query execution — the coordination primitive ITableBufferingFunction.Process/ITableBufferingFunction.Combine/the FINALIZE producer use to hand state to each other. Necessary because, unlike a table-in-out function's substream (which stays on ONE connection end to end), table-buffering's table_buffering_process / table_buffering_combine calls are each independently worker-pool-acquired unary RPCs — under the stdio/subprocess transport that can mean a SEPARATE OS PROCESS per call (see Internal.CrossProcessWorkQueue 's doc comment for the same discovery in the plain-table-function context), so in-memory state on one call is invisible to the next. The concrete implementation ( Internal.FunctionStorage ) backs this with files under the OS temp directory, keyed by the query's execution_id — durable and visible across process boundaries without any extra runtime dependency (matches vgi-python's BoundStorage role, minus the SQLite backend).

interface `ITableBufferingFunction`

source

public interface ITableBufferingFunction

Description

The raw contract a table-buffering (Sink+Source) function implements — a table-in-out-shaped call site ( f(data TABLE, …) ) that, unlike ITableInOutFunction, must see EVERY input row across EVERY substream before producing any output. Ported from vgi-python's TableBufferingFunction /vgi-java's TableBufferingFunction . Three phases, matching the C++ PhysicalVgiTableBuffering Sink+Source operator: Sink — Process, once per input batch (parallel across threads/processes unless SinkOrderDependent) — stash the batch and return an opaque state_id . Combine — Combine, once, after every Sink call completes — group/merge the collected state_id s into finalize_state_id s, one per Source output stream. Source — CreateFinalizeProducer, once per finalize_state_id — builds an ITableFunctionProducer the framework ticks until it finishes. This is what makes a GLOBALLY correct "sum every row" (or sort/dedupe/etc.) possible: unlike ITableInOutFunction's per-substream FINALIZE (which only ever sees its own substream's share of the input — wrong for a query-wide aggregate), Combine sees every Sink call's result before Source ever runs. Cross-process invariant: Process/Combine are each independently worker-pool-acquired unary RPCs (see TableBufferingProcessParams's doc comment) — any state one call needs to hand to a later call/phase MUST go through TableBufferingProcessParams.Storage/TableBufferingCombineParams.Storage, never an in-memory field on the function instance (which is only ever visible within the ONE worker process that happens to run a given call).

class `TableBufferingCombineParams`

source

public sealed class TableBufferingCombineParams

Description

Parameters an ITableBufferingFunction sees on its single Combine-phase call. See TableBufferingProcessParams's doc comment for why Arguments/Settings come from IFunctionStorage.

Public members

public Apache.Arrow.Schema? InputSchema { get; init; }

The bind-time TABLE argument's column schema (recovered, like Arguments, from the persisted bind context) — useful for a function that needs the source's column names/types at Combine time even when zero batches were ever Process ed (e.g. writing a header-only file for a genuinely empty COPY TO source).

public ICallContext? Ctx { get; init; }

In-band log sink — see TableBufferingProcessParams.Ctx.

public Protocol.CopyToContext? CopyTo { get; init; }

Non-null only for a COPY … TO sink — see TableBufferingProcessParams.CopyTo's doc comment.

public byte[]? AttachOpaqueData { get; init; }

public byte[]? Secrets { get; init; }

Already-resolved secrets — see TableBufferingProcessParams.Secrets's doc comment.

public byte[]? Settings { get; init; }

public byte[]? TransactionId { get; init; }

public required IFunctionStorage Storage { get; init; }

public required TableArguments Arguments { get; init; }

public required byte[] ExecutionId { get; init; }

public required string FunctionName { get; init; }

class `TableBufferingFinalizeParams`

source

public sealed class TableBufferingFinalizeParams

Description

Parameters an ITableBufferingFunction sees when building the FINALIZE (Source-phase) producer for one FinalizeStateId — mirrors init(phase=TABLE_BUFFERING_FINALIZE) . Unlike Process/Combine, this call DOES ride the full bind_call on the wire (the shared init RPC's InitRequest.BindCall ), so Arguments/Settings are decoded directly rather than recovered from storage — but reading them back from Storage works too if that's more convenient.

Public members

public IReadOnlyList<byte[]>? JoinKeys { get; init; }

One embedded-IPC single-column batch per IN-filter/join-key column ( InitRequest.JoinKeys ) — mirrors Table.TableInitParams.JoinKeys.

public IReadOnlyList<long>? ProjectionIds { get; init; }

public Schema ProjectedSchema;

Convenience: OutputSchema narrowed to ProjectionIds (or the full schema when ProjectionIds is null) — the schema a projection-pushdown-aware FINALIZE producer should actually emit. Mirrors Table.TableInitParams.ProjectedSchema.

public byte[]? AttachOpaqueData { get; init; }

Raw BindRequest.AttachOpaqueData — see Table.TableBindParams.AttachOpaqueData's doc comment. Needed by a function whose durable state is scoped to the ATTACH session rather than (or in addition to) this execution id — e.g. a persistent, cross-call collection keyed by attach identity.

public byte[]? PushdownFilters { get; init; }

Raw embedded-IPC pushdown-filter bytes ( InitRequest.PushdownFilters ) — null when DuckDB pushed no filters down. Only meaningful when this function advertised ITableBufferingFunction.FilterPushdown. Decode with Internal.PushdownFilterCodec. Mirrors Table.TableInitParams.PushdownFilters.

public byte[]? Settings { get; init; }

public required IFunctionStorage Storage { get; init; }

public required Schema OutputSchema { get; init; }

public required TableArguments Arguments { get; init; }

public required byte[] ExecutionId { get; init; }

public required byte[] FinalizeStateId { get; init; }

One of the ids ITableBufferingFunction.Combine returned — names which output stream this producer drains.

public required string FunctionName { get; init; }

class `TableBufferingProcessParams`

source

public sealed class TableBufferingProcessParams

Description

Parameters an ITableBufferingFunction sees on each Sink-phase ITableBufferingFunction.Process call. NOTE: unlike TableInOut.TableInOutInitParams, Arguments/Settings here are recovered from IFunctionStorage (stashed at init(phase=TABLE_BUFFERING) time) rather than riding this call's own wire request — table_buffering_process 's wire shape carries neither, since it's a standalone unary RPC that may land on a worker process that never itself ran this query's bind.

Public members

public ICallContext? Ctx { get; init; }

In-band log sink (surfaces as a duckdb_logs() row with type='VGI' ) — null only in a unit test that constructs this params object directly.

public Protocol.CopyToContext? CopyTo { get; init; }

Non-null only for a COPY … TO sink — see TableInOut.TableInOutBindParams.CopyTo's doc comment. Recovered (like Arguments/Settings) from the persisted bind context, since this call's own wire request carries neither.

public byte[]? AttachOpaqueData { get; init; }

public byte[]? Secrets { get; init; }

Already-RESOLVED secrets (any two-phase dynamic-scope retry completed back at bind time, before this Sink phase ever started) — decode with SecretArgCodec.Decode then SecretArgCodec.FindByType/ SecretArgCodec.ForScopeOfType. null when none were resolved. Recovered from the persisted bind context, like Arguments/Settings.

public byte[]? Settings { get; init; }

public byte[]? TransactionId { get; init; }

public long? BatchIndex { get; init; }

DuckDB's globally-unique batch index — populated only when this function advertised ITableBufferingFunction.RequiresInputBatchIndex.

public required IFunctionStorage Storage { get; init; }

Cross-process durable storage scoped to this execution — see IFunctionStorage's doc comment. The canonical pattern: Append this batch's data (or a derived summary) under a namespace/key of your choosing, then return a state_id that ITableBufferingFunction.Combine and/or the FINALIZE producer knows how to read back.

public required TableArguments Arguments { get; init; }

public required byte[] ExecutionId { get; init; }

public required string FunctionName { get; init; }