Table-in-out functions

On this page

Streaming a relation through, batch by batch.

class PassthroughTIOFunction

source

public abstract class PassthroughTIOFunction implements TableInOutFunction

Description

Base for table-in-out functions whose output schema equals their input schema (echo, filter, repeat, exception-finalize, etc.). Mirrors the vgi-python TableInOutGenerator default-bind shape.

Subclasses inherit a default #onBind that returns the input schema (with an empty-schema fallback for catalog enumeration). They still provide name(), metadata(), argumentSpecs(), and createExchange() as usual.

Members

BindResponse onBind(TableInOutBindParams params)

Returns the input schema as the output schema, falling back to an empty schema when no input is present (catalog enumeration).

interface RowTransformFunction

source

public interface RowTransformFunction extends TableInOutFunction

Description

Blended (“UNNEST-style”) table-in-out: positional args ARE per-row input columns.

A RowTransformFunction collapses the classic either/or between a standard table function (literal args only) and a table-in-out function (an explicit TABLE subquery arg). Its positional farm.query.vgi.function.ArgSpecs declare its per-row input columns — real typed args, NO synthetic TABLE placeholder — so ONE registration serves every call shape:

f(52, 13)                       -- literal   -> one input row
  FROM t, f(t.x, t.y)             -- columns   -> streaming input
  SELECT ... FROM t, LATERAL f(t.x, t.y)

Contract.

• Positional args are the input columns; they arrive on the exchange’s input batch (by declared name for fixed args, col0..colN-1 for varargs — read them positionally). They are NOT on the wire arguments.

• Named args stay bind-time scalars on params.arguments().named().

• Map-shaped, per-row: 1->1, 1->N, 1->0 all work. There is no finalize — #hasFinalize() is final-false here (DuckDB forbids FinalExecute under correlated LATERAL, one of the call shapes blended must serve). Accumulating functions use a classic TABLE-input table-in-out or a farm.query.vgi.buffering.TableBufferingFunction.

• A positional const arg must not be declared (in the column form DuckDB sweeps a constant into the input subquery; in the literal form it is indistinguishable from an input column). Use a named arg for optional config.

Implementing this interface (not a metadata flag) IS the blended signal — a per-arg or metadata flag could be forgotten on one of N same-named overloads; the type cannot. The wire FunctionInfo.input_from_args is derived from it, and the C++ extension reads that to enter the in-out registration branch with real-typed args and drive the literal single-row scan-mode. Mirrors vgi-python’s RowTransformFunction.

record TableInOutBindParams

source

public record TableInOutBindParams( String functionName, Arguments arguments, Schema inputSchema, Map<String, Object> settings, byte[] secrets, boolean resolvedSecretsProvided, byte[] attachOpaqueData, farm.query.vgi.storage.BoundStorage attachStorage, farm.query.vgi.protocol.CopyToContext copyTo)

Description

Bind-time inputs for a TableInOutFunction: the resolved arguments and the schema of the incoming stream, from which the function derives its output schema.

Members

TableInOutBindParams(String functionName, Arguments arguments, Schema inputSchema, Map<String, Object> settings)

Convenience constructor with no attach context (catalog enumeration).

TableInOutBindParams(String functionName, Arguments arguments, Schema inputSchema, Map<String, Object> settings, byte[] secrets, boolean resolvedSecretsProvided, byte[] attachOpaqueData, farm.query.vgi.storage.BoundStorage attachStorage)

Convenience constructor without the copyTo context (non-COPY binds).

class TableInOutExchangeState

source

public abstract class TableInOutExchangeState extends ExchangeState

Description

Base for table-in-out exchange states: each input batch produces zero or one output batch via #process. Subclasses override #onInputBatch which receives the AnnotatedBatch input root and the OutputCollector to emit on.

Members

void exchange(AnnotatedBatch input, OutputCollector out, CallContext ctx)

Final dispatch hook from the RPC layer; forwards each input batch to #onInputBatch.

void onInputBatch(AnnotatedBatch input, OutputCollector out, CallContext ctx)

Processes a single input batch, emitting zero or one output batch.

interface TableInOutFunction

source

public interface TableInOutFunction extends FunctionDescriptor

Description

A VGI table-in-out function: receives input batches and emits output batches, one output batch per input batch (echo, filter, transform). State may accumulate across exchange ticks and round-trips with the stream, so it works on both the launcher and HTTP transports.

A streaming table-in-out may additionally declare a per-substream finalize (#hasFinalize() + #finish): under per-substream worker fan-out, DuckDB runs one substream per PipelineExecutor and issues a FINALIZE-phase init (same execution_id as the substream’s INPUT phase) after input EOS. The finalize sees only this substream’s accumulated state — coordinate it through params.storage() (execution-scoped), never a global cross-substream merge.

Functions that need a global Sink+Combine+Source shape — buffer the whole input across every substream, then emit a summary at the end (sums, distributed aggregation, full buffering) — use farm.query.vgi.buffering.TableBufferingFunction instead, whose worker-side storage (keyed by execution_id) survives the stateless HTTP process→combine→finalize round-trip.

Mirrors vgi.TableInOutFunction in vgi-go.

record TableInOutInitParams

source

public record TableInOutInitParams( String functionName, Arguments arguments, Schema inputSchema, Schema outputSchema, Map<String, Object> settings, BufferAllocator allocator, farm.query.vgi.storage.BoundStorage storage, byte[] secrets, byte[] substreamId)

Description

Per-execution inputs handed to TableInOutFunction#createExchange when a stream begins: the bound arguments, the negotiated input and (possibly projection-narrowed) output schemas, and the allocator the exchange must use.

Members

TableInOutInitParams( String functionName, Arguments arguments, Schema inputSchema, Schema outputSchema, Map<String, Object> settings, BufferAllocator allocator, farm.query.vgi.storage.BoundStorage storage, byte[] secrets)

Compatibility constructor without a substream id (serial path).