Table functions

On this page

Set-returning producers, bind/init parameters, and table argument declarations.

class CopyFromFunction

source

public abstract class CopyFromFunction : ITableFunction

Description

Base class for a COPY … FROM (FORMAT '<name>', …) reader — an ordinary producer function whose output schema is dictated ENTIRELY by the COPY target (DuckDB inserts no cast for COPY FROM, so Read must emit rows matching Protocol.CopyFromContext.ExpectedSchema exactly: column-for-column, name AND type, no more no fewer). Register via Worker.RegisterCopyFromFormat, which also advertises it through catalog_copy_from_formats . Mirrors vgi-python/vgi-java's CopyFromFunction .

Public members

public ITableFunctionProducer CreateProducer(TableInitParams initParams) ;

public Schema OutputSchema { get; }

Never actually used — ResolveOutputSchema always overrides it with the COPY target's own required schema.

public Schema ResolveOutputSchema(TableBindParams bindParams) ;

public abstract Schema ArgumentsSchema { get; }

The format's OPTIONS — every field a named (optional or required) argument; never a positional or TABLE-typed argument (the row data is the COPY target itself, not a call argument).

public abstract string Description { get; }

public abstract string Name { get; }

public virtual string SchemaName;

public void Bind(TableBindParams bindParams) ;

interface ITableFunction

source

public interface ITableFunction

Description

The raw contract a table ("producer") function implements — the VGI analog of Scalar.IScalarFunction for the ProducerState stream kind (client sends empty "tick" batches; the server emits data batches until it calls OutputCollector.Finish() ). Ported from vgi-java's TableFunction /vgi-python's TableFunction , adapted to C#'s immutable-array model. A table function's SQL arguments are ALWAYS bind-time constants — there is no per-row input the way a scalar function has one (that's what makes table-in-out/ ITableInOutFunction , M4, a different interface) — so TableBindParams.Arguments/TableInitParams.Arguments decode with Internal.TableArgCodec, not Internal.ScalarArgCodec.

interface ITableFunctionProducer

source

public interface ITableFunctionProducer

Description

The per-call cursor an ITableFunction.CreateProducer returns — driven once per client "tick" by Internal.TableProducerStreamState (the ProducerState dispatch glue). Mirrors vgi-java's TableProducerState.produceTick(OutputCollector, …) .

class PlanRequest

source

public sealed class PlanRequest

Description

The inputs an ITableFunction.Plan call receives beyond its bind parameters: the pushdown it may use to emit fewer splits, and the place in the enumeration it is resuming from. Only invoked when ITableFunction.SupportsSplits is true.

Public members

public IReadOnlyList<long>? ProjectionIds { get; init; }

Columns the scan actually reads, or null for all.

public bool FiltersComplete { get; init; }

false means the client may still narrow the filter set further on a later continuation call; true (the common case) means what this call carries is final.

public byte[]? Cursor { get; init; }

A place in the ENUMERATION of splits — NOT a place in the data. Empty on the first call for a scan.

public byte[]? PushdownFilters { get; init; }

Raw embedded-IPC pushdown-filter bytes — STATIC filters only (join-key values aren't known at plan time; they arrive later, per split init, via TableInitParams.JoinKeys). On a continuation call this already includes refined_filters ' narrowing merged in. Decode with Internal.PushdownFilterCodec.Decode. null when DuckDB pushed no filters down.

public long? MaxSplitsPerResponse { get; init; }

Pagination cap for THIS call — not a sizing hint. A function that ignores it may return more splits than asked; nothing here truncates on its behalf.

public long? MinSplits { get; init; }

The parallelism FLOOR (the client's own thread count) — a small but expensive table still needs one split per thread. null when the client has none.

public long? TargetSplitBytes { get; init; }

The primary sizing lever: emit splits of roughly this many bytes each, since the client cannot see per-split cost and claims them greedily as interchangeable units. null when the client has no opinion.

class PlanResult

source

public sealed class PlanResult

Description

What an ITableFunction.Plan call produces: the splits, plus the few plan-level facts an author can meaningfully set. Deliberately NOT the wire Protocol.TableFunctionPlanResult, which carries several fields the framework fills in or that no author should have to think about. An EMPTY Splits is legal and means "no work": a fully-pruned scan reaches it, and the client produces an empty result rather than an error — this is distinct from a function that never overrides ITableFunction.Plan at all, which ITableFunction.SupportsSplits (checked BEFORE Plan is ever called) already gates.

Public members

public IReadOnlyList<ScanSplit> Splits { get; init; }

One entry per unit of work. Empty is legal (see this type's doc comment).

public IReadOnlyList<byte[]>? NextCursors { get; init; }

Continued enumeration. More than one MUST partition the remaining enumeration disjointly and exhaustively — nothing on the client (or here) checks this.

public long? CatalogVersion { get; init; }

The snapshot this plan is pinned to, or null to use the live catalog version. It is the anchor every token in this plan is stamped with and checked against at redemption — naming a version the catalog will not agree with is how a plan is made to expire (see expired_token.test ).

public long? EstimatedTotalRows { get; init; }

public long? EstimatedTotalSplits { get; init; }

public long? MaxWorkers { get; init; }

Normative cap on redemption concurrency, or null for none.

public static PlanResult Of(IReadOnlyList<ScanSplit> splits) ;

A finished plan: these splits and no continuation.

public static readonly PlanResult Empty = new();

An empty plan: no splits, no continuation.

class ScanSplit

source

public sealed class ScanSplit

Description

One named, independently redeemable unit of scan work — the author-facing counterpart of Protocol.ScanSplitWire (which additionally carries the framework-stamped token ). A split NAMES work rather than describing it: "these three files at version 47" survives a retry; "rows 0-999 of whatever this returns now" does not — and a distributed engine WILL retry, so the difference is correctness, not tidiness. The same split may be redeemed more than once (recursive CTEs, retried tasks) and may be abandoned mid-stream (LIMIT, an empty join build side); neither is an error — see the redeeming TableInitParams.SplitPayloads. Set only Payload (and, optionally, the estimate fields) — the framework stamps the consistency anchor and the bind fingerprint into the token, so an author never writes any of that bookkeeping.

Public members

public bool RowsExact { get; init; }

Whether EstimatedRows is exact rather than an estimate.

public byte[]? ColumnStatistics { get; init; }

public byte[]? EndPosition { get; init; }

Inclusive upper bound; null means UNBOUNDED.

public byte[]? PartitionBounds { get; init; }

2-row (min, max) batch in the vgi_partition_values encoding, one column per partition column — see Internal.PartitionValuesCodec.

public byte[]? StartPosition { get; init; }

public long? EstimatedBytes { get; init; }

Byte estimate — load-bearing for an engine that bin-packs splits by weight; null degrades it to round-robin by count.

public long? EstimatedRows { get; init; }

Row estimate for this split, or null if unknown.

public required byte[] Payload { get; init; }

The worker's own opaque bytes naming this unit of work — round-tripped verbatim through the token and handed back on TableInitParams.SplitPayloads when this split is redeemed.

public static ScanSplit Of(byte[] payload) ;

A split naming the given work, with no estimates.

public static ScanSplit Of(byte[] payload, long rows, long bytes) ;

A split naming the given work, with an exact row count and a byte estimate.

class TableArgFields

source

public static class TableArgFields

Description

Small convenience factories for ITableFunction.ArgumentsSchema fields — every table-function fixture needs a positional-vs-named field one way or another, so this avoids each one hand-rolling the vgi_arg=named metadata dictionary (see VgiWireMetadata).

Public members

public static Field AnyVarargs(string name) ;

An ANY-typed varargs field (e.g. constant_columns 's trailing arguments) — vgi_type=any + vgi_varargs=true metadata.

public static Field Named(string name, IArrowType type, bool nullable = true) ;

public static Field NamedWithDoc(string name, IArrowType type, string doc, bool nullable = true) ;

A named field carrying human-readable documentation ( vgi_doc metadata, e.g. a COPY TO/FROM format's option_description ) alongside the ordinary vgi_arg=named marker.

public static Field Positional(string name, IArrowType type, bool nullable = true) ;

public static Field PositionalWithRange( string name, IArrowType type, double ge = double.NaN, double gt = double.NaN, double le = double.NaN, double lt = double.NaN, bool nullable = true) ;

A positional field declaring a numeric range constraint (agent discovery via vgi_function_arguments() 's arg_range column) — each bound is double.NaN when unset (see Attributes.ConstParamAttribute's identically-shaped bounds). Surfacing the constraint is purely declarative; a caller that also wants BIND-TIME enforcement (e.g. rejecting a negative count) still checks it itself.

public static Field Table(string name) ;

The TABLE-typed argument a table-in-out/table-buffering function's ITableFunction.ArgumentsSchema-equivalent declares ( vgi_type=table metadata) — the field's own Arrow TYPE is irrelevant and never round-trips (the C++ side unconditionally overrides it to LogicalType::TABLE once this marker is seen, per vgi_arrow_utils.cpp 's BuildArgumentSpecs ), only its NAME and POSITION matter (the name becomes the table-input's registered arg name; the position is excluded from the "positional_N" renumbering Internal.TableArgCodec reads back at bind — the C++ side skips the TABLE slot entirely when building BindRequest.Arguments , so the SURVIVING positional args are renumbered contiguously starting at 0 in their original relative order).

public static Field TypedVarargs(string name, IArrowType type, bool nullable = true) ;

A TYPED varargs field (e.g. a blended row_sum(v1, v2, …) function's per-row value columns) — vgi_varargs=true metadata on a field of the DECLARED type, unlike AnyVarargs's vgi_type=any sentinel: every vararg at a call site must resolve to (or implicitly cast to) exactly type.

class TableBindParams

source

public sealed class TableBindParams

Description

Parameters an ITableFunction sees at bind time.

Public members

public Protocol.CopyFromContext? CopyFrom { get; init; }

Non-null only when this bind opened a COPY … FROM (FORMAT '<this function's name>', …) — the destination path and the exact output schema DuckDB requires (no cast is inserted for COPY FROM). See Buffering.CopyToFunction's sibling doc comment for the COPY TO side.

public Schema? InputSchema { get; init; }

The concrete per-call argument schema DuckDB resolved (decoded from Protocol.BindRequest.InputSchema), field-for-field matching ITableFunction.ArgumentsSchema's declared positional/named order — the REAL resolved type behind any ANY -typed/varargs argument (e.g. constant_columns ). null when the call site declared no such dynamic arguments.

public byte[] ArgumentsBytes { get; init; }

Opaque, not-yet-decoded serialized argument bytes — decode with TableArgCodec.Decode (or use Arguments, already decoded).

public byte[] AttachOpaqueData { get; init; }

Raw Protocol.BindRequest.AttachOpaqueData — echoed back verbatim by the C++ extension for every RPC belonging to ONE ATTACH 's lifetime, so (unlike CatalogRegistry.DefaultIdentity's name-derived, deterministic-per-name routing key) it is safe to use as a genuinely per-attach-SESSION unique key — e.g. scoping a writable fixture's durable row store so two independent ATTACH es of the same catalog (two parallel test files, or two attaches in one session) never see each other's data. Empty when the call carries none.

public byte[] TransactionOpaqueData { get; init; }

Raw Protocol.BindRequest.TransactionOpaqueData — non-empty only when this call runs inside an explicit SQL transaction ( BEGIN / COMMIT / ROLLBACK ) on a catalog that advertised Protocol.CatalogAttachResult.SupportsTransactions; empty for an autocommit statement (each gets its own fresh, effectively-unused transaction) or a catalog that doesn't support transactions at all. Use it as a per-transaction storage key (see Internal.FunctionStorage — already cross-process/durable, so it doubles as transaction-scoped storage keyed by this value instead of an execution id) for state that must survive across multiple binds within ONE transaction and be cleared on COMMIT/ROLLBACK — see table/transaction_storage.test 's tx_cached_value fixture.

public byte[]? Settings { get; init; }

Opaque, not-yet-decoded serialized settings bytes (one row, columns named by DuckDB setting key) — null when this function declared no ITableFunction.RequiredSettings.

public required Internal.SecretsAccessor Secrets { get; init; }

Secrets access for this bind attempt — statically pre-resolved secrets are readable immediately via Internal.SecretsAccessor.Resolved; a DYNAMIC (call-argument- derived) scope lookup goes through Internal.SecretsAccessor.Get from ITableFunction.Bind, which triggers the C++ extension's two-phase bind retry when unresolved. See Internal.SecretsAccessor's doc comment.

public required TableArguments Arguments { get; init; }

The decoded view of ArgumentsBytes — every positional/named SQL argument this call was made with (all bind-time constants; a table function has no per-row input).

public required string FunctionName { get; init; }

public string? AtUnit { get; init; }

The time-travel AT (VERSION => …) / AT (TIMESTAMP => …) clause unit/value this bind carries (Protocol.BindRequest.AtUnit/ AtValue ) — both null when the query has no AT clause. A function that resolves its own version (rather than relying on a catalog-level Catalog.CatalogTable.ResolveAtClause/ Catalog.CatalogTable.ResolveScanArguments swap) reads these directly — see table/time_travel_pushdown.test 's tt_pushdown_fn .

public string? AtValue { get; init; }

See AtUnit.

class TableInitParams

source

public sealed class TableInitParams

Description

Parameters an ITableFunction sees when its producer stream is opened (mirrors the init RPC) — the table-function analog of Scalar.ScalarProcessParams, except this fires once per call (not once per batch) since a table function's producer then drives its own output pace.

Public members

public IReadOnlyList<byte[]>? JoinKeys { get; init; }

One embedded-IPC single-column batch per IN-filter/join-key column ( InitRequest.JoinKeys ) — a pushdown_filters node of type "join_keys" names which one of these (by its keys_column field) holds its candidate value set. Decode with Internal.PushdownFilterCodec's join-key helpers.

public IReadOnlyList<byte[]>? SplitPayloads { get; init; }

The VERIFIED, envelope-stripped ScanSplit.Payload bytes this init is redeeming — null for an ordinary (non-split) init, and a single-element list for a split init (the client redeems exactly one split per init call; see AdvanceToNextSplit 's greedy per-split claim loop). A function that declared ITableFunction.SupportsSplits but is only ever meant to be read through the split path can check this for null and refuse the ordinary-init fallback (see splits/rollback.test 's vgi_split_scans=false scenario).

public IReadOnlyList<long>? ProjectionIds { get; init; }

Zero-based indices (into OutputSchema) of the columns DuckDB actually needs — null means "all columns". Only meaningful when this function advertised ITableFunction.ProjectionPushdown; otherwise DuckDB still expects (and will itself trim) the FULL OutputSchema from every emitted batch.

public Protocol.CopyFromContext? CopyFrom { get; init; }

Non-null only when this init opened a COPY … FROM — see TableBindParams.CopyFrom's doc comment.

public Schema ProjectedSchema;

Convenience: OutputSchema narrowed to ProjectionIds (or the full schema when ProjectionIds is null) — the schema a projection-pushdown-aware producer should actually emit.

public VgiNullOrder? OrderByNullOrder { get; init; }

public VgiOrderByDirection? OrderByDirection { get; init; }

public byte[] AttachOpaqueData { get; init; }

Raw BindRequest.AttachOpaqueData — see TableBindParams.AttachOpaqueData's doc comment.

public byte[] TransactionOpaqueData { get; init; }

See TableBindParams.TransactionOpaqueData — the same value, re-decoded from the Protocol.BindRequest embedded in this init call.

public byte[]? ExecutionId { get; init; }

public byte[]? PushdownFilters { get; init; }

Raw embedded-IPC pushdown-filter bytes ( InitRequest.PushdownFilters ) — null when DuckDB pushed no filters down. Only meaningful when this function advertised ITableFunction.FilterPushdown. Decode with PushdownFilter.Decode.

public byte[]? Secrets { get; init; }

Fully-RESOLVED secrets from the bind call that opened this producer — by the time init runs, any two-phase secret-scope retry (see Internal.SecretsAccessor) has already completed, so this is plain already-resolved data: decode with Internal.SecretArgCodec.Decode then Internal.SecretArgCodec.FindByType/ Internal.SecretArgCodec.ForScopeOfType. null when no secrets were resolved.

public byte[]? Settings { get; init; }

public double? TablesamplePercentage { get; init; }

public long? OrderByLimit { get; init; }

public long? RowLimit { get; init; }

public long? TablesampleSeed { get; init; }

public required Schema OutputSchema { get; init; }

The resolved per-call output schema (from ITableFunction.ResolveOutputSchema). Every batch the returned ITableFunctionProducer emits must use this schema — or the ProjectedSchema subset when ProjectionIds is non-null and this function advertises ITableFunction.ProjectionPushdown.

public required TableArguments Arguments { get; init; }

Same decoded arguments as the bind call that opened this producer (re-decoded rather than reused from TableBindParams — nothing about one call may be cached on the shared ITableFunction singleton).

public required string FunctionName { get; init; }

public string? AtUnit { get; init; }

See TableBindParams.AtUnit — the same value, re-decoded from the Protocol.BindRequest embedded in this init call. This is where a function-backed table with a version-independent schema (so it has no need to override ITableFunction.Bind/ITableFunction.ResolveOutputSchema) should resolve its version — e.g. inside ITableFunction.CreateProducer.

public string? AtValue { get; init; }

See AtUnit.

public string? OrderByColumnName { get; init; }