Skip to content
Query.Farm
Talk with Us

Table functions

On this page

Row generators: arguments in, a whole relation out.

struct BatchState

Section titled “struct BatchState”
source
type BatchState struct {
Remaining  int64
BatchSize  int64
Index    int64
}

Description

BatchState tracks the remaining/batchSize/index bookkeeping for table functions that generate a fixed number of rows in batches. Embed this in your state struct to use with GenerateBatch.

Index is a public field so callbacks can read the current row offset. All fields are managed by GenerateBatch — do not modify them directly.

Methods

function NewBatchState

Section titled “function NewBatchState”
source
func NewBatchState(count, batchSize int64) BatchState

NewBatchState creates a BatchState with the given total count and batch size.

interface CardinalityEstimator

Section titled “interface CardinalityEstimator”
source
type CardinalityEstimator interface {
// Cardinality estimates the function's output row count for the optimizer.
Cardinality(params *BindParams) (*TableCardinality, error)
}

Description

CardinalityEstimator is an optional interface for TypedTableFunc implementations that can estimate their output row count for query optimization.

type ColumnSet

Section titled “type ColumnSet”
source
type ColumnSet map[string]struct{}

Description

ColumnSet is a set of column names, typically used for projection pushdown.

Methods

method Contains

Section titled “method Contains”
source
func (cs ColumnSet) Contains(name string) bool

Contains returns true if the named column is in the set.

function ProjectedColumns

Section titled “function ProjectedColumns”
source
func ProjectedColumns(projectionIDs []int32, fullSchema *arrow.Schema) ColumnSet

ProjectedColumns returns the set of column names that should be generated, given projection IDs and the full (unprojected) schema. If projectionIDs is nil, all columns are included.

interface OnIniter

Section titled “interface OnIniter”
source
type OnIniter interface {
// OnInit performs global initialization (e.g. worker count, partitioning).
OnInit(params *InitParams) (*GlobalInitResponse, error)
}

Description

OnIniter is an optional interface for TypedTableFunc implementations that need custom OnInit behavior (e.g., multi-worker partitioning with work queues). If not implemented, the adapter defaults to DefaultInit() (MaxWorkers: 1).

interface StatisticsProvider

Section titled “interface StatisticsProvider”
source
type StatisticsProvider interface {
// Statistics reports per-column output statistics to the optimizer.
Statistics(params *BindParams) ([]ColumnStatistics, error)
}

Description

StatisticsProvider is an optional interface for TypedTableFunc implementations that can report per-column output statistics to the optimizer (min/max, null-ness, distinct count, string length).

struct TableCardinality

Section titled “struct TableCardinality”
source
type TableCardinality struct {
// Estimate is the estimated number of rows.
Estimate  int64  `vgirpc:"estimate,nullable"`
// Max is the maximum number of rows (-1 = unknown).
Max  int64  `vgirpc:"max,nullable"`
}

Description

TableCardinality represents the estimated cardinality of a table function’s output.

interface TableFunction

Section titled “interface TableFunction”
source
type TableFunction interface {
// Name returns the function name used in SQL.
Name() string
// Metadata returns descriptive metadata.
Metadata() FunctionMetadata
// ArgumentSpecs returns the function's argument specifications.
ArgumentSpecs() []ArgSpec
// OnBind resolves the output schema given the bind parameters.
OnBind(params *BindParams) (*BindResponse, error)
// OnInit performs one-time initialization and returns execution parameters.
OnInit(params *InitParams) (*GlobalInitResponse, error)
// NewState creates the initial mutable state for this function execution.
NewState(params *ProcessParams) (interface{}, error)
// Process generates the next output batch. It must either emit data via
// out.Emit/EmitArrays or call out.Finish() to signal end-of-stream.
Process(ctx context.Context, params *ProcessParams, state interface{}, out *vgirpc.OutputCollector) error
}

Description

TableFunction is the interface for table VGI functions. Table functions generate output without receiving input (Producer mode).

Methods

function AsTableFunction

Section titled “function AsTableFunction”
source
func AsTableFunction[S any](f TypedTableFunc[S]) TableFunction

interface TableFunctionWithCardinality

Section titled “interface TableFunctionWithCardinality”
source
type TableFunctionWithCardinality interface {
TableFunction
// Cardinality returns an estimated row count for query optimization.
Cardinality(params *BindParams) (*TableCardinality, error)
}

Description

TableFunctionWithCardinality extends TableFunction with cardinality estimation.

interface TableFunctionWithStatistics

Section titled “interface TableFunctionWithStatistics”
source
type TableFunctionWithStatistics interface {
TableFunction
// Statistics returns per-output-column stats (empty slice or nil = unknown).
Statistics(params *BindParams) ([]ColumnStatistics, error)
}

Description

TableFunctionWithStatistics extends TableFunction with per-column statistics that help the optimizer fold or skip filters before running the scan.

interface TypedTableFunc

Section titled “interface TypedTableFunc”
source
type TypedTableFunc[S any] interface {
// Name returns the function name used to invoke it in SQL.
Name() string
// Metadata returns descriptive metadata for the function.
Metadata() FunctionMetadata
// ArgumentSpecs returns the function's argument specifications.
ArgumentSpecs() []ArgSpec
// OnBind resolves the output schema and bind state from the bind parameters.
OnBind(params *BindParams) (*BindResponse, error)
// NewState creates a fresh, typed per-scan state value.
NewState(params *ProcessParams) (*S, error)
// Process generates output rows for one scan step, emitting them via out.
Process(ctx context.Context, params *ProcessParams, state *S, out *vgirpc.OutputCollector) error
}

Description

TypedTableFunc is the recommended interface for table functions. It provides compile-time type safety for state management, eliminating the unsafe state.(*myType) assertions required by the lower-level TableFunction interface.

Use the lower-level TableFunction interface only for advanced use cases that need non-standard state patterns.

Implementations may also satisfy OnIniter (custom OnInit) and/or CardinalityEstimator (cardinality estimation). These are detected automatically by AsTableFunction.

struct typedTableAdapter

Section titled “struct typedTableAdapter”
source
type typedTableAdapter[S any] struct {
inner    TypedTableFunc[S]
onInit    func(*InitParams) (*GlobalInitResponse, error)
statsProvider  StatisticsProvider
}

Description

typedTableAdapter implements TableFunction by delegating to a TypedTableFunc[S].

Methods

method ArgumentSpecs

Section titled “method ArgumentSpecs”
source
func (a *typedTableAdapter[S]) ArgumentSpecs() []ArgSpec

ArgumentSpecs forwards to the wrapped typed function’s ArgumentSpecs.

method DynamicToString

Section titled “method DynamicToString”
source
func (a *typedTableAdapter[S]) DynamicToString(ctx context.Context, params *DynamicToStringParams) ([]string, []string, error)

DynamicToString forwards to the inner typed function when it implements the hook. The framework’s RPC handler does the interface check on the adapter (the user-facing TableFunction); without this forwarder a typed function’s hook would be invisible.

method Metadata

Section titled “method Metadata”
source
func (a *typedTableAdapter[S]) Metadata() FunctionMetadata

Metadata forwards to the wrapped typed function’s Metadata.

method Name

Section titled “method Name”
source
func (a *typedTableAdapter[S]) Name() string

Name forwards to the wrapped typed function’s Name.

method NewState

Section titled “method NewState”
source
func (a *typedTableAdapter[S]) NewState(params *ProcessParams) (interface{}, error)

NewState forwards to the wrapped typed function’s NewState, returning the typed state as an untyped interface{}.

method OnBind

Section titled “method OnBind”
source
func (a *typedTableAdapter[S]) OnBind(params *BindParams) (*BindResponse, error)

OnBind forwards to the wrapped typed function’s OnBind.

method OnInit

Section titled “method OnInit”
source
func (a *typedTableAdapter[S]) OnInit(params *InitParams) (*GlobalInitResponse, error)

OnInit invokes the optional OnIniter hook if present, otherwise returns DefaultInit.

method Process

Section titled “method Process”
source
func (a *typedTableAdapter[S]) Process(ctx context.Context, params *ProcessParams, state interface{}, out *vgirpc.OutputCollector) error

Process type-asserts the untyped state to *S and forwards to the wrapped typed function’s Process, returning an error on a state type mismatch.

method Statistics

Section titled “method Statistics”
source
func (a *typedTableAdapter[S]) Statistics(params *BindParams) ([]ColumnStatistics, error)

Statistics delegates to the wrapped function’s StatisticsProvider if it implements one, returning nil otherwise.

struct typedTableAdapterWithCard

Section titled “struct typedTableAdapterWithCard”
source
type typedTableAdapterWithCard[S any] struct {
*typedTableAdapter[S]
card  CardinalityEstimator
}

Description

typedTableAdapterWithCard embeds typedTableAdapter and additionally implements TableFunctionWithCardinality, so the type assertion in protocol.go (fn.(TableFunctionWithCardinality)) succeeds.

Methods

method ArgumentSpecs

Section titled “method ArgumentSpecs”
source
func (a typedTableAdapterWithCard) ArgumentSpecs() []ArgSpec

ArgumentSpecs forwards to the wrapped typed function’s ArgumentSpecs.

method Cardinality

Section titled “method Cardinality”
source
func (a *typedTableAdapterWithCard[S]) Cardinality(params *BindParams) (*TableCardinality, error)

Cardinality forwards to the wrapped CardinalityEstimator’s Cardinality.

method DynamicToString

Section titled “method DynamicToString”
source
func (a typedTableAdapterWithCard) DynamicToString(ctx context.Context, params *DynamicToStringParams) ([]string, []string, error)

DynamicToString forwards to the inner typed function when it implements the hook. The framework’s RPC handler does the interface check on the adapter (the user-facing TableFunction); without this forwarder a typed function’s hook would be invisible.

method Metadata

Section titled “method Metadata”
source
func (a typedTableAdapterWithCard) Metadata() FunctionMetadata

Metadata forwards to the wrapped typed function’s Metadata.

method Name

Section titled “method Name”
source
func (a typedTableAdapterWithCard) Name() string

Name forwards to the wrapped typed function’s Name.

method NewState

Section titled “method NewState”
source
func (a typedTableAdapterWithCard) NewState(params *ProcessParams) (interface{}, error)

NewState forwards to the wrapped typed function’s NewState, returning the typed state as an untyped interface{}.

method OnBind

Section titled “method OnBind”
source
func (a typedTableAdapterWithCard) OnBind(params *BindParams) (*BindResponse, error)

OnBind forwards to the wrapped typed function’s OnBind.

method OnInit

Section titled “method OnInit”
source
func (a typedTableAdapterWithCard) OnInit(params *InitParams) (*GlobalInitResponse, error)

OnInit invokes the optional OnIniter hook if present, otherwise returns DefaultInit.

method Process

Section titled “method Process”
source
func (a typedTableAdapterWithCard) Process(ctx context.Context, params *ProcessParams, state interface{}, out *vgirpc.OutputCollector) error

Process type-asserts the untyped state to *S and forwards to the wrapped typed function’s Process, returning an error on a state type mismatch.

method Statistics

Section titled “method Statistics”
source
func (a typedTableAdapterWithCard) Statistics(params *BindParams) ([]ColumnStatistics, error)

Statistics delegates to the wrapped function’s StatisticsProvider if it implements one, returning nil otherwise.

function BatchFromMap

Section titled “function BatchFromMap”
source
func BatchFromMap(schema *arrow.Schema, columns map[string]arrow.Array, numRows int64) (arrow.RecordBatch, error)

BatchFromMap reorders columns from a name-keyed map to match schema field order and creates a RecordBatch. It consumes all arrays in the map — every array (both schema-matched and extra) is released on success and on error, matching GenerateBatch’s ownership model. Returns an error if a schema field is missing from the map. Extra map keys are silently ignored but their arrays are still released.

function BuildAllNullArray

Section titled “function BuildAllNullArray”
source
func BuildAllNullArray(dt arrow.DataType, n int64) arrow.Array

BuildAllNullArray creates an n-row all-NULL array of the given type. The type is created via NewBuilder so any Arrow primitive / nested type works.

function BuildArray

Section titled “function BuildArray”
source
func BuildArray[T any, B ArrayBuilder[T]](n int64, newBuilder func(memory.Allocator) B, fn func(i int64) T) arrow.Array

BuildArray creates an array of any type using the ArrayBuilder generic constraint. Use this for types not covered by BuildInt64Array, BuildFloat64Array, etc.

function BuildBinaryArray

Section titled “function BuildBinaryArray”
source
func BuildBinaryArray(n int64, fn func(i int64) []byte) arrow.Array

BuildBinaryArray creates a binary array by calling fn for each row index.

function BuildBooleanArray

Section titled “function BuildBooleanArray”
source
func BuildBooleanArray(n int64, fn func(i int64) bool) arrow.Array

BuildBooleanArray creates a boolean array by calling fn for each row index.

function BuildFloat64Array

Section titled “function BuildFloat64Array”
source
func BuildFloat64Array(n int64, fn func(i int64) float64) arrow.Array

BuildFloat64Array creates a float64 array by calling fn for each row index.

function BuildInt32Array

Section titled “function BuildInt32Array”
source
func BuildInt32Array(n int64, fn func(i int64) int32) arrow.Array

BuildInt32Array creates an int32 array by calling fn for each row index.

function BuildInt64Array

Section titled “function BuildInt64Array”
source
func BuildInt64Array(n int64, fn func(i int64) int64) arrow.Array

BuildInt64Array creates an int64 array by calling fn for each row index. The caller is responsible for releasing the returned array, unless using GenerateBatch which handles release automatically.

function BuildStringArray

Section titled “function BuildStringArray”
source
func BuildStringArray(n int64, fn func(i int64) string) arrow.Array

BuildStringArray creates a string array by calling fn for each row index.

function BuildUint64Array

Section titled “function BuildUint64Array”
source
func BuildUint64Array(n int64, fn func(i int64) uint64) arrow.Array

BuildUint64Array creates a uint64 array by calling fn for each row index.

function FindColumn

Section titled “function FindColumn”
source
func FindColumn(batch arrow.RecordBatch, name string) arrow.Array

FindColumn returns the column array from a RecordBatch by field name, or nil if not found.

function GenerateBatch

Section titled “function GenerateBatch”
source
func GenerateBatch(bs *BatchState, out *vgirpc.OutputCollector, generateFn func(size int64) ([]arrow.Array, error)) error

GenerateBatch handles one batch of a batch-splitting table function. Call this from Process — it handles the complete remaining-work pattern:

  1. If Remaining <= 0, calls out.Finish() and returns
  2. Computes size = min(Remaining, BatchSize)
  3. Calls generateFn(size) to produce arrays
  4. Emits the arrays via out.EmitArrays
  5. Releases all returned arrays
  6. Updates Remaining and Index

VGI’s Process is called once per batch by the framework. This function processes exactly one batch per call — it does NOT loop.

The generateFn closure should capture bs.Index at the start to know the current row offset before GenerateBatch advances it.

function GenerateBatchMap

Section titled “function GenerateBatchMap”
source
func GenerateBatchMap(bs *BatchState, out *vgirpc.OutputCollector, schema *arrow.Schema, generateFn func(size int64) (map[string]arrow.Array, error)) error

GenerateBatchMap handles one batch of a batch-splitting table function using name-keyed columns. Same lifecycle as GenerateBatch (check remaining, compute size, call callback, update state) but the callback returns map[string]arrow.Array. Uses BatchFromMap internally and emits via out.Emit.

function OptionalBool

Section titled “function OptionalBool”
source
func OptionalBool(args *Arguments, name string, defaultVal bool) bool

OptionalBool extracts a named bool argument, returning defaultVal if the argument is null, not present, or cannot be converted to bool.

function OptionalFloat64

Section titled “function OptionalFloat64”
source
func OptionalFloat64(args *Arguments, name string, defaultVal float64) float64

OptionalFloat64 extracts a named float64 argument, returning defaultVal if the argument is null, not present, or cannot be converted to float64.

function OptionalInt64

Section titled “function OptionalInt64”
source
func OptionalInt64(args *Arguments, name string, defaultVal int64) int64

OptionalInt64 extracts a named int64 argument, returning defaultVal if the argument is null, not present, or cannot be converted to int64. This is a convenience helper that never returns an error — type mismatches silently return the default value.

function OptionalString

Section titled “function OptionalString”
source
func OptionalString(args *Arguments, name string, defaultVal string) string

OptionalString extracts a named string argument, returning defaultVal if the argument is null, not present, or cannot be converted to string.

function assertGobEncodable

Section titled “function assertGobEncodable”
source
func assertGobEncodable(t reflect.Type, path string, seen map[reflect.Type]bool)

assertGobEncodable panics if t — reachable from a state field via path — holds a value gob cannot encode. seen guards against self-referential types.

function implementsGobEncoder

Section titled “function implementsGobEncoder”
source
func implementsGobEncoder(t reflect.Type) bool

AsTableFunction wraps a TypedTableFunc into a TableFunction for registration with Worker.RegisterTable. The adapter:

  • Provides type-safe state casting (returns error instead of panic)
  • Defaults OnInit to DefaultInit() (MaxWorkers: 1) unless OnIniter is implemented
  • Delegates Cardinality if CardinalityEstimator is implemented

Usage:

func NewSequenceFunction() vgi.TableFunction {
return vgi.AsTableFunction[sequenceState](&SequenceFunction{})
}

implementsGobEncoder reports whether t (or *t) defines its own gob encoding, in which case it manages its own serialization and is exempt from inspection.

function validateGobState

Section titled “function validateGobState”
source
func validateGobStateS any

validateGobState fails fast (at registration) when the per-scan state type S cannot be gob-encoded for HTTP rehydration. gob otherwise surfaces these only mid-query, on the first HTTP continuation, with a cryptic message. Two pitfalls are caught by walking S’s exported, gob-reachable fields:

  • a struct whose fields are all unexported (e.g. struct{ done bool }): gob
encodes nothing and reports "type ... has no exported fields".
  • an exported field of a kind gob cannot encode — an interface (the common
case being an Arrow `arrow.Record`/`arrow.Array` stashed in state to emit
later), a chan, a func, or an unsafe.Pointer. The fix is to store plain
serializable Go values (slices, scalars) in state and rebuild the Arrow
batch in Process, mirroring the SDK's static_data.go example.

A truly empty struct{} is fine, and any type providing its own gob.GobEncoder is exempt (at the top level or as a nested field). Mirrors vgi-python enforcing serializable state at class-definition time.