Skip to content
Query.Farm
Talk with Us

Arrow helpers

On this page

Building and emitting Arrow batches.

struct BatchBuilder

Section titled “struct BatchBuilder”
source
type BatchBuilder struct {
schema    *arrow.Schema
mem    memory.Allocator
columns    []columnBuilder
rowCount  int64
}

Description

BatchBuilder accumulates rows for a single output schema and emits an arrow.RecordBatch.

Methods

method AppendNullRow

Section titled “method AppendNullRow”
source
func (b *BatchBuilder) AppendNullRow()

AppendNullRow appends a row of all NULLs. Useful for placeholders.

method AppendRow

Section titled “method AppendRow”
source
func (b *BatchBuilder) AppendRow(row map[string]any) error

AppendRow writes one row from a name-keyed value map. Missing keys, nil values, and nil pointers produce NULLs. Returns the first per-column error it encounters; the row is left in a partially-appended state if that happens (call Release() to discard).

method Build

Section titled “method Build”
source
func (b *BatchBuilder) Build() (arrow.RecordBatch, error)

Build finalizes the accumulated rows into a RecordBatch and releases the underlying builders. The returned batch is owned by the caller and must be Released by them when no longer needed.

function NewBatchBuilder

Section titled “function NewBatchBuilder”
source
func NewBatchBuilder(schema *arrow.Schema) *BatchBuilder

NewBatchBuilder constructs a BatchBuilder for the given schema using the default Arrow allocator.

function NewBatchBuilderWithAllocator

Section titled “function NewBatchBuilderWithAllocator”
source
func NewBatchBuilderWithAllocator(schema *arrow.Schema, mem memory.Allocator) *BatchBuilder

NewBatchBuilderWithAllocator constructs a BatchBuilder with a caller-provided memory allocator. Useful when integrating with an existing arena.

method Release

Section titled “method Release”
source
func (b *BatchBuilder) Release()

Release discards any accumulated state without producing a batch. Safe to call after Build() (no-op).

method Rows

Section titled “method Rows”
source
func (b *BatchBuilder) Rows() int64

Rows returns the number of rows appended so far.

method Schema

Section titled “method Schema”
source
func (b *BatchBuilder) Schema() *arrow.Schema

Schema returns the output schema.

func type EmitOption

Section titled “func type EmitOption”
source
type EmitOption func(map[string]string) error

Description

EmitOption contributes annotation metadata to an emitted batch. Options are applied in order, so a later option overwrites an earlier one’s keys.

Methods

function WithCacheControl

Section titled “function WithCacheControl”
source
func WithCacheControl(cc *CacheControl) EmitOption

WithCacheControl advertises a cacheable result. The cache-control keys are read once per result, off the FIRST data batch of the stream — passing this on a later batch has no effect. Nil is a no-op, so a caller can write

var cc *vgi.CacheControl
if firstBatch { cc = &vgi.CacheControl{Ttl: vgi.Seconds(300)} }
vgi.Emit(out, batch, vgi.WithCacheControl(cc))

function WithMetadata

Section titled “function WithMetadata”
source
func WithMetadata(key, value string) EmitOption

WithMetadata sets one arbitrary annotation key on the emitted batch.

struct PartitionValue

Section titled “struct PartitionValue”
source
type PartitionValue struct {
Min  any
Max  any
}

Description

PartitionValue is the (min, max) pair for one partition column. Values are Go scalars compatible with the column’s Arrow type (see appendValue).

struct columnBuilder

Section titled “struct columnBuilder”
source
type columnBuilder struct {
field    arrow.Field
builder    array.Builder
appender  func(v any) error
}

Description

columnBuilder pairs a typed Arrow builder with the field it serves. The appender closure encapsulates the type-specific dispatch path.

function BatchToSecretsMap

Section titled “function BatchToSecretsMap”
source
func BatchToSecretsMap(batch arrow.RecordBatch) map[string]map[string]interface{}

BatchToSecretsMap converts a secrets RecordBatch to a map of maps.

function BatchToSettingsMap

Section titled “function BatchToSettingsMap”
source
func BatchToSettingsMap(batch arrow.RecordBatch) map[string]interface{}

BatchToSettingsMap converts a single-row settings RecordBatch to a map.

function BuildArgSchema

Section titled “function BuildArgSchema”
source
func BuildArgSchema(specs []ArgSpec) *arrow.Schema

BuildArgSchema creates an Arrow schema from ArgSpecs with VGI metadata markers.

function DeserializeRecordBatch

Section titled “function DeserializeRecordBatch”
source
func DeserializeRecordBatch(data []byte) (arrow.RecordBatch, error)

DeserializeRecordBatch reads a RecordBatch from IPC bytes.

function DeserializeSchema

Section titled “function DeserializeSchema”
source
func DeserializeSchema(data []byte) (*arrow.Schema, error)

DeserializeSchema reads an Arrow schema from IPC bytes.

function Emit

Section titled “function Emit”
source
func Emit(out *vgirpc.OutputCollector, batch arrow.RecordBatch, opts …EmitOption) error

Emit emits a batch with the annotation metadata contributed by opts. With no options it is equivalent to out.Emit(batch).

function EmitBatchIndex

Section titled “function EmitBatchIndex”
source
func EmitBatchIndex(out *vgirpc.OutputCollector, batch arrow.RecordBatch, batchIndex int64, opts …EmitOption) error

EmitBatchIndex emits a batch tagged with vgi_batch_index. Use it from a table function that declares SupportsBatchIndex. The C++ extension enforces monotonicity and the per-pipeline cap.

function EmitParentRows

Section titled “function EmitParentRows”
source
func EmitParentRows(out *vgirpc.OutputCollector, batch arrow.RecordBatch, parentRows []int32, opts …EmitOption) error

EmitParentRows emits a batch declaring, per output row, which input row produced it — the provenance map for the batched correlated LATERAL operator (a blended function under FROM t, f(t.x) / LATERAL f(t.x)): the C++ extension ships a whole input chunk to the worker in ONE exchange and reads ONE output batch, then maps each output row back to the input row that produced it via this array — so a 1->N fan-out or 1->0 filter can be batched instead of driven row-by-row.

parentRows[i] is the 0-based index (into the input batch) of the row that produced output row i. Encoded as a raw little-endian int32 array (NOT Arrow IPC), base64-encoded, under vgi_rpc.parent_row#b64 — the C++ side reinterprets the bytes directly. Absent metadata means an identity 1->1 map (the common case: the extension assumes it, and requires output rows == input rows).

Contract: len(parentRows) MUST equal batch.NumRows() (a mismatch is a worker bug that would corrupt the stamping). Values are range-checked against the input width on the C++ side (which knows it authoritatively). Mirrors vgi-python’s out.emit(…, parent_rows=…).

function EmitPartitioned

Section titled “function EmitPartitioned”
source
func EmitPartitioned(out *vgirpc.OutputCollector, batch arrow.RecordBatch, partitionFields []arrow.Field, kind PartitionKind, explicit map[string]PartitionValue, opts …EmitOption) error

EmitPartitioned emits a batch annotated with vgi_partition_values for a partition-aware table function. partitionFields are the declared partition columns (build them with PartitionField). For each, the (min, max) pair comes from explicit when present, else is auto-extracted from the same-named column in the emitted batch. Mirrors vgi-python’s _merge_partition_values contract: the SINGLE_VALUE distinct-value check, the “requires partition-annotated fields” guard, and the annotated-but-absent error.

function EmptyBatch

Section titled “function EmptyBatch”
source
func EmptyBatch(schema *arrow.Schema) arrow.RecordBatch

EmptyBatch creates a zero-row batch with the given schema.

function PartitionField

Section titled “function PartitionField”
source
func PartitionField(name string, typ arrow.DataType, nullable bool) arrow.Field

PartitionField builds an Arrow field annotated as a partition column (schema metadata vgi.partition_column=true). Mirrors vgi-python’s partition_field.

function ProjectSchema

Section titled “function ProjectSchema”
source
func ProjectSchema(projectionIDs []int32, schema *arrow.Schema) *arrow.Schema

ProjectSchema returns a new schema with only the fields at the given indices.

function SchemaFromOrderedFields

Section titled “function SchemaFromOrderedFields”
source
func SchemaFromOrderedFields(names []string, types []arrow.DataType) *arrow.Schema

SchemaFromOrderedFields creates an Arrow schema preserving insertion order.

function SerializeRecordBatch

Section titled “function SerializeRecordBatch”
source
func SerializeRecordBatch(batch arrow.RecordBatch) ([]byte, error)

SerializeRecordBatch serializes a RecordBatch to IPC bytes.

function SerializeSchema

Section titled “function SerializeSchema”
source
func SerializeSchema(schema *arrow.Schema) ([]byte, error)

SerializeSchema serializes an Arrow schema to IPC bytes.

function ValidateTypeBounds

Section titled “function ValidateTypeBounds”
source
func ValidateTypeBounds(specs []ArgSpec, inputSchema *arrow.Schema) error

ValidateTypeBounds validates that the input schema field types satisfy the TypeBound predicates declared on each ArgSpec. For each non-const ArgSpec with ArrowType “any” and non-nil TypeBound, the corresponding input schema field must satisfy at least one predicate (OR logic). For varargs, all fields from the spec’s position onward are validated.

function applyEmitOptions

Section titled “function applyEmitOptions”
source
func applyEmitOptions(base map[string]string, opts []EmitOption) (map[string]string, error)

applyEmitOptions folds opts into a fresh metadata map, seeded with base. Returns nil when nothing was contributed, so the caller can emit unannotated.

function argTypeToArrowType

Section titled “function argTypeToArrowType”
source
func argTypeToArrowType(t string) arrow.DataType

argTypeToArrowType converts a VGI arg type string to an Arrow DataType.

function bbToBool

Section titled “function bbToBool”
source
func bbToBool(v any) (bool, bool)

function bbToBytes

Section titled “function bbToBytes”
source
func bbToBytes(v any) ([]byte, bool)

function bbToFloat64

Section titled “function bbToFloat64”
source
func bbToFloat64(v any) (float64, bool)

function bbToInt64

Section titled “function bbToInt64”
source
func bbToInt64(v any) (int64, bool)

function bbToSliceAny

Section titled “function bbToSliceAny”
source
func bbToSliceAny(v any) ([]any, bool)

toSliceAny coerces []T (for any T) and []any to a []any view via reflection. Returns ok=false for non-slices.

function bbToString

Section titled “function bbToString”
source
func bbToString(v any) (string, bool)

function bbToTime

Section titled “function bbToTime”
source
func bbToTime(v any) (time.Time, bool)

function bbToUint64

Section titled “function bbToUint64”
source
func bbToUint64(v any) (uint64, bool)

function columnMinMax

Section titled “function columnMinMax”
source
func columnMinMax(col arrow.Array) (min, max any, moreThanOne bool, err error)

columnMinMax returns the min and max of a non-null Arrow column, plus whether the column carries more than one distinct value. Supports the scalar types used by partition columns. moreThanOne is exactly (min != max) over the non-null values, so no per-row set is needed to detect a SINGLE_VALUE violation.

function deref

Section titled “function deref”
source
func deref(v any) any

deref dereferences pointer types so toX helpers can accept *string, *int64, etc.

function deserializeJoinKeys

Section titled “function deserializeJoinKeys”
source
func deserializeJoinKeys(entries [][]byte) map[string]arrow.Array

deserializeJoinKeys unpacks a list of IPC-serialized RecordBatches into a map from column name to Arrow array. Each batch should be a single-column batch; all columns across batches are flattened into one map by field name.

function extractScalarValue

Section titled “function extractScalarValue”
source
func extractScalarValue(col arrow.Array, idx int) interface{}

extractScalarValue extracts a Go value from an Arrow array at the given index.

function isNil

Section titled “function isNil”
source
func isNil(v any) bool

isNil returns true for untyped nil, nil interface, nil pointer, nil slice, nil map, and nil chan/func. Anything else is non-nil — including a non-nil pointer to a zero value.

function makeAppender

Section titled “function makeAppender”
source
func makeAppender(builder array.Builder, dt arrow.DataType) func(any) error

makeAppender returns a closure that appends a single value to builder, dispatching on the field’s Arrow type. The closure is built once per column at construction time so AppendRow’s hot path is one map lookup + one virtual call per column.

function matchesAnyBound

Section titled “function matchesAnyBound”
source
func matchesAnyBound(fieldType arrow.DataType, bounds []TypeBoundPredicate) bool

function predicateNames

Section titled “function predicateNames”
source
func predicateNames(bounds []TypeBoundPredicate) []string

predicateNames recovers the symbolic name of each TypeBoundPredicate via runtime reflection. Used in TypeBoundError messages to keep the wire-level error text aligned with vgi-python’s “_is_multipliable_type” / “_is_addable_type” convention (the SQL tests assert on these substrings).

function typeErr

Section titled “function typeErr”
source
func typeErr(v any, target string) error