Scalar functions

On this page

One row in, one value out — the per-row transform.

interface ArrayBuilder

source

type ArrayBuilder[T any] interface {
// Append adds a non-null value of type T to the end of the array being built.
Append(T)
// AppendNull adds a null entry to the end of the array being built.
AppendNull()
// Reserve pre-allocates capacity for at least n additional elements.
Reserve(int)
// NewArray finalizes the builder and returns the constructed Arrow array.
NewArray() arrow.Array
// Release frees the buffers held by the builder.
Release()
}

Description

ArrayBuilder is a constraint for Arrow typed builders used by the Map* functions.

interface ScalarFunction

source

type ScalarFunction interface {
// Name returns the function name used in SQL.
Name() string
// Metadata returns descriptive metadata.
Metadata() FunctionMetadata
// ArgumentSpecs returns the function's argument specifications.
ArgumentSpecs() []ArgSpec
// OnBind resolves the output schema given the bind parameters.
OnBind(params *BindParams) (*BindResponse, error)
// Process transforms an input batch into an output batch.
// The output batch must have the same number of rows as the input.
Process(ctx context.Context, params *ProcessParams, batch arrow.RecordBatch) (arrow.RecordBatch, error)
}

Description

ScalarFunction is the interface for scalar VGI functions. Scalar functions perform 1:1 row mapping: each input batch produces one output batch with the same number of rows.

Methods

function AsScalarFunction

source

func AsScalarFunction[A any](f TypedScalarFunc[A]) ScalarFunction

AsScalarFunction wraps a TypedScalarFunc[A] into a ScalarFunction so it can be passed to Worker.RegisterScalar. ArgumentSpecs is derived once from A’s struct tags; OnBind and Process bind A from params.Args on each call.

Column args (fields tagged const=false) are auto-populated from the input batch at Process time. Two field shapes are supported:

• arrow.Array — accepts any column type, advertises arrow_type=“any” plus

whatever `bound=` predicate is declared. This is the polymorphic mode
used by add_values, double, sum_values, etc. — output type is computed
in OnBindTyped from the actual input.

• *array.Int64 / *array.String / *array.Float64 / … — accepts only that

concrete fixed type. Advertises the matching arrow_type and lets the
body read values without a type assertion. Used by multiply, etc.

Varargs slice fields ([]arrow.Array) consume every remaining batch column.

Nested or parametric column types (struct, list, fixed_list, decimal, timestamp, duration) are not in the concrete-type registry because the Go pointer doesn’t carry the inner shape. Declare those as arrow.Array and validate the shape in OnBindTyped — same approach as vgi-python.

interface TypedScalarFunc

source

type TypedScalarFunc[A any] interface {
// Name returns the function name used in SQL.
Name() string
// Metadata returns descriptive metadata.
Metadata() FunctionMetadata
// OnBindTyped resolves the output schema given the bind parameters and
// the bound argument struct. args is populated from params.Args via
// BindArgs; column-arg fields are left at their zero value.
OnBindTyped(args *A, params *BindParams) (*BindResponse, error)
// ProcessTyped transforms an input batch into an output batch using the
// bound argument struct.
ProcessTyped(ctx context.Context, args *A, params *ProcessParams, batch arrow.RecordBatch) (arrow.RecordBatch, error)
}

Description

TypedScalarFunc is the declarative variant of ScalarFunction. The argument schema is described once by the type parameter A — a struct whose vgi:"..." tags drive both ArgumentSpecs (via DeriveArgSpecs) and per-call argument binding (via BindArgs). Use AsScalarFunction to wrap an implementation for registration with Worker.RegisterScalar.

Column arguments (vgi:"const=false,...") are left at zero on the bound struct — the function should read column values from the batch directly. Constant scalar arguments are populated.

struct columnFieldBinding

source

type columnFieldBinding struct {
FieldIndex  int
FieldType  reflect.Type
Varargs    bool  // true when the field is []arrow.Array
}

Description

columnFieldBinding describes one struct field that maps to an input batch column (rather than a const scalar argument).

Methods

function columnFieldBindings

source

func columnFieldBindings(t reflect.Type) ([]columnFieldBinding, error)

columnFieldBindings returns the subset of fieldBinding entries that refer to column args (IsConst=false) whose Go type the binder can populate:

• the arrow.Array interface (catch-all, polymorphic)
• any concrete *array.X type that implements arrow.Array (strict)
• []arrow.Array for varargs columns

Other column-arg field types are silently ignored — BindArgs leaves them at the zero value.

struct typedScalarAdapter

source

type typedScalarAdapter[A any] struct {
inner    TypedScalarFunc[A]
specs    []ArgSpec
columnBindings  []columnFieldBinding
}

Methods

method ArgumentSpecs

source

func (a *typedScalarAdapter[A]) ArgumentSpecs() []ArgSpec

ArgumentSpecs returns the argument specs derived once from A’s struct tags.

method Metadata

source

func (a *typedScalarAdapter[A]) Metadata() FunctionMetadata

Metadata forwards to the wrapped typed function’s Metadata.

method Name

source

func (a *typedScalarAdapter[A]) Name() string

Name forwards to the wrapped typed function’s Name.

method OnBind

source

func (a *typedScalarAdapter[A]) OnBind(params *BindParams) (*BindResponse, error)

OnBind binds the argument struct A from params.Args and forwards to the wrapped typed function’s OnBindTyped.

method Process

source

func (a *typedScalarAdapter[A]) Process(ctx context.Context, params *ProcessParams, batch arrow.RecordBatch) (arrow.RecordBatch, error)

Process binds the argument struct A from params.Args, populates its column-arg fields from the input batch, and forwards to the wrapped typed function’s ProcessTyped.

method bindColumnArgs

source

func (a *typedScalarAdapter[A]) bindColumnArgs(target *A, batch arrow.RecordBatch)

bindColumnArgs populates column-arg fields of args from batch columns in declaration order. Field-type-vs-column-type mismatches are impossible if ValidateTypeBounds at bind time agrees with the field’s declared type; any such mismatch panics through reflect.Set and is caught by the dispatch recovery in protocol.go, surfacing as a WorkerPanicError.

function AsTyped

source

func AsTyped[T any](col arrow.Array) (T, bool)

AsTyped safely casts an arrow.Array to a specific concrete type. Returns (zero, false) if the cast fails. Use MustTyped for error-returning variant.

function BuildResultBatch

source

func BuildResultBatch(params *ProcessParams, resultArr arrow.Array, numRows int64) arrow.RecordBatch

BuildResultBatch wraps a single result array into a RecordBatch using the output schema from ProcessParams. This is a low-level building block for Process implementations that need manual builder control (e.g., complex binary construction) but still want to avoid the batch-wrapping boilerplate.

function CommonTypeForAddition

source

func CommonTypeForAddition(dt1, dt2 arrow.DataType) arrow.DataType

CommonTypeForAddition determines the output type when adding two numeric types. If either type is floating point, the result is float64. For two decimals the common type widens to the max precision AND max scale of the inputs (so adding DECIMAL(5,2) + DECIMAL(7,3) yields a scale-3 result); otherwise the wider integer type is promoted. The common type is then promoted for overflow headroom via PromoteForAddition.

function Float64Accessor

source

func Float64Accessor(col arrow.Array) func(i int) float64

Float64Accessor is the hoisted form of GetFloat64Value (see Int64Accessor). For decimal columns it also captures the scale once, avoiding the per-row DataType() assertion that GetFloat64Value repeats.

function GenerateColumn

source

func GenerateColumn[T any, B ArrayBuilder[T]](
params *ProcessParams,
batch arrow.RecordBatch,
newBuilder func(memory.Allocator) B,
generateFn func(i int) T,
) (arrow.RecordBatch, error)

GenerateColumn creates an output column by calling generateFn for each row. No input columns are read; batch is used only for its row count. This is for generators and constant-per-row functions.

function GetFloat64Value

source

func GetFloat64Value(col arrow.Array, i int) float64

GetFloat64Value extracts a float64 from any numeric column type. Large int64/uint64 values may lose precision when converted to float64.

function GetInt64Value

source

func GetInt64Value(col arrow.Array, i int) int64

GetInt64Value extracts an int64 from any integer column type. For uint64 values exceeding math.MaxInt64, the result wraps to negative.

function GetStringValue

source

func GetStringValue(col arrow.Array, i int) string

GetStringValue extracts a string from a String or Dictionary column.

function Int64Accessor

source

func Int64Accessor(col arrow.Array) func(i int) int64

Int64Accessor resolves the concrete column type once and returns a per-row accessor over it. It is the hoisted form of GetInt64Value: call it once before a per-row loop and invoke the returned closure inside the loop, so the type switch runs once per column instead of once per row (and the closure closes over the concrete array, letting the compiler inline Value(i)). Unsupported column types yield an accessor that returns 0, mirroring GetInt64Value’s default.

function IsAddableType

source

func IsAddableType(dt arrow.DataType) bool

IsAddableType checks if an Arrow type supports addition operations. Matches Python’s _is_addable_type: integer, floating, decimal, or temporal.

function IsDecimalType

source

func IsDecimalType(dt arrow.DataType) bool

IsDecimalType checks if an Arrow type is a decimal type.

function IsFloatingType

source

func IsFloatingType(dt arrow.DataType) bool

IsFloatingType checks if an Arrow type is a floating point type.

function IsIntegerType

source

func IsIntegerType(dt arrow.DataType) bool

IsIntegerType checks if an Arrow type is an integer type.

function IsMultipliableType

source

func IsMultipliableType(dt arrow.DataType) bool

IsMultipliableType checks if an Arrow type supports multiplication. Matches Python’s _is_multipliable_type: integer, floating, or decimal. Temporal types are excluded — doubling a date is not well-defined.

function IsNumericType

source

func IsNumericType(dt arrow.DataType) bool

IsNumericType checks if an Arrow type is numeric (integer or floating point).

function IsTemporalType

source

func IsTemporalType(dt arrow.DataType) bool

IsTemporalType checks if an Arrow type is a temporal type (date, time, timestamp, duration, or interval).

function MapAllColumns

source

func MapAllColumns[T any, B ArrayBuilder[T]](
params *ProcessParams,
batch arrow.RecordBatch,
newBuilder func(memory.Allocator) B,
transform func(cols []arrow.Array, i int) T,
) (arrow.RecordBatch, error)

MapAllColumns maps all input columns to a single output column. If any column is null for a given row, the output is null.

function MapColumn

source

func MapColumn[T any, B ArrayBuilder[T]](
params *ProcessParams,
batch arrow.RecordBatch,
colIndex int,
newBuilder func(memory.Allocator) B,
transform func(col arrow.Array, i int) T,
) (arrow.RecordBatch, error)

MapColumn maps a single input column to an output column. Null inputs are propagated as null outputs (DEFAULT null handling).

function MapColumnCustomNulls

source

func MapColumnCustomNulls[T any, B ArrayBuilder[T]](
params *ProcessParams,
batch arrow.RecordBatch,
colIndex int,
newBuilder func(memory.Allocator) B,
transform func(col arrow.Array, i int) T,
) (arrow.RecordBatch, error)

MapColumnCustomNulls maps a single input column to an output column without automatic null propagation. The transform is called for every row including nulls, giving the author full control over null handling.

function MapColumns

source

func MapColumns[T any, B ArrayBuilder[T]](
params *ProcessParams,
batch arrow.RecordBatch,
colIndices []int,
newBuilder func(memory.Allocator) B,
transform func(cols []arrow.Array, i int) T,
) (arrow.RecordBatch, error)

MapColumns maps multiple input columns to a single output column. If any input column is null for a given row, the output is null.

function MustTyped

source

func MustTyped[T any](col arrow.Array) (T, error)

MustTyped safely casts an arrow.Array to a specific concrete type, returning an error if the cast fails.

function NumericDispatch

source

func NumericDispatch(
params *ProcessParams,
batch arrow.RecordBatch,
intFn func(cols []arrow.Array, i int) int64,
floatFn func(cols []arrow.Array, i int) float64,
) (arrow.RecordBatch, error)

NumericDispatch creates a result batch by dispatching to the appropriate numeric builder based on the output schema type. For integer output types, intFn is called; for floating point types, floatFn is called. Null propagation is handled automatically across all input columns.

Supported output types: INT8 through UINT64, FLOAT32, FLOAT64. FLOAT16 is not supported. The int64/float64 results from intFn/floatFn are cast to the output type; callers are responsible for ensuring values fit (overflow silently truncates, matching Go’s standard integer conversion behavior).

function NumericTypeSize

source

func NumericTypeSize(dt arrow.DataType) int

NumericTypeSize returns the byte size of a numeric type for comparison.

function PromoteForAddition

source

func PromoteForAddition(dt arrow.DataType) arrow.DataType

PromoteForAddition returns the promoted output type for addition operations. Integer types promote to the next wider type; floating point types stay the same; decimal128 types add one digit of precision (capped at 38, decimal128’s limit; values that overflow at the cap fault at compute time).

function StringAccessor

source

func StringAccessor(col arrow.Array) func(i int) string

StringAccessor is the hoisted form of GetStringValue (see Int64Accessor). For a dictionary column it captures the decoded dictionary once.

function numericBuild

source

func numericBuild[T any, B ArrayBuilder[T]](
builder B,
cols []arrow.Array,
n int,
anyNull func(int) bool,
transform func([]arrow.Array, int) T,
) arrow.Array

numericBuild is a generic helper for NumericDispatch that builds an array with null propagation.

function rescaleDecimal128

source

func rescaleDecimal128(v decimal128.Num, fromScale, toScale int32) decimal128.Num

rescaleDecimal128 aligns a decimal128 raw value from fromScale to toScale. The output scale is the max of the inputs’ scales (see CommonTypeForAddition), so toScale >= fromScale and only a scale increase is ever required.