vgi.scalar_function

Module overview

Scalar functions: per-row transforms with single-column output.

Scalar functions are the simplest function type in VGI. They transform each input row into exactly one output value, producing a single column of results.

Key characteristics:

• 1:1 row mapping: Output has exactly the same number of rows as input
• Single column output: Output schema has exactly one column named “result”
• No finish(): Processing ends when the caller closes the input stream.

Common use cases:

• Mathematical operations: multiply, add, abs
• String transforms: upper, lower, concat, trim
• Type conversions: cast, parse
• Field extraction: get nested values, parse JSON fields

This module provides two base classes:

ScalarFunction (recommended)

Declarative API using Param/ConstParam/Returns annotations on compute(). Also supports Setting, Secret, and OutputLength annotations. Override output_type() only if the output type depends on input schema.

ScalarFunctionGenerator (advanced)

Per-batch callback API for fine-grained control. Override output_type() and process().

class BindParameters

source

Description

Parameters passed to a scalar function’s bind() method.

Attributes

attribute constant_arguments

Arguments

Constant arguments provided at query planning time.

attribute arguments_schema

pa.Schema

Schema describing the input columns.

attribute settings

pa.RecordBatch | None

DuckDB settings as a single-row RecordBatch, or None.

attribute secrets

SecretsAccessor

SecretsAccessor for accessing resolved and dynamic secrets.

attribute auth_context

AuthContext

Authentication context for the current request.

attribute attach_opaque_data

bytes | None

Catalog attach ID, if the function was invoked through an ATTACHed catalog.

attribute transaction_opaque_data

bytes | None

Catalog transaction ID, if invoked inside a catalog transaction.

class BindResult

source

Bases: ArrowSerializableDataclass

Description

Result of calling bind() on a scalar function.

Unlike table functions which return a full schema, scalar functions return a single output type since they produce one value per row.

Attributes

attribute output_type

pa.DataType

Arrow data type for the output value.

attribute opaque_data

ArrowSerializableDataclass | None

Optional serialized data, opaque to the caller, that will be passed to global_init() and process().

class RowCountMismatchError

source

Bases: Exception

Description

Raised when scalar function output row count doesn’t match input.

Scalar functions must produce exactly one output row for each input row. This error indicates the compute() method returned an array with the wrong number of elements.

Attributes

attribute input_rows

int | None

Number of rows in the input batch.

attribute output_rows

int | None

Number of rows in the output batch.

attribute function_name

str

Name of the function that produced the mismatch.

class ScalarFunction

source

Bases: ScalarFunctionGenerator

Description

Base class for scalar functions (1:1 row mapping, single output column).

Scalar functions transform each input row to exactly one output value. Use Param/ConstParam/Returns annotations on compute() to declare types.

Type Validation

Input and output types are validated at runtime:

• Param types are checked against actual array types
• Returns type is checked against compute() result
• AnyArrow parameters skip validation
• TypeMismatchError is raised on mismatch

Methods to Override

compute(self, …) -> pa.Array

Transform input arrays to output. Use Param/ConstParam annotations.

output_type(params) -> pa.DataType (classmethod)

Override when output type depends on input schema or arguments.

Methods

method catalog_output_schema

source

catalog_output_schema() -> pa.Schema

Return output schema for catalog introspection.

Returns the output schema with a single “result” field using the type from the Returns() annotation. If no explicit type was declared (dynamic type), returns null() with metadata indicating “any” type.

method output_type

source

output_type(params: BindParameters) -> pa.DataType

Return the Arrow type for the output column.

Default implementation uses _returns_output_type from Returns() annotation. Override when the output type depends on input schema or arguments (use params.arguments_schema, params.constant_arguments).

Parameters

params

Bind parameters including arguments and input schema.

Returns

The Arrow DataType of the function’s output column.

method on_bind

source

on_bind(params: BindParameters) -> BindResult

Produce the output type during the bind phase.

Override to perform custom bind-time logic such as validating arguments, examining input schema, or computing a dynamic output type.

Note

Constant arguments needed during process() are automatically serialized by the protocol. The opaque_data field is for additional bind-time state you need to pass forward.

Parameters

params

Bind parameters including arguments, input schema, settings, and secrets.

Returns

BindResult with output_type and optional opaque_data.

method process

source

process(
*,
batch: pa.RecordBatch,
init_call: InitRequest,
init_response: BaseInitResponse,
storage: BoundStorage,
auth_context: AuthContext,
) -> pa.RecordBatch

Convert compute() to per-batch callback.

This method calls your compute() method for the input batch. Keyword-only parameters in compute() are automatically populated from the batch columns.

Inherited members (8)

• get_metadata method · from MetadataMixin — Get the resolved metadata for this function class.
• describe method · from MetadataMixin — Get metadata as a dictionary (for JSON serialization).
• logger attribute · from Function
• storage attribute · from Function
• CACHE_CONTROL attribute · from ScalarFunctionGenerator
• bind method · from ScalarFunctionGenerator — Bind protocol entry point. Do not override; use on_bind() instead.
• on_init method · from ScalarFunctionGenerator — Initialize the function during the init API call.
• global_init method · from ScalarFunctionGenerator — Global init protocol entry point. Do not override; use on_init() instead.

class ScalarFunctionGenerator

source

Bases: vgi.function.Function

Description

Per-batch callback base class for scalar functions.

This is the advanced API for scalar functions. For most use cases, use ScalarFunction instead, which provides a simpler compute() callback.

Scalar functions have these constraints:

• 1:1 row mapping: Output row count must equal input row count
• Single value output: Produces one value per input row
• No finalization: Processing ends when input is exhausted

Methods to Override

output_type(params) -> pa.DataType

Return the Arrow type for the output value. Required.

process(…) -> pa.RecordBatch

Process one input batch. Must return output with same row count. Required.

on_bind(params) -> BindResult

Optional. Override to perform custom bind-time logic.

on_init(…) -> GlobalInitResponse

Optional. Override to perform custom initialization.

Protocol Entry Points (called by worker, do not override)

bind(input) -> BindResponse

Handles the bind API call

global_init(input) -> GlobalInitResponse

Handles the global_init API call

Attributes

attribute CACHE_CONTROL

CacheControl | None

Opt into the extension’s result cache: when set, this CacheControl’s vgi.cache.* metadata is attached to every output batch, so the C++ side memoizes the scalar’s output per distinct input value (see docs/exchange_dedup_pervalue.md). A pure, deterministic scalar only — advertising this on a non-pure scalar serves stale rows.

Methods

method output_type

source

output_type(params: BindParameters) -> pa.DataType

Return the Arrow type for the output value.

Parameters

params

Bind parameters including arguments and input schema.

Returns

The Arrow DataType of the function’s output column.

method on_bind

source

on_bind(params: BindParameters) -> BindResult

Produce the output type during the bind API call.

Override to perform custom bind-time logic such as validating arguments or computing a dynamic output type.

Parameters

params

Bind parameters including arguments and schema.

Returns

BindResult with output_type and optional opaque_data.

method catalog_output_schema

source

catalog_output_schema() -> pa.Schema

Return output schema for catalog introspection.

A generator-style scalar function computes its output type at bind time, so no static type is known here. Report a single dynamic result column (null() tagged vgi:any) so catalog consumers treat the type as resolved-at-bind. ScalarFunction overrides this when a static Returns() type is available.

method bind

source

bind(
input: BindRequest,
*,
ctx: CallContext | None = None,
attach_plaintext: bytes | None = None,
) -> BindResponse

Bind protocol entry point. Do not override; use on_bind() instead.

Constructs BindParameters, validates type bounds, calls on_bind(), and wraps the result for transmission to global_init. If on_bind() triggers dynamic secret lookups or if compute() declares Secret() annotations that haven’t been resolved, returns a secret scope request.

method on_init

source

on_init(
*,
bind_call: BindRequest,
opaque_data: bytes | None,
storage: BoundStorage,
) -> GlobalInitResponse

Initialize the function during the init API call.

Override to perform one-time setup that should happen after bind but before processing batches. The default returns max_processes=1.

Parameters

bind_call

The original BindCall with arguments and schema.

opaque_data

Bytes from on_bind()’s BindResult.opaque_data (after the framework’s serialize-to-bytes shim), or None if on_bind didn’t set it. Reconstruct via MyConcreteDataclass.deserialize_from_bytes(opaque_data) — the consumer always knows what concrete type to expect, so explicit reconstruction is preferred over a framework- level class-name registry.

storage

BoundStorage for storing data across calls.

Returns

GlobalInitResponse with max_processes and optional opaque data.

method global_init

source

global_init(
input: InitRequest,
*,
attach_plaintext: bytes | None = None,
) -> GlobalInitResponse

Global init protocol entry point. Do not override; use on_init() instead.

Deserializes the wrapped bind data, calls on_init(), and wraps the result for transmission to process().

attach_plaintext is the full framework plaintext (uuid||catalog bytes) the worker unwrapped; storage shards on its UUID. Scalar on_init does not expose the attach to bodies, so only storage uses it.

method process

source

process(
*,
batch: pa.RecordBatch,
init_call: InitRequest,
init_response: BaseInitResponse,
storage: BoundStorage,
auth_context: AuthContext,
) -> pa.RecordBatch

Process one input batch.

Override this method to implement your scalar transformation. Must return an output RecordBatch with exactly the same number of rows as the input batch.

Parameters

batch

The input RecordBatch to process.

init_call

The parameters from global_init.

init_response

The response from the init call.

storage

BoundStorage for storing data across calls.

auth_context

Authentication context for the current request.

Returns

Output RecordBatch with same row count as input.

Inherited members (4)

• get_metadata method · from MetadataMixin — Get the resolved metadata for this function class.
• describe method · from MetadataMixin — Get metadata as a dictionary (for JSON serialization).
• logger attribute · from Function
• storage attribute · from Function

class TypeMismatchError

source

Bases: TypeError

Description

Raised when array type doesn’t match declared parameter or return type.

This error indicates a mismatch between the declared type in Param() or Returns() and the actual array type at runtime.

Attributes

attribute param_name

str

Name of the parameter with the type mismatch.

attribute expected_type

pa.DataType | None

The declared Arrow type.

attribute actual_type

pa.DataType | None

The actual Arrow type found.

attribute function_name

str

Name of the function class.