vgi.arguments

Module overview

Argument parsing and validation for VGI functions.

This module provides classes for handling function arguments in VGI:

class AnyArrow

source

Description

Sentinel type for arguments accepting multiple Arrow types.

Use this with AnyArrowValue in the Annotated pattern when an argument should accept multiple valid Arrow types, validated via the type_bound parameter. When accessed, returns an AnyArrowValue containing the value plus metadata (position and name).

Choosing Between Specific Types and AnyArrowValue

• Single required type: Use Annotated[str, Arg(...)] or similar. The argument will only accept that exact type.

• Multiple valid types: Use Annotated[AnyArrowValue, Arg(...)] with type_bound to specify which types are acceptable. For example, numeric operations that work on integers, floats, and decimals should use AnyArrowValue.

The type_bound parameter is ONLY meaningful for AnyArrowValue arguments. Using it with other types will emit a warning.

Examples using Annotated (recommended):

from typing import Annotated
from vgi import Arg, AnyArrowValue

# Single type: function only works with strings
class UpperCaseFunction(TableFunctionGenerator):
  column: Annotated[str, Arg(0, doc="String column to uppercase")]

# Multiple types: function works with any numeric type
class DoubleFunction(TableFunctionGenerator):
  column: Annotated[
      AnyArrowValue,
      Arg(0, type_bound=[pa.types.is_integer, pa.types.is_floating])
  ]

  def on_bind(self) -> None:
      # Access column metadata for dynamic output type
      self._output_type = self.column.value

# Any type: function works with all types
class IdentityFunction(TableFunctionGenerator):
  column: Annotated[AnyArrowValue, Arg(0, doc="Column to pass through")]

Accessing values

When using AnyArrowValue, access the value via the .value attribute:

val = self.column.value     # The column name as a string
pos = self.column.position  # The positional index

Note

Unlike TableInput, AnyArrow arguments have actual Arrow values - they are just not constrained to a specific Arrow type.

Attributes

attribute value

Any

The resolved Arrow value of the argument.

attribute position

int | str

The argument’s positional index or name used to resolve it.

attribute name

str

The argument’s name.

class AnyArrowValue

source

Description

Wrapper for AnyArrow argument values with metadata.

When an Arg returns an AnyArrow type, accessing the attribute returns an AnyArrowValue instead of just the raw value. This provides access to both the value and the argument’s position/name for schema lookups.

Example using Annotated (recommended):

from typing import Annotated

class MyFunction(TableFunctionGenerator):
  col1: Annotated[AnyArrowValue, Arg(0, doc="First column")]

  def on_bind(self) -> None:
      # self.col1 is an AnyArrowValue
      print(self.col1.value)     # The column name
      print(self.col1.position)  # The positional index

Example using legacy Arg[AnyArrow] syntax:

class MyFunction(TableFunctionGenerator):
  col1 = Arg[AnyArrow](0, doc="First column")  # type: ignore[assignment]

Attributes

attribute value

Any

The Python value (from scalar.as_py()).

attribute position

int | str

The positional index from the Arg definition (int for positional, str for named arguments).

attribute name

str

The Python attribute name of the Arg.

class Arg

source

Description

Descriptor for declarative argument parsing with optional validation.

Use as a class attribute to declare function arguments that are automatically parsed from self.arguments when accessed. This eliminates the need to override __init__ for simple argument parsing.

Note

For named arguments (string position), the Python attribute name should match the SQL key. This is the standard convention:

format = Arg[str]("format")  # Recommended: attribute == key

Avoid using different names:

output_format = Arg[str]("format")  # Not recommended

While this works at runtime, it can cause issues with metadata serialization where only one name is preserved.

Attributes

attribute position

int | str

Positional index (int) or named key (str).

attribute default

ArgT | Any

Default value if argument not provided. Omit for required arguments.

attribute doc

str

Documentation string for this argument.

attribute ge

float | int | None

Value must be >= this (for numeric types).

attribute le

float | int | None

Value must be <= this (for numeric types).

attribute gt

float | int | None

Value must be > this (for numeric types).

attribute lt

float | int | None

Value must be < this (for numeric types).

attribute choices

Sequence[ArgT] | None

Value must be one of these options.

attribute pattern

str | None

Value must match this regex pattern (for strings).

attribute varargs

If True, collect all remaining positional arguments from this position onwards. Returns tuple[ArgT, …]. Requires at least 1 value. Must be positional (not named).

attribute arrow_type

Explicit Arrow type for this argument. If not provided, type is inferred from the type hint using PYTHON_TO_ARROW.

attribute type_bound

Type predicate(s) for Arg[AnyArrow] column type validation. Accepts a single predicate (e.g., pa.types.is_integer) or a sequence of predicates where any match is valid (OR logic). Only meaningful for Arg[AnyArrow] arguments; issues a warning if used with other types.

attribute const

If True, marks this argument as constant-folded (ConstParam). Constant arguments have their values known at planning time.

attribute is_any

If True, indicates this argument accepts any Arrow type (AnyArrow). Used for tracking when AnyArrow was specified in the type hint.

Methods

method format_error

source

format_error(message: str) -> str

Format an error message with argument context.

Use this method when performing custom validation to produce error messages that include the argument’s position and name.

Parameters

message

The error message describing what went wrong.

Returns

Formatted error message prefixed with argument context.

method validate_type_bound

source

validate_type_bound(field_type: pa.DataType) -> None

Validate that the field type satisfies the type bound predicate(s).

This method is called during function initialization for Arg[AnyArrow] arguments that have type_bound specified.

If multiple predicates are provided, uses OR logic (any match is valid).

Parameters

field_type

The Arrow type of the column to validate.

Raises

SchemaValidationError

If the type bound is not satisfied.

class Arguments

source

Description

Container for function arguments.

Access arguments using get() for Python values:

# Positional arguments (by index)
count = args.get(0)                      # First argument
name = args.get(1, default="unnamed")    # With default

# Named arguments (by string)
separator = args.get("sep", default=",")
threshold = args.get("threshold")

# With type validation (optional, for strict checking)
count = args.get(0, type=pa.int64())

For direct Arrow Scalar access, use positional/named attributes:

scalar = args.positional[0]              # pa.Scalar | None
scalar = args.named["sep"]               # pa.Scalar

Attributes

attribute positional

tuple[Scalar[Any] | None, …]

Tuple of positional argument values as pa.Scalar.

attribute named

dict[str, Scalar[Any]] | None

Dictionary mapping argument names to pa.Scalar values.

Methods

method get

source

get(
key: int | str,
*,
type: pa.DataType | None = None,
default: Any = _MISSING,
) -> Any

Get argument as Python value.

SQL NULL is a real value, distinct from “argument not provided”. default is consulted only when the caller omitted the argument entirely; an explicit SQL NULL returns None.

Parameters

key

Positional index (int) or argument name (str).

type

Expected Arrow type. Raises TypeError if mismatch.

default

Value to return if argument is omitted (not provided by the caller). If not provided, raises an exception for missing args. default is not consulted for explicit SQL NULL — that case returns None.

Returns

The argument value as a Python object. None if the caller passed an explicit SQL NULL.

Raises

IndexError

Positional argument not provided (no default).

KeyError

Named argument not provided (no default).

TypeError

Argument type doesn’t match type parameter.

method get_varargs

source

get_varargs(
start: int,
*,
type: pa.DataType | None = None,
) -> tuple[Any, …]

Get all positional arguments from start position onwards.

Parameters

start

Starting positional index (inclusive).

type

Expected Arrow type for all values. Raises TypeError if mismatch.

Returns

Tuple of argument values as Python objects.

method encoded_dict

source

encoded_dict() -> dict[str, Scalar[Any] | None]

Convert arguments to a dictionary suitable for serialization.

Positional arguments are stored with keys “positional_0”, “positional_1”, etc. Named arguments are stored with their actual names prefixed by “named_”.

The reason why a dictionary is used is to facilitate serialization with Arrow, which can easily handle flat structures, but doesn’t handle variable typed arrays of arbitrary objects.

Returns

Dictionary mapping argument names to their values.

method schema

source

schema() -> pa.Schema

Return Arrow schema for serializing these Arguments.

Creates a schema with one field per argument: “positional_0”, “positional_1”, etc. for positional args, and “named_<name>” for named args. Field types are taken directly from scalar values to handle Arrow extension types.

Returns

Arrow schema matching the structure returned by encoded_dict().

method decode

source

decode(data: pa.StructScalar) -> Arguments

Decode Arguments from a serialized dictionary.

Parameters

data

Dictionary containing serialized argument fields.

Returns

Deserialized Arguments instance.

method serialize_to_bytes

source

serialize_to_bytes() -> bytes

Serialize Arguments to bytes using Arrow IPC format.

Creates a single-row RecordBatch with the arguments encoded as a struct column, then serializes it to IPC stream bytes.

Builds the batch with explicit types from scalar values to handle Arrow extension types (e.g., HUGEINT) that from_pylist() cannot infer.

Returns

Serialized bytes containing the Arguments.

method deserialize_from_bytes

source

deserialize_from_bytes(
data: bytes,
ipc_validation: Any = None,
) -> Arguments

Deserialize Arguments from bytes.

Parameters

data

Bytes serialized via serialize_to_bytes().

ipc_validation

Unused, accepted for compatibility with ArrowSerializableDataclass._convert_value_for_deserialization.

Returns

Deserialized Arguments instance.

class ArgumentValidationError

source

Bases: ValueError

Description

Raised when an argument fails validation.

This exception provides detailed context about what went wrong and suggests how to fix the issue.

Attributes

attribute arg_name

str | None

Name of the argument that failed validation.

attribute position

int | str | None

Positional index or named key of the argument.

attribute value

Any

The invalid value that was provided.

attribute constraint

str | None

Description of the constraint that was violated.

attribute doc

str | None

Documentation string for the argument (if provided).

attribute valid_range

str | None

Human-readable description of valid values.

attribute default

Any

Default value (if any) that could be used instead.

attribute choices

Sequence[Any] | None

Valid choices, if the argument is constrained to a set.

class Auth

source

Description

Metadata for auth context parameter in compute().

Use with Annotated to declare a parameter that receives the AuthContext for the current request. Returns AuthContext.anonymous() when no authentication is configured (including stdio transport).

class ConstParam

source

Description

Metadata for constant scalar parameters in compute().

Use with Annotated to declare parameters that receive constant (non-columnar) values known at planning time. The type is inferred from the Annotated first argument (e.g., Annotated[int, ConstParam(...)] infers pa.int64()).

Attributes

attribute doc

str

Documentation string describing this parameter.

attribute arrow_type

pa.DataType | type | None

Optional explicit Arrow type. If not provided, type is inferred from the Annotated first argument.

attribute position

int | None

Position in the argument list (optional for ScalarFunction where position is inferred from signature).

attribute phase

str

Phase when this const param is needed (aggregate functions only). "all" = every callback, "update" = only update, "finalize" = only finalize.

attribute choices

Sequence[Any] | None

Closed set of allowed values. Surfaced for agent discovery (via vgi_function_arguments()) AND enforced at bind: a const value outside the set raises ArgumentValidationError.

attribute ge

float | int | None

Value must be >= this (inclusive lower bound); enforced at bind.

attribute le

float | int | None

Value must be <= this (inclusive upper bound); enforced at bind.

attribute gt

float | int | None

Value must be > this (exclusive lower bound); enforced at bind.

attribute lt

float | int | None

Value must be < this (exclusive upper bound); enforced at bind.

attribute pattern

str | None

Regex the value must match (string params); enforced at bind.

class OutputLength

source

Description

Metadata for output length parameter in compute().

Use with Annotated to declare a parameter that receives the number of rows in the input batch. This is useful for scalar functions that don’t take any column arguments but need to know how many output values to produce.

class Param

source

Description

Metadata for columnar parameters in compute() or class-level declarations.

Use with Annotated to declare parameters that receive pa.Array values at runtime. The type information is used for catalog registration and argument validation.

For ScalarFunction compute() methods, position is inferred from parameter order.

Example (ScalarFunction compute() - position inferred):

class AddColumns(ScalarFunction):
  @classmethod
  def compute(
      cls,
      left: Annotated[pa.Array, Param(pa.int64(), "First value")],
      right: Annotated[pa.Array, Param(pa.int64(), "Second value")],
  ) -> Annotated[pa.Array, Returns(pa.int64())]:
      return pc.add(left, right)

Example (AnyArrow with type_bound):

class Double(ScalarFunction):
  @classmethod
  def compute(
      cls,
      value: Annotated[pa.Array, Param(doc="Numeric value",
                                        type_bound=pa.types.is_numeric)],
  ) -> Annotated[pa.Array, Returns()]:
      return pc.multiply(value, 2)

Attributes

attribute arrow_type

pa.DataType | type | None

The Arrow data type, Python type (int/str/float/bool/bytes), or None for AnyArrow (accepts any type).

attribute doc

str

Documentation string describing this parameter.

attribute type_bound

TypeBoundPredicate | Sequence[TypeBoundPredicate] | None

Type predicate(s) for validating input column types. Only meaningful when arrow_type is None (AnyArrow).

attribute varargs

bool

If True, this parameter collects all remaining positional arguments as a list of arrays.

attribute position

int | None

Explicit column position (for class-level attributes). None means position is inferred from method signature order.

attribute choices

Sequence[Any] | None

Closed set of allowed values, surfaced for agent discovery. Advisory for a columnar Param: it is published through vgi_function_arguments() but NOT enforced per row by the framework (validate column contents in compute() if required). On a scalar ConstParam the equivalent constraint IS enforced at bind.

attribute ge

float | int | None

Value must be >= this (inclusive lower bound). Advisory on Param (see choices).

attribute le

float | int | None

Value must be <= this (inclusive upper bound). Advisory on Param.

attribute gt

float | int | None

Value must be > this (exclusive lower bound). Advisory on Param.

attribute lt

float | int | None

Value must be < this (exclusive upper bound).

attribute pattern

str | None

Regex the value must match (for string parameters).

class Returns

source

Description

Metadata for compute() return type.

Use with Annotated to declare the output Arrow type for catalog registration. The annotation indicates that compute() returns a pa.Array of the specified type.

Attributes

attribute arrow_type

pa.DataType | None

The Arrow data type of the output, or None for AnyArrow (dynamic output type determined at bind time).

class Secret

source

Description

Metadata for secrets parameter in compute() or on_bind().

Use with Annotated to declare parameters that receive secret values from the DuckDB SecretManager. Secrets contain multiple key-value pairs where keys are strings and values can be any DuckDB type.

Attributes

attribute secret_type

str

The secret type to look up (e.g., “vgi_example”, “s3”). Required — C++ enforces type matching.

attribute name

str | None

Optional secret name for name-based lookup.

attribute scope

str | None

Optional static scope for pre-resolution (resolved before first bind call).

class SecretLookupEntry

source

Bases: ArrowSerializableDataclass

Description

A request to look up a specific secret.

Used both in function metadata (static requirements from annotations) and in runtime requests (dynamic scoped lookups). Also used directly as the catalog-level secret requirement type (replacing the former CatalogSecretRequirement which had identical fields).

Extends ArrowSerializableDataclass so it can be serialized in catalog FunctionInfo payloads.

secret_type is required — C++ enforces type matching.

Supported lookup patterns:

• By type only: SecretLookupEntry(secret_type=“s3”)
• By type + scope: SecretLookupEntry(secret_type=“s3”, scope=“s3://bucket/”)
• By type + name: SecretLookupEntry(secret_type=“s3”, secret_name=“my_cred”)
• By type + scope + name: all three fields set

Attributes

attribute secret_type

str

The DuckDB secret type to match (required; C++ enforces type matching).

attribute scope

str | None

Optional URI prefix the secret must apply to.

attribute secret_name

str | None

Optional name of the specific secret to resolve.

Methods

method to_dict

source

to_dict() -> dict[str, str | None]

Convert to dictionary for serialization.

method from_dict

source

from_dict(d: dict[str, Any]) -> SecretLookupEntry

Create from dictionary.

class Setting

source

Description

Metadata for settings parameter in compute().

Use with Annotated to declare parameters that receive setting values from the DuckDB session. Settings are string key-value pairs.

Attributes

attribute key

str | None

The setting key name. If not provided, uses the parameter name.

class TableInput

source

Description

Sentinel type for table input parameters in table-in-out functions.

Use this as the type parameter for Arg to declare which argument receives the streaming table input. Every TableInOutFunction must have exactly one TableInput argument, and it must be positional (not named).

The TableInput argument determines which table expression feeds the function when called from SQL. It doesn’t correspond to an actual Arrow value - the table data arrives as streaming RecordBatches via process().

class TaggedUnion

source

Description

A decoded union-typed argument: which member is set (tag) and its value.

DuckDB UNION / Arrow union arguments are tagged: the discriminator (which member is present) lives in the Arrow UnionScalar.type_code, not in the member value. Plain Scalar.as_py() returns only the member value and drops that tag, so union arguments are decoded into this wrapper instead — tag is the active member’s field name and value is its Python value.

Example:

config: Annotated[TaggedUnion, Arg("config", arrow_type=pa.sparse_union([...]))]
...
cfg = params.args.config            # TaggedUnion(tag=..., value=...)
if cfg.tag == "random_forest_classifier":
  grid = cfg.value                # the member struct, as a dict

Attributes

attribute tag

str | None

attribute value

Any

function validate_const_arg_constraints

source

validate_const_arg_constraints(
const_params: Mapping[str, Arg[Any]],
arguments: Arguments,
) -> None

Enforce const-argument value constraints at bind time.

Const arguments are bind-time scalars, so validate them once (not per batch / per group). Reuses Arg._validate, raising ArgumentValidationError for any value that violates a declared choices/ge/le/gt/lt/ pattern constraint — so a bad value fails fast at bind instead of silently reaching compute()/update(). Shared by the scalar and aggregate bind paths so both enforce identically (and match the legacy descriptor path).

Parameters

const_params

The function’s const parameters, keyed by name (each value an Arg carrying the constraints and its _resolution_index).

arguments

The bound call arguments; const scalars live in positional.

Raises

ArgumentValidationError

If a const value violates a declared constraint.