Argument serialization

How VGI turns a function’s argument specifications into Apache Arrow schemas for IPC transmission and DuckDB function registration — the metadata keys, field ordering, and parse rules.

This is the wire format, so every SDK produces and parses the same schema; a Go worker and a Python worker advertising the same signature are byte-identical here. You don’t need any of it to write a worker — declare arguments in your language’s idiom (Python Annotated[...], Go vgi:"..." struct tags) and the SDK handles the rest. The examples below are Python.

Quick reference

Metadata Key	Value	Meaning
vgi_arg	named	Named argument (not positional)
vgi_type	table	Table input argument
vgi_type	any	Any Arrow type argument
vgi_varargs	true	Variable arguments

Schema format

Arguments are serialized as a single Arrow schema where each field represents one argument.

Field order

1. Positional arguments come first, in order (field index = position index)
2. Named arguments follow, marked with metadata

Field components

Component	Source
Field name	Python attribute name
Field type	Exact Arrow data type
Field metadata	Markers for named, table, any, varargs

Positional arguments

Positional arguments have no special metadata. Their position index is determined by their order in the schema.

Copy Copied!

# count: Annotated[int, Arg(0)] becomes:
pa.field("count", pa.int64())

# name: Annotated[str, Arg(1)] becomes:
pa.field("name", pa.utf8())

Named arguments

Named arguments have vgi_arg=named metadata. The field name is the argument key used in SQL.

Copy Copied!

# format: Annotated[str, Arg("format")] becomes:
pa.field("format", pa.utf8(), metadata={b"vgi_arg": b"named"})

Special types

Table input

Table input arguments (Annotated[TableInput, Arg(...)]) receive streaming RecordBatches rather than scalar values.

• Arrow type: pa.null()
• Metadata: {b"vgi_type": b"table"}

Copy Copied!

# data: Annotated[TableInput, Arg(1)] becomes:
pa.field("data", pa.null(), metadata={b"vgi_type": b"table"})

Any type

Any-type arguments (Annotated[AnyArrowValue, Arg(...)]) accept any valid Arrow scalar type at runtime.

• Arrow type: pa.null()
• Metadata: {b"vgi_type": b"any"}

Copy Copied!

# value: Annotated[AnyArrowValue, Arg(0)] becomes:
pa.field("value", pa.null(), metadata={b"vgi_type": b"any"})

Variable arguments

Varargs arguments (varargs=True) collect all remaining positional arguments from their position onwards.

• Arrow type: The element type (e.g., pa.int64() for int varargs)
• Metadata: {b"vgi_varargs": b"true"}

Copy Copied!

# columns: Annotated[tuple[str, ...], Arg(0, varargs=True)] becomes:
pa.field("columns", pa.utf8(), metadata={b"vgi_varargs": b"true"})

Union types

DuckDB UNION / Arrow union arguments are tagged: the discriminator — which member is present — lives in the Arrow UnionScalar.type_code, not in the member value. A plain Scalar.as_py() returns only the member value and drops the tag, which loses exactly the information a union argument exists to carry.

Since 0.8.3, union arguments decode into a TaggedUnion instead, preserving both halves. Table varargs decode the same way since 0.9.0.

Copy Copied!

config: Annotated[TaggedUnion, Arg("config", arrow_type=pa.sparse_union([...]))]
...
cfg = params.args.config          # TaggedUnion(tag=..., value=...)
if cfg.tag == "random_forest_classifier":
  grid = cfg.value              # the member struct, as a dict

tag is the active member’s field name (or None) and value is its Python value. Every non-union type is unchanged — TaggedUnion appears only where a union is declared.

Combined metadata

Fields can have multiple metadata keys. For example, a named argument that accepts any type:

Copy Copied!

# threshold: Annotated[AnyArrowValue, Arg("threshold")] becomes:
pa.field("threshold", pa.null(), metadata={
  b"vgi_arg": b"named",
  b"vgi_type": b"any",
})

Complete examples

Example 1: a simple function

Copy Copied!

from typing import Annotated

class MyFunction(TableInOutFunction):
  count: Annotated[int, Arg(0)]           # Positional 0
  name: Annotated[str, Arg(1)]            # Positional 1
  verbose: Annotated[bool, Arg("verbose")] # Named

# Serializes to:
schema = pa.schema([
  pa.field("count", pa.int64()),
  pa.field("name", pa.utf8()),
  pa.field("verbose", pa.bool_(), metadata={b"vgi_arg": b"named"}),
])

Example 2: a function with table input

Copy Copied!

from typing import Annotated
from vgi.arguments import TableInput

class TransformFunction(TableInOutFunction):
  multiplier: Annotated[float, Arg(0)]
  data: Annotated[TableInput, Arg(1)]

# Serializes to:
schema = pa.schema([
  pa.field("multiplier", pa.float64()),
  pa.field("data", pa.null(), metadata={b"vgi_type": b"table"}),
])

Example 3: a function with varargs

Copy Copied!

from typing import Annotated

class SumValuesFunction(TableInOutFunction):
  columns: Annotated[tuple[str, ...], Arg(0, varargs=True)]

# Serializes to:
schema = pa.schema([
  pa.field("columns", pa.utf8(), metadata={b"vgi_varargs": b"true"}),
])

Example 4: a complex function

Copy Copied!

from typing import Annotated
from vgi import AnyArrowValue
from vgi.arguments import TableInput

class ComplexFunction(TableInOutFunction):
  count: Annotated[int, Arg(0)]
  data: Annotated[TableInput, Arg(1)]
  extra: Annotated[tuple[float, ...], Arg(2, varargs=True)]
  format: Annotated[str, Arg("format")]
  threshold: Annotated[AnyArrowValue, Arg("threshold")]

# Serializes to:
schema = pa.schema([
  pa.field("count", pa.int64()),
  pa.field("data", pa.null(), metadata={b"vgi_type": b"table"}),
  pa.field("extra", pa.float64(), metadata={b"vgi_varargs": b"true"}),
  pa.field("format", pa.utf8(), metadata={b"vgi_arg": b"named"}),
  pa.field("threshold", pa.null(), metadata={
      b"vgi_arg": b"named",
      b"vgi_type": b"any",
  }),
])

Serialization code

Serialize to bytes

Copy Copied!

from vgi.argument_spec import argument_specs_to_schema

# Create schema from specs
schema = argument_specs_to_schema(specs)

# Serialize to bytes
schema_bytes = schema.serialize().to_pybytes()

Deserialize from bytes

Copy Copied!

import pyarrow as pa
from vgi.argument_spec import schema_to_argument_specs

# Deserialize schema
schema = pa.ipc.read_schema(pa.py_buffer(schema_bytes))

# Convert to ArgumentSpec objects
specs = schema_to_argument_specs(schema)

Parsing algorithm

To parse a schema back to argument specifications:

1. Initialize position_index = 0
2. For each field in schema:
   • Check if field has vgi_arg=named metadata
   • If named: position = field.name (string)
   • If positional: position = position_index, then increment position_index
   • Check for vgi_type metadata (table or any)
   • Check for vgi_varargs metadata
   • Create ArgumentSpec with extracted info

Copy Copied!

def parse_schema(schema):
  specs = []
  position_index = 0

  for field in schema:
      metadata = field.metadata or {}

      # Determine position type
      if metadata.get(b"vgi_arg") == b"named":
          position = field.name  # Named argument
      else:
          position = position_index  # Positional argument
          position_index += 1

      # Check special types
      vgi_type = metadata.get(b"vgi_type")
      is_table_input = (vgi_type == b"table")
      is_any_type = (vgi_type == b"any")
      is_varargs = (metadata.get(b"vgi_varargs") == b"true")

      specs.append(ArgumentSpec(
          name=field.name,
          position=position,
          arrow_type=field.type,
          is_table_input=is_table_input,
          is_any_type=is_any_type,
          is_varargs=is_varargs,
      ))

  return specs

Discovery metadata

Since 0.9.0 the per-argument documentation, and since 0.10.0 the declared constraints, ride on the same Arrow field metadata — so a client (or an agent introspecting the catalog) can describe an argument without a Python runtime. These keys are surfaced by the C++ vgi_function_arguments() diagnostic.

Key	Value	Source
vgi_doc	UTF-8 description	Arg(doc=…) / Param(doc=…)
vgi_default	JSON scalar	the arg’s default (named/optional args only)
vgi_choices	JSON array	choices= — the closed set of allowed values
vgi_range	Interval notation, e.g. “[0, 100]”	built from ge/le/gt/lt
vgi_pattern	Raw regex	pattern= — an open set

The same vgi_doc key carries per-parameter documentation for macros (0.9.0+), so a macro’s parameters are described the same way a function’s are.

Const constraints are enforced; columnar ones are advisory

Since 0.10.0, choices/ge/le/gt/lt/pattern on a ConstParam are validated once at bind by validate_const_arg_constraints(), raising ArgumentValidationError — so a constraint you advertised is actually binding. Constraints on a columnar Param remain advisory: they describe the argument for discovery but are not checked per row.

Not included

Runtime-only concerns stay out of the schema — it carries the argument type specification needed for function registration, plus the discovery metadata above, and nothing else. Behaviour that depends on the call itself (coercion rules, row-level validation, the bind-time checks described above) lives in the Python function runtime.

Next steps

• Declare the arguments in the first place → Function API → Declaring value constraints.
• What else a function advertises → Function metadata.
• Exact types → API Reference: arguments.