Function API

The deep reference for the callback APIs behind the scalar, table, and table-in-out shapes — the exact bind / init / process / finalize contracts, the simplified-vs-generator base classes, the lifecycle decorators, and the rules each shape places on its output schema.

DuckDB UDF API? VGI wraps DuckDB, but this is not DuckDB's Python UDF API

DuckDB sees registered SQL functions and table functions. The classes on this page are the VGI worker-side API: they describe how your Python process answers DuckDB’s calls over Arrow, not the native DuckDB Python client UDF interface.

Looking for a runnable worker?

Function patterns is the recipe — a complete, runnable worker for each of the five shapes. This page is the API reference behind those recipes; it documents method contracts and edge cases rather than repeating the templates. For how the callbacks fire across a query, see Function lifecycle, and for the generated signatures the scalar_function, table_function, and table_in_out_function API pages.

Most shapes ship two base classes: a declarative one (annotations on a single callback — the common case) and a generator one (a per-batch process() callback for full control). Reach for the generator variant only when you need custom per-batch init/response handling or to manage OutputCollector yourself.

Scalar functions

ScalarFunction (declarative) transforms each input row to exactly one output value. You write compute(), annotating inputs with Param/ConstParam and the return with Returns; the framework infers the output schema, validates types, and enforces the row-count contract. See Function patterns → Scalar for a runnable template.

Constraints (enforced at runtime)

Rule	Detail
Single-column output	The output schema has exactly one column, named result.
1:1 row mapping	Output num_rows must equal input num_rows — a mismatch raises RowCountMismatchError. (Filter or expand rows? Use a table or table-in-out function.)
Type match	Each Param array and the compute() result are checked against their declared types; a mismatch raises TypeMismatchError (castable types like int32→int64 are cast silently).
No finalize phase	Processing ends when the input stream closes — there is no per-call finalize.

Declaring value constraints

Param and ConstParam accept choices, ge, le, gt, lt and pattern (0.10.0+). They serve two purposes at once: they are encoded into the argument schema’s field metadata, so vgi_function_arguments() and any agent introspecting the catalog can describe the argument without running Python — and on a ConstParam they are enforced.

Copy Copied!

precision: Annotated[int, ConstParam(doc="Rounding precision", ge=0, le=10)]

A violating const value raises ArgumentValidationError once at bind, rather than reaching compute() / update(). The same check runs for the aggregate bind dispatcher. Constraints on a columnar Param stay advisory — they describe the argument but are not checked per row. See Argument serialization → Discovery metadata.

Typed const scalars

Since 0.8.10, a ConstParam annotated as an Arrow scalar type is delivered as a typed pyarrow.Scalar rather than a coerced Python value, so the declared type is what you receive.

Method contracts

Method	When to override	Default
compute(…)	Always — the per-row transform.	Required.
output_type(params)	Only when the output type depends on input schema/arguments.	Returns the static Returns() type.
on_bind(params)	Custom bind-time validation, dynamic types, secret resolution.	Wraps output_type().
on_init(…)	One-time setup after bind, before the first batch.	Returns max_processes=1.

The bind() and global_init() protocol entry points are @final — do not override them; use on_bind() / on_init(). A dynamic output type means annotating compute() with Returns() (no arrow type) and overriding output_type(params) to compute it from params.arguments_schema or params.constant_arguments.

Generator variant

ScalarFunctionGenerator gives you a per-batch process() (keyword-only: batch, init_call, init_response, storage, auth_context) and an abstract output_type(). Return a RecordBatch with the same row count as batch. Use it only when compute()’s columnar model is too restrictive.

Copy Copied!

class MyScalarGen(ScalarFunctionGenerator):
  @classmethod
  def output_type(cls, params) -> pa.DataType:
      return params.arguments_schema.field(0).type

  @classmethod
  def process(cls, *, batch, init_call, init_response, storage, auth_context) -> pa.RecordBatch:
      return pa.RecordBatch.from_arrays(
          [batch.column(0)],
          schema=pa.schema([("result", batch.schema[0].type)]),
      )

Table functions

TableFunctionGenerator[TArgs, TState] generates rows from arguments, with no input table. You declare a typed FunctionArguments dataclass (Annotated[T, Arg(...)] fields), an output schema, and a process() that emits batches until out.finish(). State (TState) persists between process() calls and must be encodable — it needs serialize_to_bytes() / deserialize_from_bytes(), which ArrowSerializableDataclass writes for you — so it survives HTTP round-trips. See Function patterns → Table and the Streaming with state section for runnable templates.

Method contracts

Method / attribute	When to override	Default
process(params, state, out)	Always — emit batches via out.emit(), end with out.finish().	Required.
FIXED_SCHEMA or on_bind(params)	Define the output columns (static schema vs. computed at bind).	Required.
initial_state(params)	Initialize per-worker TState.	Returns None.
cardinality(params)	Provide a row-count estimate for the optimizer.	Returns None.

process() is called repeatedly until you call out.finish() — emit a bounded chunk per call and remember your place in state. FunctionArguments is auto-extracted from the first generic parameter when you write TableFunctionGenerator[MyArgs, MyState], or you can set it explicitly.

Lifecycle decorators

Two class decorators wire up the common single-worker pattern so you don’t hand-write the boilerplate:

Decorator	Effect
@bind_fixed_schema	Generates an on_bind() that returns cls.FIXED_SCHEMA as the output schema. Requires FIXED_SCHEMA to be a pa.Schema.
@init_single_worker	Pins the function to a single worker process (no parallel fan-out) — the common case for stateful generators.

In-band logging

From inside process(), surface progress to the client with out.client_log(level, message):

Copy Copied!

from vgi_rpc.log import Level

@classmethod
def process(cls, params, state, out):
  if state.index == 0:
      out.client_log(Level.INFO, "Starting generation")
  if state.remaining <= 0:
      out.client_log(Level.INFO, "Generation complete")
      out.finish()
      return
  out.emit(batch)
  state.index += 1

Table-in-out functions

For transforming a streamed input relation, use TableInOutFunction (declarative) or TableInOutGenerator (generator). Both stream input batch-by-batch; the difference is the callback surface. See Function patterns → Table-in-out for a runnable template, and the Buffering section when output depends on every row.

TableInOutFunction (declarative)

You override transform() and optionally finish(); state management is automatic. TState is the second type parameter — when it supplies a serialization codec (an ArrowSerializableDataclass, or your own serialize_to_bytes / deserialize_from_bytes), the framework saves it to storage after each transform() for distributed processing. When omitted, transform() always receives state=None.

Method	Role	Default
transform(batch, params, state)	Transform one input batch → one batch (or a list, which is concatenated).	Passthrough.
finish(params, states)	Emit final output after all input is seen, given the accumulated per-partition states.	Returns [].
initial_state(params)	Create the initial TState.	Returns None.
on_bind(params)	Declare the output schema.	Pass-through: output schema = input schema.

Defining finish() changes the DuckDB registration

The framework only advertises a finalize callback to DuckDB when a real finish()/finalize() override is present (auto-detected, or forced via Meta.has_finalize). DuckDB rejects LATERAL with correlated input on table functions that register a finalize callback — so a passthrough/streaming transform should not define finish() if it needs to support correlated LATERAL.

TableInOutGenerator (generator)

For full control, override process(params, state, batch, out) and emit via OutputCollector. Override finalize(params) to return any buffered output. The default process() is a passthrough (echo), so an empty subclass is a valid identity function.

Copy Copied!

class MyFunction(TableInOutGenerator[MyArgs, MyState]):
  @classmethod
  def on_bind(cls, params: BindParams[MyArgs]) -> BindResponse:
      assert params.bind_call.input_schema is not None
      return BindResponse(output_schema=params.bind_call.input_schema)

  @classmethod
  def process(cls, params, state, batch, out: OutputCollector) -> None:
      out.emit(batch)

  @classmethod
  def finalize(cls, params) -> list[pa.RecordBatch]:
      return []

To emit more than one output batch for a single input, set the vgi.status metadata to HAVE_MORE_OUTPUT:

Copy Copied!

def process(cls, params, state, batch, out: OutputCollector) -> None:
  out.emit(batch, custom_metadata={b"vgi.status": b"HAVE_MORE_OUTPUT"})

OutputCollector also carries out.client_log(level, message) for in-band logging, and (over HTTP) out.remaining_response_bytes / out.externalization_enabled for sizing emits within the response budget.

Cancellation hook

TableInOutGenerator exposes on_cancel(params, state), fired when DuckDB tears down a scan early (upstream LIMIT, Ctrl-C, exception unwind). Override it to release per-stream resources held in state — database cursors, streaming sessions, file handles, GPU buffers.

Pitfalls

A reference checklist of the mistakes that bite most often:

Forgetting out.finish() in a table function

process() is called in a loop; without out.finish() the client hangs forever waiting for more output.

Copy Copied!

# WRONG — never signals completion
@classmethod
def process(cls, params, state, out):
  if state.remaining <= 0:
      return  # missing out.finish()!
  out.emit(batch)

# CORRECT
@classmethod
def process(cls, params, state, out):
  if state.remaining <= 0:
      out.finish()
      return
  out.emit(batch)

Not advancing state in process()

If process() emits without decrementing its counter (or otherwise advancing state), the loop never terminates.

Copy Copied!

# WRONG — infinite loop
@classmethod
def process(cls, params, state, out):
  out.emit(batch)
  # missing: state.remaining -= 1

# CORRECT
@classmethod
def process(cls, params, state, out):
  if state.remaining <= 0:
      out.finish()
      return
  out.emit(batch)
  state.remaining -= 1

Changing row count in a scalar compute()

Scalar functions require 1:1 row mapping — filtering or expanding rows raises RowCountMismatchError. Use a table-in-out function (N→M) or a buffering function instead.

State that can’t be turned into bytes

Table-generator and table-in-out TState is serialized between calls (and across workers / HTTP round-trips), so it has to be encodable. Since 0.24.0 the requirement is a structural protocol, StreamStateCodec: the state class needs serialize_to_bytes() and deserialize_from_bytes(). Extending ArrowSerializableDataclass writes both for you and stays the default; implementing them yourself lets you own the encoding (see State doesn’t have to be Arrow). A plain dataclass that has neither raises a TypeError at class-definition time — it would appear to work on the subprocess transport, where the worker is long-lived, and break on HTTP, where each tick is an independent request.

Next steps

• Runnable templates → Function patterns.
• When each callback fires → Function lifecycle.
• Aggregates, which have their own contracts → Aggregate functions.
• Exact signatures → scalar_function · table_function · table_in_out_function.