Function lifecycle

Knowing when each lifecycle method runs is key to managing resources and getting distributed processing right. Every VGI function shape is driven by the engine through the same skeleton — bind → init → per-batch processing → (optionally) finalize — but the shapes differ in whether they have a finalize phase and how they behave across parallel workers. The sequence diagrams below trace the calls between the DuckDB engine and your worker.

This page describes the protocol, so it applies to every SDK. The phase names are the protocol’s; each language spells the callbacks in its own idiom. Not every shape runs every phase — a scalar function has no init and no state, an aggregate accumulates with update rather than a per-batch process — so read this as a glossary, and the per-shape sections below for what actually fires:

Phase	Python	Go
Bind	on_bind()	OnBind()
Init	on_init() / global_init()	OnInit()
Initial state	initial_state()	NewState()
Per batch	compute() (scalar) / process()	Process()
Accumulate (aggregate)	update()	Update()
Merge partials	combine()	Combine()
Finalize	finalize()	Finalize()

DuckDB callbacks? DuckDB lifecycle, VGI worker callbacks

The phases mirror DuckDB’s execution model, but the method names above are the VGI worker API, not DuckDB’s. The VGI extension translates DuckDB’s bind/init/scan/aggregate work into typed worker calls, then moves batches across that boundary as Arrow.

The shapes at a glance

A runnable worker for every shape lives in the Function patterns guide for your language: Python · Go.

Shape	Lifecycle	Finalize phase?
Scalar	bind → init → compute per batch	No — ends when input ends
Table (producer)	bind → init → process emits batches	Self-terminated via out.finish()
Table-in-out	bind → init(INPUT) → process per batch → init(FINALIZE) → finalize	Yes — except blended, which has none
Aggregate	initial_state → update → combine → finalize	Yes — one row per group
Buffering	process (sink) → combine → finalize (source)	Yes — streamed from storage

The producer Table lifecycle is the simplest stateful generator — init builds state, then process is called repeatedly to emit batches until it calls out.finish(); see the tutorial (Python · Go). The remaining shapes are detailed below.

Scalar — batches in, one column out

1 row → 1 value

A pure per-row transform.

The simplest lifecycle: bind, then init, then a compute call for each input batch returning an output array with the same row count — ending when the input stream closes. There is no finalize phase and no distributed state.

Table-in-out — stream through, then finalize

N rows → M rows

Consumes a relation and streams a transformed relation back.

In a single worker (max_workers=1), a table-in-out function processes every input batch, then gets a distinct finalize phase to emit anything it held back.

Running table-in-out in parallel — primary and secondary workers

The same table-in-out function, but with max_workers > 1 the engine spawns several worker processes. One becomes the primary (it runs finalize); the rest are secondaries and never finalize.

Aspect	Primary worker	Secondary workers
global_init() called?	Yes	No (uses the primary’s execution_id)
finalize() called?	Yes	No
Which batches?	A round-robin subset	A round-robin subset

Aggregate — update, combine, finalize

N rows → 1 value

Folds many rows into one value per group.

Aggregates are driven by DuckDB’s GROUP BY and run four hooks per invocation:

1. initial_state(params) — the identity state for a group, created the first time a group_id is seen (e.g. 0 for sum, an empty list for list-agg).
2. update(states, group_ids, …) — fold each input batch into per-group state. Runs in every worker, in parallel, over that worker’s share of the rows.
3. combine(source, target) — merge two partial states for the same group across parallel workers into one.
4. finalize(group_ids, states) — produce the result, one output row per group.

Buffering — sink, combine, source

stream → [state] → stream

Holds every input row before emitting — sorts, top-k, full reductions.

Use a buffering function when output depends on the whole input (a global sort, top-k, or a full reduction). It runs in three phases, and because they can run in different worker processes, state lives in params.storage (a BoundStorage keyed by execution_id), not in memory:

1. process(batch, params) — the sink: stash each batch in params.storage and return a state_id. Runs per input batch, in parallel across workers.
2. combine(state_ids, params) — reduce all the partials once on the coordinator into the finalize_state_id(s).
3. finalize(params, fid, state, out) — the source: stream the result out, one batch per tick.

Aggregate vs. buffering

Both fold many rows down, but an aggregate is grouped (GROUP BY) and keeps small per-group state in memory across update/combine/finalize; a buffering function holds the entire relation in cross-process params.storage because it needs every row before emitting any output.

Which hook runs when

Hook	Used by	When / use for
on_bind()	all shapes	At bind — validate arguments, set the output schema
compute()	scalar	Per batch — transform a column into one output column
process() / transform()	table, table-in-out, buffering	Per input batch — emit (or stash, for buffering) output
initial_state() / update()	aggregate	Create per-group state, then fold each batch into it
combine()	aggregate, buffering	Merge partial states across parallel workers
finalize()	table-in-out, aggregate, buffering	Emit final results (one row per group for aggregate; streamed for buffering). Not available on a blended RowTransformFunction — overriding it is rejected at resolve_metadata, because DuckDB forbids FinalExecute under correlated LATERAL.
params.storage	buffering	Cross-process state keyed by execution_id (sink → source)

Next steps

• A runnable worker for each shape → Function patterns: Python · Go.
• The exact callback contracts → Python Function API · Python aggregates · Go API reference.
• Why phases can’t share memory → Persist state across workers.