Skip to content
Query.Farm
Talk with Us

Expose a catalog

How a worker presents itself to DuckDB as a catalog — a named namespace of schemas, functions, tables, and views you reach with ATTACH. Read this once you’ve done the tutorial and want to understand how your functions get qualified names, or to expose data (not just functions).

Prerequisites

Section titled “Prerequisites”

The model

Section titled “The model”

Every worker exposes one Catalog with a name. Inside it are one or more Schema namespaces (DuckDB’s default is main), each holding functions — and optionally tables and views. You attach the catalog and address its contents by name:

ATTACH 'calc' (TYPE vgi, LOCATION 'uv run calc_worker.py');

-- catalog.function  (functions in `main` are reachable as catalog.name)
SELECT calc.double(21);

-- catalog.schema.object  (fully qualified)
SELECT * FROM calc.main.series(3);

The worker from the tutorial is exactly this — a catalog named calc with a main schema holding the two functions:

# /// script
# requires-python = ">=3.13"
# dependencies = ["vgi-python"]
# ///
"""The full tutorial worker: a scalar function and a table function.

Run from a DuckDB-compatible engine (Haybarn shown here)::

    uvx haybarn-cli
    ATTACH 'calc' (TYPE vgi, LOCATION 'uv run calc_worker.py');
    SELECT calc.double(21);            -- scalar -> 42
    SELECT * FROM calc.series(3);       -- table  -> 0, 1, 2
"""

from dataclasses import dataclass
from typing import Annotated, ClassVar

import pyarrow as pa
import pyarrow.compute as pc

from vgi import Arg, Param, Returns, ScalarFunction, Worker
from vgi.catalog import Catalog, Schema
from vgi.table_function import (
    OutputCollector,
    ProcessParams,
    TableFunctionGenerator,
    bind_fixed_schema,
    init_single_worker,
)


class Double(ScalarFunction):
    """Double each input value (one row in, one row out)."""

    @classmethod
    def compute(
        cls,
        value: Annotated[pa.Int64Array, Param(doc="Values to double")],
    ) -> Annotated[pa.Int64Array, Returns()]:
        """Multiply the whole column by 2."""
        return pc.multiply(value, 2)




@dataclass(slots=True, frozen=True, kw_only=True)
class SeriesArgs:
    """Arguments for :class:`Series` (one positional ``count``)."""

    count: Annotated[int, Arg(0, doc="How many numbers to generate", ge=0)]


@init_single_worker
@bind_fixed_schema
class Series(TableFunctionGenerator[SeriesArgs]):
    """Generate the integers ``0 .. count-1`` as a one-column table.

    Stateless: it emits every row in a single ``process`` call, then finishes.
    ``@bind_fixed_schema`` derives the output schema from ``FIXED_SCHEMA``;
    ``@init_single_worker`` runs the generator in a single worker.
    """

    FIXED_SCHEMA: ClassVar[pa.Schema] = pa.schema([("n", pa.int64())])

    @classmethod
    def process(cls, params: ProcessParams[SeriesArgs], state: None, out: OutputCollector) -> None:
        """Emit all rows at once, then signal completion."""
        out.emit(pa.RecordBatch.from_pydict({"n": list(range(params.args.count))}, schema=params.output_schema))
        out.finish()




class CalcWorker(Worker):
    """A worker exposing the ``calc`` catalog with both functions."""

    catalog = Catalog(
        name="calc",
        schemas=[Schema(name="main", functions=[Double, Series])],
    )


if __name__ == "__main__":
    CalcWorker().run()

The SQL name of a function is the snake_case of its class name (Double → double), unless you override it with a Meta.name (as sum_worker.py does for vgi_sum).

Exposing data: tables and views

Section titled “Exposing data: tables and views”
How this differs from native DuckDB tables

VGI exposes tables and views through DuckDB’s catalog model, so you query them with normal qualified names. The difference is where rows come from: native DuckDB tables are stored and managed by DuckDB, while VGI tables delegate scanning to a table function or another scan function such as read_parquet.

A catalog can expose more than functions:

  • View — a named SQL query DuckDB evaluates. Pure SQL; no data provider needed:
from vgi.catalog import View
View(name="recent", definition="SELECT * FROM calc.series(5)")
  • Table — a queryable table. Define it with an explicit columns schema (you supply the scan) or back it with a TableFunctionGenerator so the schema is derived from the function.

Both are passed to a Schema(..., tables=[...], views=[...]). The full set of options — constraints, generated columns, column comments, filter requirements — is covered in the Catalog Interface reference.

Beyond the basics

Section titled “Beyond the basics”

Two catalog features that matter once a catalog fronts a real data source, both covered in the Catalog Interface reference:

  • Required filters — refuse an unbounded scan when your upstream needs a key, rather than issuing the request anyway.
  • Companion catalogs — have the client ATTACH a lakehouse (DuckLake, Iceberg, Postgres) alongside yours.

Next steps

Section titled “Next steps”