Skip to content
Query.Farm
Talk with Us

Expose a catalog

How a worker presents itself to DuckDB as a catalog — a named namespace of schemas, functions, tables and views you reach with ATTACH. Read this once you’ve done the tutorial and want to expose data, not just functions.

Prerequisites

Section titled “Prerequisites”

The model

Section titled “The model”

Every worker already has a catalog — vgi.WithCatalogName("calc") in the tutorial named it. What this guide adds is contents: schemas holding tables and views, addressed by qualified name.

ATTACH 'cat' (TYPE vgi, LOCATION './catalogworker');

-- catalog.schema.object
SELECT * FROM cat.data.cities;

-- functions land in the catalog's default schema
SELECT cat.main.some_function(1);

Here is a worker that exposes one table and one view:

catalog/main.go 
// Copyright 2025, 2026 Query Farm LLC - https://query.farm

// Command catalog is the catalog example for the vgi-go documentation.
//
// A worker does not have to be a bag of functions. It can present itself as a
// database: a named catalog you ATTACH, holding schemas that hold tables and
// views, queried with ordinary qualified names.
//
// The table here is *function-backed*: `RegisterCatalogTable` is given a table
// function plus the arguments to call it with, so `SELECT * FROM cat.data.cities`
// runs the function with those arguments baked in. The user never passes them,
// and never sees the function.
//
//	go build -o catalogworker .
//	# then, in a Haybarn shell:
//	ATTACH 'cat' (TYPE vgi, LOCATION './catalogworker');
//	SELECT * FROM cat.data.cities;
//	SELECT * FROM cat.data.big_cities;   -- a view over the table
package main

import (
	"context"
	"flag"
	"log"

	"github.com/Query-farm/vgi-go/vgi"
	"github.com/Query-farm/vgi-rpc-go/vgirpc"
	"github.com/apache/arrow-go/v18/arrow"
	"github.com/apache/arrow-go/v18/arrow/array"
	"github.com/apache/arrow-go/v18/arrow/memory"
)

// The table's shape. Declaring Columns explicitly on the CatalogTable lets
// DuckDB describe the table without calling the worker at all.
var citiesSchema = arrow.NewSchema([]arrow.Field{
	{Name: "name", Type: arrow.BinaryTypes.String},
	{Name: "population", Type: arrow.PrimitiveTypes.Int64},
}, nil)

// Fields are exported because this type ends up inside the scan state, and
// state has to be gob-encodable so it can survive an HTTP continuation. The SDK
// checks at registration: a struct whose fields are all unexported panics with
// "type ... has no exported fields" the moment AsTableFunction is called, rather
// than mid-query on the first continuation.
type city struct {
	Name string
	Pop  int64
}

// Stands in for whatever the worker actually fronts — a remote API, a file
// format, a device.
var cities = []city{
	{Name: "Charlottesville", Pop: 51_000},
	{Name: "Richmond", Pop: 230_000},
	{Name: "Virginia Beach", Pop: 457_000},
}

type citiesArgs struct {
	MinPopulation int64 `vgi:"pos=0,ge=0,doc=Only return cities at least this large"`
}

// citiesState materializes the whole filtered result up front. That is fine for
// three rows and wrong for three million: state is carried between Process calls
// and gob-encoded across an HTTP continuation, so anything held here is paid for
// repeatedly. A real scan keeps a *cursor* — an offset, a page token, an open
// iterator id — and fetches each batch in Process.
type citiesState struct {
	vgi.BatchState
	Rows []city
}

// CitiesFn is the scan behind the table. It is an ordinary table function —
// nothing about it knows it is backing a catalog table.
type CitiesFn struct{}

var _ vgi.TypedTableFunc[citiesState] = (*CitiesFn)(nil)

func (*CitiesFn) Name() string { return "cities_scan" }

func (*CitiesFn) Metadata() vgi.FunctionMetadata {
	return vgi.FunctionMetadata{
		Description: "Scans the cities table, optionally filtered by population",
		Stability:   vgi.StabilityConsistent,
	}
}

func (*CitiesFn) ArgumentSpecs() []vgi.ArgSpec { return vgi.DeriveArgSpecs(citiesArgs{}) }

func (*CitiesFn) OnBind(_ *vgi.BindParams) (*vgi.BindResponse, error) {
	return vgi.BindSchema(citiesSchema)
}

func (*CitiesFn) NewState(params *vgi.ProcessParams) (*citiesState, error) {
	var args citiesArgs
	if err := vgi.BindArgs(params.Args, &args); err != nil {
		return nil, err
	}
	rows := selectCities(args.MinPopulation)
	return &citiesState{
		BatchState: vgi.NewBatchState(int64(len(rows)), 1024),
		Rows:       rows,
	}, nil
}

// selectCities is the filter, kept separate from the plumbing so it can be read
// and tested on its own.
func selectCities(minPopulation int64) []city {
	var out []city
	for _, c := range cities {
		if c.Pop >= minPopulation {
			out = append(out, c)
		}
	}
	return out
}

func (*CitiesFn) Process(_ context.Context, _ *vgi.ProcessParams, state *citiesState, out *vgirpc.OutputCollector) error {
	return vgi.GenerateBatch(&state.BatchState, out, func(size int64) ([]arrow.Array, error) {
		start := state.Index
		mem := memory.NewGoAllocator()

		names := array.NewStringBuilder(mem)
		defer names.Release()
		pops := array.NewInt64Builder(mem)
		defer pops.Release()

		for i := int64(0); i < size; i++ {
			row := state.Rows[start+i]
			names.Append(row.Name)
			pops.Append(row.Pop)
		}
		return []arrow.Array{names.NewArray(), pops.NewArray()}, nil
	})
}

// NewCitiesScan returns the registration-ready scan function.
func NewCitiesScan() vgi.TableFunction {
	return vgi.AsTableFunction[citiesState](&CitiesFn{})
}

func main() {
	httpMode := flag.Bool("http", false, "serve over HTTP instead of stdio")
	logFlags := vgi.RegisterLoggingFlags(flag.CommandLine)
	flag.Parse()
	if err := logFlags.Apply(); err != nil {
		log.Fatalf("logging flags: %v", err)
	}

	w := vgi.NewWorker(
		vgi.WithCatalogName("cat"),
		vgi.WithCatalogComment("Documentation example: a worker presented as a database"),
	)

	// A function-backed table. FuncArgs are bound at scan time, so the user
	// writes `SELECT * FROM cat.data.cities` with no arguments at all.
	//
	// Registering the table also registers its backing function in the
	// catalog's default schema — that is where the extension resolves the scan.
	w.RegisterCatalogTable("data", vgi.CatalogTable{
		Name:     "cities",
		Comment:  "Every city the worker knows about",
		Columns:  citiesSchema,
		Function: NewCitiesScan(),
		FuncArgs: []vgi.CatalogTableArg{
			{Position: 0, Value: int64(0), Type: arrow.PrimitiveTypes.Int64},
		},
		NotNull:        []string{"name"},
		ColumnComments: map[string]string{"population": "Most recent estimate"},
	})

	// A view is pure SQL that DuckDB evaluates — no worker round trip at all
	// once the definition has been advertised.
	w.RegisterCatalogView("data", vgi.CatalogView{
		Name:       "big_cities",
		Comment:    "Cities with a population of at least 100,000",
		Definition: "SELECT * FROM cat.data.cities WHERE population >= 100000",
	})

	if *httpMode {
		if err := w.RunHttp("127.0.0.1:0"); err != nil {
			log.Fatal(err)
		}
		return
	}
	w.RunStdio()
}

SELECT * FROM cat.data.cities ORDER BY name;
┌─────────────────┬────────────┐
│      name       │ population │
├─────────────────┼────────────┤
│ Charlottesville │      51000 │
│ Richmond        │     230000 │
│ Virginia Beach  │     457000 │
└─────────────────┴────────────┘

Function-backed tables

Section titled “Function-backed tables”

A CatalogTable with a Function and FuncArgs is function-backed: DuckDB scans it by calling that table function with those arguments already bound. The user writes SELECT * FROM cat.data.cities and never passes an argument, never sees the function.

w.RegisterCatalogTable("data", vgi.CatalogTable{
  Name:     "cities",
  Comment:  "Every city the worker knows about",
  Columns:  citiesSchema,
  Function: NewCitiesScan(),
  FuncArgs: []vgi.CatalogTableArg{
      {Position: 0, Value: int64(0), Type: arrow.PrimitiveTypes.Int64},
  },
  NotNull:        []string{"name"},
  ColumnComments: map[string]string{"population": "Most recent estimate"},
})

Declaring Columns explicitly means DuckDB can describe the table without calling the worker at all. Leave it nil and the columns are derived from the function’s OnBind response instead — one fewer place to keep in sync, at the cost of a round trip to describe.

Registering the table also registers its backing function in the catalog’s default schema, not the table’s. That is where the extension resolves a function-backed table’s scan.

How this differs from native DuckDB tables

VGI tables live in DuckDB’s catalog model, so you query them with ordinary qualified names. The difference is where the rows come from: a native table is stored and managed by DuckDB, while a VGI table delegates its scan to your worker.

Views

Section titled “Views”

A view is pure SQL. Once the definition has been advertised at attach time, DuckDB evaluates it itself — there is no worker round trip for the view as such, only for whatever tables it reads.

w.RegisterCatalogView("data", vgi.CatalogView{
  Name:       "big_cities",
  Comment:    "Cities with a population of at least 100,000",
  Definition: "SELECT * FROM cat.data.cities WHERE population >= 100000",
})
Qualifying the definition with the catalog name is safe

cat is hardcoded there, which normally would be fragile — but the name in ATTACH is not a free alias. It must match WithCatalogName, and attaching under any other name fails at once with “No worker handles catalog ‘mydb’”. So the catalog name your worker declares is the name the view will always be read under, and qualifying with it is correct rather than a latent bug.

Scan state must be gob-encodable, and the SDK checks at registration

A table function’s per-scan state has to survive an HTTP continuation, so it is gob-encoded. Two things gob cannot handle, both caught by AsTableFunction at registration rather than mid-query:

  • a struct whose fields are all unexported — gob encodes nothing and reports “type … has no exported fields”;
  • an exported field of a kind gob can’t encode: an interface (commonly an arrow.Record stashed to emit later), a chan, a func, or an unsafe.Pointer.

That is why city in the example exports Name and Pop. Store plain serializable Go values in state and rebuild Arrow batches in Process.

Keep a cursor in state, not the result set

The example puts the whole filtered slice in citiesState.Rows, which is honest for three rows and the wrong shape for three million: state is carried between Process calls and gob-encoded across an HTTP continuation, so whatever you hold there is paid for repeatedly. A real scan stores a cursor — an offset, a page token, an iterator id — and fetches each batch inside Process.

What else a table can declare

Section titled “What else a table can declare”

CatalogTable carries considerably more than the example uses. The full list is on Catalogs; the ones worth knowing about early:

FieldWhat it does
NotNull, Unique, PrimaryKey, Check, ForeignKeyConstraints, surfaced to DuckDB’s planner and to duckdb_constraints().
Defaults, GeneratedColumn defaults and generated (virtual) column expressions.
ColumnComments, TagsDocumentation that shows up in duckdb_columns() and duckdb_tables().
Statistics, CardinalityEstimateOptimizer hints — see Integrate with the optimizer.
RequiredFiltersRefuse an unbounded scan. AND-of-ORs of column paths; the extension throws a BinderException naming any unsatisfied group.
SupportsTimeTravelThe table answers AT (VERSION …) / AT (TIMESTAMP …) queries.

Requiring a filter

Section titled “Requiring a filter”

A table fronting a remote API often cannot be scanned wholesale — the upstream needs a key. Declaring RequiredFilters makes the extension reject such a scan at bind rather than issuing an unbounded request:

// "accession_number AND one of (ticker, cik)"
RequiredFilters: [][]string{{"accession_number"}, {"ticker", "cik"}},

The outer list is an AND of groups, each inner group an OR of dotted-path column references. Satisfaction is prefix-based, so a filter on a shorter path satisfies every path it prefixes. Empty (the default) means no enforcement.

Worker-level options

Section titled “Worker-level options”

Some capabilities are declared on the worker rather than a table:

OptionWhat it does
vgi.WithCatalogName, vgi.WithCatalogCommentName and describe the catalog itself.
vgi.WithGlobalFunctions, vgi.WithGlobalFunctionPrefixPublish selected functions into DuckDB’s global namespace, so they are callable without naming the catalog. Best-effort and advisory — treat it as an alias for the qualified name.
vgi.WithAttachCatalogsAsk the client to ATTACH companion catalogs (a lakehouse, a Postgres) alongside yours. Namespace the alias by your own catalog identity; collisions are rejected, never merged.

Next steps

Section titled “Next steps”